Uptime falling every time, but chunk are producing

[strangeweb3guy]

1. Have a strong machine AX41-NVME https://www.hetzner.com/dedicated-rootserver/matrix-ax with good internet
2. Logs are clear
3. 24567 - opened and accessible from my local machine
4. Keys are the same But my uptime falling every time and day

My account: nearuaguild.shardnet.near

[stanlagermin]

I think problem is from location.

[strangeweb3guy]

Location is Finland, how it could be a problem?

[DDeAlmeida]

@bowenwang1996 If your team can check

[mm-near]

@kiskesis - hey, thanks for reporting.

From looking at the logs, your system seems healthy, but there can be multiple reasons why your node didn't produce the chunk/block.

The most two common reasons:

• node was 'behind'
• node didn't get enough acknowledgements

Looking at your logs, the second case seems more probable. You can also see yourself, by looking at your debug page (http://localhost:3030/debug/pages/validator)

Here's the example from one of our validators (boot1): And as you can see, block 995 was not produced, as it didn't get enough approvals on time.

(this is due to quite unreliable network between validators - which we expected during this shardnet experiment - and we're actively working on improvements in this area).

[strangeweb3guy]

@mm-near So, you say that the problem could be on the validator software side? Do I need just to wait until when the network will be more stable?

Actually, my debug page looks different a little bit http://65.109.26.103:3030/debug/pages/validator

[gueux]

I have the same issue with my node. It has a big gap between produced and expected chunks at the end of epoch. Thats how it looks like on monitoring dashboard:

My node locates in Frankfurt. Meanwhile the debug page http://18.185.240.197:31635/debug/pages/validator looks the same as for @kiskesis

A lot of "yellow" (i guess not approved blocks).

[Strafi]

+1 to this issue Node is located in the UK Validator debug page: http://149.102.154.201:3030/debug/pages/validator

[joesixpack]

(this is due to quite unreliable network between validators - which we expected during this shardnet experiment - and we're actively working on improvements in this area).

Did you change something? Because suddenly the network is performing significantly better starting sometime during the past 13 hours as of this post.

[Thesephi]

(this is due to quite unreliable network between validators - which we expected during this shardnet experiment - and we're actively working on improvements in this area).

Did you change something? Because suddenly the network is performing significantly better starting sometime during the past 13 hours as of this post.

Hey @joesixpack , good to know it works better for you. But unfortunately I gotta chime in and share that it has not been getting any better for me. In fact, for the past 24 hours or so, it's getter worse for me. Before, my node was constantly in the validator list, not it gets kicked out once every few Epochs 🤕

[msannikov]

I have the same issue with my validator max.factory.shardnet.near, that is located in UK. I've thought about migrating it to US or getting a dedicated EU server.

@mm-near do you know countries where boot nodes are spinned up? I guess being closer to them might be beneficial for online (but not for decentralisation of course). I am wondering whether there is an api for getting ip addresses of validators, so I could play with data myself.

[strangeweb3guy]

@mm-near so maybe it will help you. So a new epoch started and I noticed that the first chunk was produced and others - did not, so I opened the journal and saw next:

And after each that message, I saw that my chunks fall behind

Also, I've noticed those warnings, but tbh I don't think that they related to the problem with chunks production

Maybe it will help you to help me 😂

[joesixpack]

@Thesephi Seems like the "good times" ended as of epoch 5znueneSL7gZcvr42in9fRJTJ6DpKjtcbWQd72tcTXU6 which started 2h 23m 48s ago. Maybe your performance is better since then?

[mm-near]

Thanks a lot for these logs/screenshots.

We were able to find one issue with our network code - that has caused some amount of packets to be lost (and therefore many nodes didn't get their approvals/chunks on time).

We're currently working on the fix - and will let you know once it is ready.

[mm-near]

(one of the PRs is https://github.com/near/nearcore/pull/7322)

[strangeweb3guy]

Okay, after updating with the new commit situation looks better, but still a lot of missed chunks and blocks. It could be because not everybody is updated, IDK, will monitor.

But still, see a lot of warnings like this WARN network: Peer stream error err=Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }

[gueux]

@mm-near

After updating to new commit, the node has a lot of "missing chunks" messages into the logs

1701930 GBfrRp7aw7NATiGAMomMe5uVSSr2pxE7gNsHYKzkBiX5 WaitingForChunks in progress for 400ms  missing chunks for 388ms Chunks:(✔⬇XX))
--
  | 1701929 3aRGM64b6RxP81kzMaVL8nFnwBAZq5BNY3oXnEWLVxGH Processed in progress for 810ms  missing chunks for 784ms Chunks:(✔✔✔✔))
  | 1701928 CNL4N5tAaCr79KSpMsH1eoS7hAAPmuichKg2PApVqhs7 Processed in progress for 712ms  missing chunks for 691ms Chunks:(✔✔✔✔))
  | 1701927 9Sz3eCx9sVQX3YsVddgGN3e9jfQxwpcnEXqEPExDrJq1 Processed in progress for 729ms  missing chunks for 703ms Chunks:(✔✔✔✔))
  | 1701926 5yDAJC5fAvjbESQcu43qcgUAYHNKPR2Si1aaZQAJ3z3o Processed in progress for 732ms  missing chunks for 705ms Chunks:(✔✔X✔))
  | 1701925 9RnE1DJXpsueVgH4e9NGduCbkBGyPvt864eXqauSWww8 Processed in progress for 689ms  missing chunks for 662ms Chunks:(✔✔✔✔))
  | 1701924 DQit7SwveVo5aLACq3row72Sf4jbEZcfNwqQbdKa2n3Q Processed in progress for 843ms  missing chunks for 823ms Chunks:(✔✔✔✔))
  | 1701923 DUcqgsUTn34oGwNzz7ANRw7rh5x7tYkBE7crHpGbtP8x Processed in progress for 612ms  missing chunks for 578ms Chunks:(✔✔✔✔))
  | 1701922 AVQ9VPWzZ2Tby8u21AZBZS1sGRYsjw7pk9EwzCzp4hG4 Processed in progress for 634ms  missing chunks for 616ms Chunks:(✔✔✔✔))
  | 1701920 4hjjh4w2TnXyz2vWd9DjVwVquCFuxYnYWRwRSNkoSNVe Processed in progress for 796ms  missing chunks for 763ms Chunks:(✔✔X✔))
  | 1701919 Hi9d6F7LZ7sa3H4xG1SSQCfigKV6SXLywnzquFvQjwGj Processed in progress for 565ms  missing chunks for 548ms Chunks:(✔✔✔✔))
  | 1701918 9PmrE25ko4zjGwnzQnnVWuxoTwFBKET47LsfuYcDpuhh Processed in progress for 792ms  missing chunks for 770ms Chunks:(✔✔✔✔))
  | 1701917 7haLVyP1pkL7YwPSPFWTEmAW77r7eWvoygbg5GaC8JwD Processed in progress for 794ms  missing chunks for 784ms Chunks:(✔✔✔✔))
  | 1701916 GgM8SDMG2xjW8K8ZMmAjnu7xwH88FYnSFB1FGgyEHsy2 Processed in progress for 607ms  missing chunks for 587ms Chunks:(✔✔✔✔))
  | 1701915 G8afuJjmnn3djS9nW9Cnjy3LSk7U34V5LYf9cf9Zhqab Processed in progress for 756ms  missing chunks for 734ms Chunks:(✔✔✔✔))
  | 1701914 GTYLu8kVhxYRxJDLzDEhZaeHQmZCBmGEm6maDpCxC278 Processed in progress for 727ms  missing chunks for 691ms Chunks:(✔✔✔✔))
  | 1701913 GDK8P23b8G75FQRE6gUrs2kAfUkmHXDjHctZCb2hJ4P3 Processed in progress for 694ms  missing chunks for 682ms Chunks:(✔✔✔✔))
  | 1701912 D4wnBUG1Ufn9qz3UkLz1w48xK1g8h8KHvij5NRQRqE3g Processed in progress for 706ms orphan for 78ms missing chunks for 609ms Chunks:(✔✔✔✔))
  | 1701911 75EKqbD5tXpG4mJobjHx1bQvUWesM8PEZiuyHBdDSDFs Processed in progress for 632ms  missing chunks for 600ms Chunks:(✔✔X✔))
  | 1701910 583E1TRKRrkn4rjYPUYPkLf6tzPw71P78mqAuwereJrK Processed in progress for 809ms  missing chunks for 783ms Chunks:(✔✔✔X))
  | 1701909 DvvPHb7R7Yf36YLjqfdedpEc4QveNRgjoQHLBxrN8N9Y Processed in progress for 672ms  missing chunks for 657ms Chunks:(✔✔✔✔))
  | 1701907 Ar8cXDmsSTMeJCr5AYWN221mPCYhc3rTqRSZSzgP3cuq Processed in progress for 716ms  missing chunks for 695ms Chunks:(✔✔XX))
  | 1701906 AmQpac51keTm9v69WVJo1cUxbcrDozgCoaEn6JVSzLey Processed in progress for 649ms  missing chunks for 628ms Chunks:(✔✔XX))
  | 1701904 HMf3k1MsmD9nwW575pS8YCQxEPGU4L16EomAyM5ZGdyR Processed in progress for 871ms  missing chunks for 859ms Chunks:(✔✔✔✔))
  | 1701903 CHXp3mVVCNDTfVuar38dX18tSpwenKYfikLajfBo7jDy Processed in progress for 769ms  missing chunks for 748ms Chunks:(✔✔✔X))
  | 1701902 F8UmhtZWpatEcjfbQmtk9kb4Y1L38Kn7s3cP3oqn8VGE Processed in progress for 625ms  missing chunks for 602ms Chunks:(✔✔✔X))
  | 1701901 4uU3XrZHk5hXECvKNqrvAz7rUqycfEtxZ2qPigQoVoCE Processed in progress for 480ms orphan for 125ms missing chunks for 326ms Chunks:(✔✔✔✔))
  | 1701900 B1dn2KuJuciqsgiF2j4oSDAKfYy1CQgELJBkL5dMxc5p Processed in progress for 633ms  missing chunks for 605ms Chunks:(✔✔XX))
  | 1701898 FjEFpguz3ahnP1U5UxpF5ENoKGkBJGFWfC3F5Edoz9k Processed in progress for 936ms  missing chunks for 922ms Chunks:(✔✔✔✔))
  | 1701897 7xbGEGZswfGeGVTmbyKkvv2Gz9iQMGqauZLB9WKLK2xB Processed in progress for 932ms  missing chunks for 918ms Chunks:(✔✔✔✔))
  | 1701896 Ec4TQF5a9Vrd9K2rwRyLMtQjo8X39YaMErjHpvyuFDAv Processed in progress for 666ms  missing chunks for 642ms Chunks:(✔✔✔✔))
  | 1701895 4xC9Ri4cktnpM6KdzDftELBs7Y9dRFiYsU4VKeW3udHv Processed in progress for 700ms  missing chunks for 679ms Chunks:(✔✔✔X))
  | 1701894 5LYmNG2CLsCMffpraDHumYu6kEiWh6qQKKp8NMNkZEWp Processed in progress for 694ms  missing chunks for 676ms Chunks:(✔✔✔✔))
  | 1701893 8C7ddW6Rgrui9JJo65e2kdKaQqoSoeEjKsR6iyK7MLWd Processed in progress for 702ms  missing chunks for 692ms Chunks:(✔✔✔✔))
  | 1701892 4ArdUTeKMiVGP9WYkdDXGErxWXUyp7ErdfcriENi9DJi Processed in progress for 790ms  missing chunks for 773ms Chunks:(✔✔✔✔))
  | 1701891 GPSV7km4cG2Af6z4cW93DrHAUbZSsivkuGig7qb2Kvvo Processed in progress for 672ms  missing chunks for 649ms Chunks:(✔✔✔X))
  | 1701890 BSC2pRGwmzqzRNAoE3s85VRQrBrobc98tyrei2124eie Processed in progress for 628ms  missing chunks for 609ms Chunks:(✔✔✔✔))
  | 1701889 9XD1gHzeyMaWv6NZy9PdjEhft7odDNLJLFZpnvNDeJG5 Processed in progress for 833ms  missing chunks for 811ms Chunks:(✔✔✔✔))
  | 1701888 3wPfCKHYmNiFC99CzqiDwUPnHm3EuRu8XqhWDTuuTmUq Processed in progress for 681ms  missing chunks for 657ms Chunks:(✔✔XX))
  | 1701887 GdUqskbY7tXWAHo52BsyDmmuhYsdDS31fzSeZD569PJA Processed in progress for 594ms  missing chunks for 567ms Chunks:(✔X✔✔))
  | 1701886 E8j2kVtkYxn9JbXS61XEV4xNMCirRNHKTnTbCmHMiVdS Processed in progress for 924ms  missing chunks for 913ms Chunks:(✔✔✔✔))
  | 1701885 PdmBm5n7UMwjgrTVKCLbtwLn6C9R1d74oM8AdeZGTWE Processed in progress for 776ms  missing chunks for 757ms Chunks:(✔✔✔✔))
  | 1701884 29jx6mG5uHUGfsiAnXVFynDpQoyjWBQNejpYvRXZLqze Processed in progress for 640ms  missing chunks for 617ms Chunks:(✔✔X✔))
  | 1701883 88rGbcoGRSC2zLbK3VqcRnPMK2eZuhKTEAdPBWK9uNLZ Processed in progress for 664ms  missing chunks for 638ms Chunks:(✔✔X✔))
  | 1701882 rALdqWam8FezNmSt3kewgrZPLzndUyPwPkMxGUdcQU6 Processed in progress for 866ms  missing chunks for 831ms Chunks:(✔✔✔✔))
  | 1701881 9XUBhzvhD5KARac5mYy2LbZ245XvNwngCdpfbrhAVoPp Processed in progress for 607ms  missing chunks for 573ms Chunks:(✔✔✔✔))
  | 1701880 4tWxMgahhxAFS1xcyFqpvJ7zzGUawcPUHMVQVoNG1QP1 Processed in progress for 696ms  missing chunks for 675ms Chunks:(✔✔✔✔))
  | 1701879 EnApPptm8m3Jn3XhzNCEncX6THudUv39L58F9oUsfW1j Processed in progress for 763ms  missing chunks for 740ms Chunks:(✔✔✔✔))
  | 2022-08-03T14:23:19.350023Z  WARN telemetry: Failed to send telemetry data err=Timeout endpoint="https://explorer.shardnet.near.org/api/nodes"
  | 2022-08-03T14:23:20.101291Z  WARN network: Received invalid tx from peer Some(ed25519:8TVEKKFk4YaC2j2fppzK29LBiAUHhEkhJAQRT97ZunfC@185.229.119.225:24567): Transaction nonce 1267977002863 must be larger than nonce of the used access key 1267977002865
  | 2022-08-03T14:23:29.347630Z  INFO stats: # 1701936 4iPf9L53jnp7kPwu1m7GnV9uZjDeBCoNBWDAiCTj2EMV Validator \| 100 validators 30 peers ⬇ 1.09 MB/s ⬆ 1.61 MB/s 0.60 bps 1.76 Tgas/s CPU: 152%, Mem: 3.94 GB
  | 2022-08-03T14:23:29.349014Z DEBUG stats: EpochId(`7P2rGDXpGDN4TZE5YMiFk1519AvUHhzuatMGw8QhCrdo`) Orphans: 0 With missing chunks: 0 In processing 0
  | 1701936 4iPf9L53jnp7kPwu1m7GnV9uZjDeBCoNBWDAiCTj2EMV Processed in progress for 564ms  missing chunks for 538ms Chunks:(✔✔✔X))
  | 1701935 BruUGb8cLUXiumQrkgAs3phWPiRqUWuAfovsr1FAPgb4 Processed in progress for 795ms  missing chunks for 768ms Chunks:(✔✔✔✔))
  | 1701934 HJTzxwyMGyqPDzacMWCQPEUBvpRP9EGVNunzaM7L7HFs Processed in progress for 719ms  missing chunks for 698ms Chunks:(✔✔✔✔))
  | 1701933 5vPaUwCr6yM7iPCVSaH9vqctnQyAvjXn31A2F3NcKmPW Processed in progress for 558ms  missing chunks for 543ms Chunks:(✔✔✔✔))
  | 1701931 23Go54ehEUq5cNvw4VkBLdV8GAND5sdHUbYUvwfPn2L4 Processed in progress for 574ms  missing chunks for 552ms Chunks:(✔✔X✔))
  | 1701930 GBfrRp7aw7NATiGAMomMe5uVSSr2pxE7gNsHYKzkBiX5 Processed in progress for 694ms  missing chunks for 668ms Chunks:(✔✔XX))
  | 1701929 3aRGM64b6RxP81kzMaVL8nFnwBAZq5BNY3oXnEWLVxGH Processed in progress for 810ms  missing chunks for 784ms Chunks:(✔✔✔✔))
  | 1701928 CNL4N5tAaCr79KSpMsH1eoS7hAAPmuichKg2PApVqhs7 Processed in progress for 712ms  missing chunks for 691ms Chunks:(✔✔✔✔))
  | 1701927 9Sz3eCx9sVQX3YsVddgGN3e9jfQxwpcnEXqEPExDrJq1 Processed in progress for 729ms  missing chunks for 703ms Chunks:(✔✔✔✔))
  | 1701926 5yDAJC5fAvjbESQcu43qcgUAYHNKPR2Si1aaZQAJ3z3o Processed in progress for 732ms  missing chunks for 705ms Chunks:(✔✔X✔))
  | 1701925 9RnE1DJXpsueVgH4e9NGduCbkBGyPvt864eXqauSWww8 Processed in progress for 689ms  missing chunks for 662ms Chunks:(✔✔✔✔))
  | 1701924 DQit7SwveVo5aLACq3row72Sf4jbEZcfNwqQbdKa2n3Q Processed in progress for 843ms  missing chunks for 823ms Chunks:(✔✔✔✔))
  | 1701923 DUcqgsUTn34oGwNzz7ANRw7rh5x7tYkBE7crHpGbtP8x Processed in progress for 612ms  missing chunks for 578ms Chunks:(✔✔✔✔))
  | 1701922 AVQ9VPWzZ2Tby8u21AZBZS1sGRYsjw7pk9EwzCzp4hG4 Processed in progress for 634ms  missing chunks for 616ms Chunks:(✔✔✔✔))
  | 1701920 4hjjh4w2TnXyz2vWd9DjVwVquCFuxYnYWRwRSNkoSNVe Processed in progress for 796ms  missing chunks for 763ms Chunks:(✔✔X✔))
  | 1701919 Hi9d6F7LZ7sa3H4xG1SSQCfigKV6SXLywnzquFvQjwGj Processed in progress for 565ms  missing chunks for 548ms Chunks:(✔✔✔✔))
  | 1701918 9PmrE25ko4zjGwnzQnnVWuxoTwFBKET47LsfuYcDpuhh Processed in progress for 792ms  missing chunks for 770ms Chunks:(✔✔✔✔))
  | 1701917 7haLVyP1pkL7YwPSPFWTEmAW77r7eWvoygbg5GaC8JwD Processed in progress for 794ms  missing chunks for 784ms Chunks:(✔✔✔✔))
  | 1701916 GgM8SDMG2xjW8K8ZMmAjnu7xwH88FYnSFB1FGgyEHsy2 Processed in progress for 607ms  missing chunks for 587ms Chunks:(✔✔✔✔))
  | 1701915 G8afuJjmnn3djS9nW9Cnjy3LSk7U34V5LYf9cf9Zhqab Processed in progress for 756ms  missing chunks for 734ms Chunks:(✔✔✔✔))
  | 1701914 GTYLu8kVhxYRxJDLzDEhZaeHQmZCBmGEm6maDpCxC278 Processed in progress for 727ms  missing chunks for 691ms Chunks:(✔✔✔✔))
  | 1701913 GDK8P23b8G75FQRE6gUrs2kAfUkmHXDjHctZCb2hJ4P3 Processed in progress for 694ms  missing chunks for 682ms Chunks:(✔✔✔✔))
  | 1701912 D4wnBUG1Ufn9qz3UkLz1w48xK1g8h8KHvij5NRQRqE3g Processed in progress for 706ms orphan for 78ms missing chunks for 609ms Chunks:(✔✔✔✔))
  | 1701911 75EKqbD5tXpG4mJobjHx1bQvUWesM8PEZiuyHBdDSDFs Processed in progress for 632ms  missing chunks for 600ms Chunks:(✔✔X✔))
  | 1701910 583E1TRKRrkn4rjYPUYPkLf6tzPw71P78mqAuwereJrK Processed in progress for 809ms  missing chunks for 783ms Chunks:(✔✔✔X))
  | 1701909 DvvPHb7R7Yf36YLjqfdedpEc4QveNRgjoQHLBxrN8N9Y Processed in progress for 672ms  missing chunks for 657ms Chunks:(✔✔✔✔))
  | 1701907 Ar8cXDmsSTMeJCr5AYWN221mPCYhc3rTqRSZSzgP3cuq Processed in progress for 716ms  missing chunks for 695ms Chunks:(✔✔XX))
  | 1701906 AmQpac51keTm9v69WVJo1cUxbcrDozgCoaEn6JVSzLey Processed in progress for 649ms  missing chunks for 628ms Chunks:(✔✔XX))
  | 1701904 HMf3k1MsmD9nwW575pS8YCQxEPGU4L16EomAyM5ZGdyR Processed in progress for 871ms  missing chunks for 859ms Chunks:(✔✔✔✔))
  | 1701903 CHXp3mVVCNDTfVuar38dX18tSpwenKYfikLajfBo7jDy Processed in progress for 769ms  missing chunks for 748ms Chunks:(✔✔✔X))
  | 1701902 F8UmhtZWpatEcjfbQmtk9kb4Y1L38Kn7s3cP3oqn8VGE Processed in progress for 625ms  missing chunks for 602ms Chunks:(✔✔✔X))
  | 1701901 4uU3XrZHk5hXECvKNqrvAz7rUqycfEtxZ2qPigQoVoCE Processed in progress for 480ms orphan for 125ms missing chunks for 326ms Chunks:(✔✔✔✔))
  | 1701900 B1dn2KuJuciqsgiF2j4oSDAKfYy1CQgELJBkL5dMxc5p Processed in progress for 633ms  missing chunks for 605ms Chunks:(✔✔XX))
  | 1701898 FjEFpguz3ahnP1U5UxpF5ENoKGkBJGFWfC3F5Edoz9k Processed in progress for 936ms  missing chunks for 922ms Chunks:(✔✔✔✔))
  | 1701897 7xbGEGZswfGeGVTmbyKkvv2Gz9iQMGqauZLB9WKLK2xB Processed in progress for 932ms  missing chunks for 918ms Chunks:(✔✔✔✔))
  | 1701896 Ec4TQF5a9Vrd9K2rwRyLMtQjo8X39YaMErjHpvyuFDAv Processed in progress for 666ms  missing chunks for 642ms Chunks:(✔✔✔✔))
  | 1701895 4xC9Ri4cktnpM6KdzDftELBs7Y9dRFiYsU4VKeW3udHv Processed in progress for 700ms  missing chunks for 679ms Chunks:(✔✔✔X))
  | 1701894 5LYmNG2CLsCMffpraDHumYu6kEiWh6qQKKp8NMNkZEWp Processed in progress for 694ms  missing chunks for 676ms Chunks:(✔✔✔✔))
  | 1701893 8C7ddW6Rgrui9JJo65e2kdKaQqoSoeEjKsR6iyK7MLWd Processed in progress for 702ms  missing chunks for 692ms Chunks:(✔✔✔✔))
  | 1701892 4ArdUTeKMiVGP9WYkdDXGErxWXUyp7ErdfcriENi9DJi Processed in progress for 790ms  missing chunks for 773ms Chunks:(✔✔✔✔))
  | 1701891 GPSV7km4cG2Af6z4cW93DrHAUbZSsivkuGig7qb2Kvvo Processed in progress for 672ms  missing chunks for 649ms Chunks:(✔✔✔X))
  | 1701890 BSC2pRGwmzqzRNAoE3s85VRQrBrobc98tyrei2124eie Processed in progress for 628ms  missing chunks for 609ms Chunks:(✔✔✔✔))
  | 1701889 9XD1gHzeyMaWv6NZy9PdjEhft7odDNLJLFZpnvNDeJG5 Processed in progress for 833ms  missing chunks for 811ms Chunks:(✔✔✔✔))
  | 1701888 3wPfCKHYmNiFC99CzqiDwUPnHm3EuRu8XqhWDTuuTmUq Processed in progress for 681ms  missing chunks for 657ms Chunks:(✔✔XX))
  | 1701887 GdUqskbY7tXWAHo52BsyDmmuhYsdDS31fzSeZD569PJA Processed in progress for 594ms  missing chunks for 567ms Chunks:(✔X✔✔))
  | 1701886 E8j2kVtkYxn9JbXS61XEV4xNMCirRNHKTnTbCmHMiVdS Processed in progress for 924ms  missing chunks for 913ms Chunks:(✔✔✔✔))
  | 2022-08-03T14:23:29.350372Z  WARN telemetry: Failed to send telemetry data err=Timeout endpoint="https://explorer.shardnet.near.org/api/nodes"
  | 2022-08-03T14:23:39.351584Z  INFO stats: # 1701941 5mCgz4TUYe46X7EvYHXc9bxfhef7kqp2QrqtwxeiVexL Validator \| 100 validators 30 peers ⬇ 1.14 MB/s ⬆ 1.68 MB/s 0.50 bps 1.33 Tgas/s CPU: 148%, Mem: 3.94 GB
  | 2022-08-03T14:23:39.354628Z DEBUG stats: EpochId(`7P2rGDXpGDN4TZE5YMiFk1519AvUHhzuatMGw8QhCrdo`) Orphans: 0 With missing chunks: 1 In processing 0
  | 1701943 4KcNnevoLij3LoFpFtTStF6Ti7uzBVVYf4saPUBJ8GjN WaitingForChunks in progress for 130ms  missing chunks for 121ms Chunks:(✔⬇X✔))
  | 1701941 5mCgz4TUYe46X7EvYHXc9bxfhef7kqp2QrqtwxeiVexL Processed in progress for 723ms  missing chunks for 701ms Chunks:(✔✔✔X))
  | 1701940 8JsRNJ1LmHBCPAxbPczT7pHEGQ4eCMpBRvRaCMk3cvu8 Processed in progress for 702ms  missing chunks for 690ms Chunks:(✔✔✔✔))
  | 1701939 28baXDJx7EU7u293MrywePcVmmCaBuL7o4zgAh1QBExt Processed in progress for 830ms  missing chunks for 819ms Chunks:(✔✔✔✔))
  | 1701938 AZqUeBfMPWueii9hjn1NwrgcwvMciDV2Fm6KQtA1xajN Processed in progress for 535ms  missing chunks for 519ms Chunks:(✔✔✔✔))
  | 1701937 Hb8J4o1UGneS56KZuUEEVS8fc5oRRAv2b1CXYRJUaVkv Processed in progress for 711ms  missing chunks for 683ms Chunks:(✔✔✔X))
  | 1701936 4iPf9L53jnp7kPwu1m7GnV9uZjDeBCoNBWDAiCTj2EMV Processed in progress for 564ms  missing chunks for 538ms Chunks:(✔✔✔X))
  | 1701935 BruUGb8cLUXiumQrkgAs3phWPiRqUWuAfovsr1FAPgb4 Processed in progress for 795ms  missing chunks for 768ms Chunks:(✔✔✔✔))
  | 1701934 HJTzxwyMGyqPDzacMWCQPEUBvpRP9EGVNunzaM7L7HFs Processed in progress for 719ms  missing chunks for 698ms Chunks:(✔✔✔✔))
  | 1701933 5vPaUwCr6yM7iPCVSaH9vqctnQyAvjXn31A2F3NcKmPW Processed in progress for 558ms  missing chunks for 543ms Chunks:(✔✔✔✔))
  | 1701931 23Go54ehEUq5cNvw4VkBLdV8GAND5sdHUbYUvwfPn2L4 Processed in progress for 574ms  missing chunks for 552ms Chunks:(✔✔X✔))
  | 1701930 GBfrRp7aw7NATiGAMomMe5uVSSr2pxE7gNsHYKzkBiX5 Processed in progress for 694ms  missing chunks for 668ms Chunks:(✔✔XX))
  | 1701929 3aRGM64b6RxP81kzMaVL8nFnwBAZq5BNY3oXnEWLVxGH Processed in progress for 810ms  missing chunks for 784ms Chunks:(✔✔✔✔))
  | 1701928 CNL4N5tAaCr79KSpMsH1eoS7hAAPmuichKg2PApVqhs7 Processed in progress for 712ms  missing chunks for 691ms Chunks:(✔✔✔✔))
  | 1701927 9Sz3eCx9sVQX3YsVddgGN3e9jfQxwpcnEXqEPExDrJq1 Processed in progress for 729ms  missing chunks for 703ms Chunks:(✔✔✔✔))
  | 1701926 5yDAJC5fAvjbESQcu43qcgUAYHNKPR2Si1aaZQAJ3z3o Processed in progress for 732ms  missing chunks for 705ms Chunks:(✔✔X✔))
  | 1701925 9RnE1DJXpsueVgH4e9NGduCbkBGyPvt864eXqauSWww8 Processed in progress for 689ms  missing chunks for 662ms Chunks:(✔✔✔✔))
  | 1701924 DQit7SwveVo5aLACq3row72Sf4jbEZcfNwqQbdKa2n3Q Processed in progress for 843ms  missing chunks for 823ms Chunks:(✔✔✔✔))
  | 1701923 DUcqgsUTn34oGwNzz7ANRw7rh5x7tYkBE7crHpGbtP8x Processed in progress for 612ms  missing chunks for 578ms Chunks:(✔✔✔✔))
  | 1701922 AVQ9VPWzZ2Tby8u21AZBZS1sGRYsjw7pk9EwzCzp4hG4 Processed in progress for 634ms  missing chunks for 616ms Chunks:(✔✔✔✔))
  | 1701920 4hjjh4w2TnXyz2vWd9DjVwVquCFuxYnYWRwRSNkoSNVe Processed in progress for 796ms  missing chunks for 763ms Chunks:(✔✔X✔))
  | 1701919 Hi9d6F7LZ7sa3H4xG1SSQCfigKV6SXLywnzquFvQjwGj Processed in progress for 565ms  missing chunks for 548ms Chunks:(✔✔✔✔))
  | 1701918 9PmrE25ko4zjGwnzQnnVWuxoTwFBKET47LsfuYcDpuhh Processed in progress for 792ms  missing chunks for 770ms Chunks:(✔✔✔✔))
  | 1701917 7haLVyP1pkL7YwPSPFWTEmAW77r7eWvoygbg5GaC8JwD Processed in progress for 794ms  missing chunks for 784ms Chunks:(✔✔✔✔))
  | 1701916 GgM8SDMG2xjW8K8ZMmAjnu7xwH88FYnSFB1FGgyEHsy2 Processed in progress for 607ms  missing chunks for 587ms Chunks:(✔✔✔✔))
  | 1701915 G8afuJjmnn3djS9nW9Cnjy3LSk7U34V5LYf9cf9Zhqab Processed in progress for 756ms  missing chunks for 734ms Chunks:(✔✔✔✔))
  | 1701914 GTYLu8kVhxYRxJDLzDEhZaeHQmZCBmGEm6maDpCxC278 Processed in progress for 727ms  missing chunks for 691ms Chunks:(✔✔✔✔))
  | 1701913 GDK8P23b8G75FQRE6gUrs2kAfUkmHXDjHctZCb2hJ4P3 Processed in progress for 694ms  missing chunks for 682ms Chunks:(✔✔✔✔))
  | 1701912 D4wnBUG1Ufn9qz3UkLz1w48xK1g8h8KHvij5NRQRqE3g Processed in progress for 706ms orphan for 78ms missing chunks for 609ms Chunks:(✔✔✔✔))
  | 1701911 75EKqbD5tXpG4mJobjHx1bQvUWesM8PEZiuyHBdDSDFs Processed in progress for 632ms  missing chunks for 600ms Chunks:(✔✔X✔))
  | 1701910 583E1TRKRrkn4rjYPUYPkLf6tzPw71P78mqAuwereJrK Processed in progress for 809ms  missing chunks for 783ms Chunks:(✔✔✔X))
  | 1701909 DvvPHb7R7Yf36YLjqfdedpEc4QveNRgjoQHLBxrN8N9Y Processed in progress for 672ms  missing chunks for 657ms Chunks:(✔✔✔✔))
  | 1701907 Ar8cXDmsSTMeJCr5AYWN221mPCYhc3rTqRSZSzgP3cuq Processed in progress for 716ms  missing chunks for 695ms Chunks:(✔✔XX))
  | 1701906 AmQpac51keTm9v69WVJo1cUxbcrDozgCoaEn6JVSzLey Processed in progress for 649ms  missing chunks for 628ms Chunks:(✔✔XX))
  | 1701904 HMf3k1MsmD9nwW575pS8YCQxEPGU4L16EomAyM5ZGdyR Processed in progress for 871ms  missing chunks for 859ms Chunks:(✔✔✔✔))
  | 1701903 CHXp3mVVCNDTfVuar38dX18tSpwenKYfikLajfBo7jDy Processed in progress for 769ms  missing chunks for 748ms Chunks:(✔✔✔X))
  | 1701902 F8UmhtZWpatEcjfbQmtk9kb4Y1L38Kn7s3cP3oqn8VGE Processed in progress for 625ms  missing chunks for 602ms Chunks:(✔✔✔X))
  | 1701901 4uU3XrZHk5hXECvKNqrvAz7rUqycfEtxZ2qPigQoVoCE Processed in progress for 480ms orphan for 125ms missing chunks for 326ms Chunks:(✔✔✔✔))
  | 1701900 B1dn2KuJuciqsgiF2j4oSDAKfYy1CQgELJBkL5dMxc5p Processed in progress for 633ms  missing chunks for 605ms Chunks:(✔✔XX))
  | 1701898 FjEFpguz3ahnP1U5UxpF5ENoKGkBJGFWfC3F5Edoz9k Processed in progress for 936ms  missing chunks for 922ms Chunks:(✔✔✔✔))
  | 1701897 7xbGEGZswfGeGVTmbyKkvv2Gz9iQMGqauZLB9WKLK2xB Processed in progress for 932ms  missing chunks for 918ms Chunks:(✔✔✔✔))
  | 1701896 Ec4TQF5a9Vrd9K2rwRyLMtQjo8X39YaMErjHpvyuFDAv Processed in progress for 666ms  missing chunks for 642ms Chunks:(✔✔✔✔))
  | 1701895 4xC9Ri4cktnpM6KdzDftELBs7Y9dRFiYsU4VKeW3udHv Processed in progress for 700ms  missing chunks for 679ms Chunks:(✔✔✔X))
  | 1701894 5LYmNG2CLsCMffpraDHumYu6kEiWh6qQKKp8NMNkZEWp Processed in progress for 694ms  missing chunks for 676ms Chunks:(✔✔✔✔))
  | 1701893 8C7ddW6Rgrui9JJo65e2kdKaQqoSoeEjKsR6iyK7MLWd Processed in progress for 702ms  missing chunks for 692ms Chunks:(✔✔✔✔))
  | 1701892 4ArdUTeKMiVGP9WYkdDXGErxWXUyp7ErdfcriENi9DJi Processed in progress for 790ms  missing chunks for 773ms Chunks:(✔✔✔✔))
  | 1701891 GPSV7km4cG2Af6z4cW93DrHAUbZSsivkuGig7qb2Kvvo Processed in progress for 672ms  missing chunks for 649ms Chunks:(✔✔✔X))

And the count of produced blocks goes down, while count of produced chunks goes up:

Is it expected behaviour, because of not all my peers updated to latest version?

[joesixpack]

(one of the PRs is near/nearcore#7322)

Seemed to get better for a bit, but now it's back to the usual sad story.

IMO, the no. 1 problem is the validators that got lucky or game the network for 70%+ uptime are exploiting the location factor. Which is not illustrative of how an actual decentralized mainnet would operate.