search

paritytech / polkadot-sdk

The Parity Polkadot Blockchain SDK
https://polkadot.network/
1.87k stars 685 forks source link

new validator missing all p/v points #4914

Closed tugytur closed 4 months ago

tugytur commented 4 months ago

This issue is with polkadot 1.13.0-d5160c1d567 We have a new Polkadot validator that joined the active set for the first time today.

This one is configured just like all the others, incl. Kusama that don't have any issue and always perform as they should.

It's missing so far all p/v points. But Relay chain blocks get produced. Incoming p2p connection works, we have constantly new incoming connections. The validator has the same hardware specs as the other ones in the active set that work fine. Node identity key is the same from when the session keys were generated, which was done when the validator was fully in sync.

I tried to restore the db from backup for both paritydb and rocksdb, didn't fix it. Then I tried both with warp sync and also no points. When I added --no-private-ip it was producing blocks for the remaining time of that epoch but when the new epoch started it started missing all blocks again.

occasionally I have the following error log: 

Some network error occurred when fetching erasure chunk origin=Public(3ab47a4988af07adb94bb14335a0a3699c2afd78a3c96697f0fd6e94cb96e96d (12KyPeFZ...)) relay_parent=0x953a6dc4b2c529278b0f504c703a9ca95c5723305e177eed2b3304076cd121ad group_index=GroupIndex(42) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xa0e6a7c67d6b721dca7ff271bb40b3dac72e6af7cc8f836a06f726998dbf5c06 err=NotConnected traceID=213874110491445057862163958590655738842

This is the systemd file which works for all the other validators:

#
# Ansible managed
#
[Unit]
Description=validator polkadot node
Wants=network-online.target
After=syslog.target network-online.target

[Service]
User=blockchain
Group= blockchain
ExecStart=/usr/local/bin/polkadot \
        --name="Amforc 3" \
        --database=paritydb \
        --state-pruning=1000 \
        --base-path=/var/blockchain/data \
        --keystore-path=/var/blockchain/keystore \
        --chain=polkadot \
        --no-mdns \
        --no-hardware-benchmarks \
        --prometheus-port=9615 \
        --rpc-port=9944 \
        --in-peers-light=0 \
        --telemetry-url='wss://telemetry-backend.w3f.community/submit 1' \
        --validator \
        --no-private-ip \
        --listen-addr=/ip4/0.0.0.0/tcp/30333 \
        --sync=warp

Restart=always
TimeoutStopSec=60
RestartSec=60
LimitNOFILE=500000
ExecStartPre=/bin/sleep 60
CapabilityBoundingSet=
LockPersonality=true
NoNewPrivileges=true
PrivateDevices=true
PrivateMounts=true
PrivateTmp=true
PrivateUsers=true
ProtectClock=true
ProtectControlGroups=true
ProtectHostname=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectSystem=strict
ReadWritePaths=/var/blockchain
RemoveIPC=true
RestrictAddressFamilies=AF_INET AF_INET6 AF_NETLINK AF_UNIX
RestrictNamespaces=false
RestrictSUIDSGID=true
SystemCallArchitectures=native
SystemCallFilter=@system-service
SystemCallFilter=landlock_add_rule landlock_create_ruleset landlock_restrict_self seccomp mount umount2
SystemCallFilter=~@clock @module @reboot @swap @privileged
SystemCallFilter=pivot_root
UMask=0027

[Install]
WantedBy=multi-user.target

The validator has a public IP assigned, I also additionally specified the public IP with --public-addr and --listen-addr, which didn't help either.

The only difference this validator has is that it the stash account joined the active set for the first time ever.

tugytur commented 4 months ago

Two minutes after posting this issue the validator started producing blocks..

alexggh commented 4 months ago

Can you provide some timelines:

  1. Session your validator joined for the first time.
  2. How long it took for the validator to start producing p/v points.
  3. Link to https://apps.turboflakes.io/#/dashboard for your validator.
  4. All logs you have for that period.
  5. The network id of your validator, you can find in the following log line Local node identity is:

One possible root-cause, is this one https://github.com/paritytech/polkadot-sdk/issues/4324, where validators entering the active set have to pre compile the PVFs for all parachains, so they might fall behind on backing work, but the validator should catch-up and recover by itself in 5-10 minutes. Restarting won't help because, all pvfs are recompiled when you restart your validator.

tugytur commented 4 months ago

@alexggh sure,

1

8976

2

First time troubleshooting various things after like 2 hours, when I set the --no-private-ip flag, but then the next session it started missing all again. Then I tried various things again after ~2-3 hours it started producing blocks and has been good since then.

3

https://apps.turboflakes.io/?chain=polkadot&app=onet#/validator/5DKbgoyHA4LaP7rMoa6N7iA6iDFRVrnjJd4q4Bsuu6cb4bmz

4

Mostly syncing logs expect like once a minute this one:

Some network error occurred when fetching erasure chunk origin=Public(94176052dce88ba60b48e3db95facaec024120bc8bc6590322f04380e5a2c817 (14MB2okG...)) relay_parent=0xce27768c3d83a8dec148a1a344404a03002960792be9c1f5ce1af1b72959490e group_index=GroupIndex(53) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x58a6a01ca06f6e0240d868690c57586f9dde265bd2f4e2c18a1ed126ace0f498 err=NotConnected traceID=117837232361166824707812005931291990127

And these after restarting the validator various times:

2024-06-30 21:07:16 👮‍♀️ Running in Secure Validator Mode. It is highly recommended that you operate according to our security guidelines.
More information: https://wiki.polkadot.network/docs/maintain-guides-secure-validator#secure-validator-mode
2024-06-30 21:07:17 🔍 Discovered new external address for our node: /ip4/91.90.166.5/tcp/30333/p2p/12D3KooWHRd8ANXyUvkRNJiXsSUzB2pSaE7QvVATdsTNRCjRvdiP
2024-06-30 21:07:18 Received msg before first active leaves update. This is not expected - message will be dropped. msg=DetermineUndisputedChain { base: (21447704, 0xcdb616525111c91bb49c31548048108e28200d3a915286b9c6c0d0e6096e2414), block_descriptions: [], tx: Sender { complete: false } }
2024-06-30 21:07:18 Call to `DetermineUndisputedChain` failed error=DetermineUndisputedChainCanceled(Canceled)
2024-06-30 21:07:21 🥩 BEEFY pallet available: block 21447794 beefy genesis 20351234
2024-06-30 21:07:21 Candidate included without being backed? candidate_hash=0xb79b4acd67157f83cbb3c49d130eb7e28c1e3ffdd540880fdac18dfbb0c00b95 traceID=244055046413699434297175036917091055586
2024-06-30 21:07:21 Candidate included without being backed? candidate_hash=0xa433c80f1bc90f077a3c342337605976b06022796c31dfda6ff816c4730e6780 traceID=218262256127453807741680547729600698742
2024-06-30 21:07:21 Candidate included without being backed? candidate_hash=0xb9574c2f716062cc589a427f23c098429691ab3db703c4442b5dd64bca29b296 traceID=246360454268843811738193144184400025666
2024-06-30 21:07:21 Candidate included without being backed? candidate_hash=0x638edba914dec4b3a07a6eeaf23a3f767fcc3b1b4c3bcb26629552921010d84e traceID=132335332980340292192966984847807102838
2024-06-30 21:07:21 Candidate included without being backed? candidate_hash=0x72be319301014a016187d6a5b58b46071d0f44f4149b8f05629a75a4640546af traceID=152519533407523538705398456716878366215
2024-06-30 21:07:21 Candidate included without being backed? candidate_hash=0x2877e7e4a15c55ff8bc07b7bfcb806618bc3c5533023bfb2e61df9e5766aa609 traceID=53791706508140490063866383821251806817
2024-06-30 21:07:21 Candidate included without being backed? candidate_hash=0xf8f0adcadaea06e0086ac8add1d9d134fbaaa0b1bdef920392d69afd01438218 traceID=330898219129408407981090142525293449524
2024-06-30 21:07:21 Candidate included without being backed? candidate_hash=0xc7b10c4025cf6cece39571463854e7259e6cbfa3707fe794e43bf048fa9301a0 traceID=265435656176378275324531081463553386277
2024-06-30 21:07:21 🥩 Parent of header number 21447456 not found. BEEFY gadget waiting for header sync to finish ...
2024-06-30 21:07:21 ⏩ Block history, #2075904 (8 peers), best: #21447798 (0x8aa9…b594), finalized #21447794 (0x8e92…02a5), ⬇ 1.6MiB/s ⬆ 188.4kiB/s
2024-06-30 21:07:24 🏆 Imported #21447799 (0x8aa9…b594 → 0x86e1…e326)
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(a4c1b6ec21b490241d3db57f8c42af5565c276376fab107127665b492295d137 (14j2Q6yy...)) relay_parent=0x0de44d17c6edbb8e53ec9b9db0a7d9d320df780e18a4a3f3bfa28f95b5ac64f7 group_index=GroupIndex(51) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x1b466918a416bba67bbb50c47de8f5c2be04133709cfd4221a6552f5ce5a0f45 err=Network(DialFailure) traceID=36254748271557474402478276741368968642
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(288616b7548602fcfbefa7ee5bd4290940d0711b2e29fd26d8874f167297a366 (1v8kPUzT...)) relay_parent=0x8aa95da893954dedd55feaa06aefd037f08274ad01bd14979c30530f428db594 group_index=GroupIndex(16) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xddfe2f9e5730173f9ad8a7e8bcbc556546c59f2434c2ac414090e1396efc3b92 err=Network(DialFailure) traceID=295079196288818637093193730210167280997
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(a4c1b6ec21b490241d3db57f8c42af5565c276376fab107127665b492295d137 (14j2Q6yy...)) relay_parent=0x21fd5ae0cafae248b0127c35430710ed1f8b589b8999ae47d6773f1ddd47640b group_index=GroupIndex(51) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xce1e90cde91fcd6b702675b947f396ebe3a761da91c86ed089a549d40d95fac8 err=Network(DialFailure) traceID=273979673018353409967563759089802778347
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(288616b7548602fcfbefa7ee5bd4290940d0711b2e29fd26d8874f167297a366 (1v8kPUzT...)) relay_parent=0x4023781a0abb9c617153737f6f7b697b48c38c446fef307064dd76dc98e9d52b group_index=GroupIndex(16) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x8325bc21d335709a78dfd76690f5f1ac081f6cc4f1809d65d04fff11fb96e7a5 err=Network(DialFailure) traceID=174324798204490559352715166569698029996
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(7433f7efa4fd520b225629edd3bb4b0a1c8a79ba00198d81b2838fcb883ed464 (13dMzZkg...)) relay_parent=0x8aa95da893954dedd55feaa06aefd037f08274ad01bd14979c30530f428db594 group_index=GroupIndex(11) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x529efe76398e7331025066f44dfe615e5ae2acb5e8e813869ad7d2bc17ba8cef err=NotConnected traceID=109822239656786965553304198269785497950
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(8a2a3d2aa8595a2b4e60b5e89ebdd0ea864ca2a68d640c67334cca91d6d74e4f (148AA2gf...)) relay_parent=0x0de44d17c6edbb8e53ec9b9db0a7d9d320df780e18a4a3f3bfa28f95b5ac64f7 group_index=GroupIndex(56) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xbbfce123ae335f5d2762fcc43e9568653b69be2c4067334677303edc196b66c3 err=NotConnected traceID=249878660389189051689414377405261637733
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(ac8f055f6c7f695cad037a0a0ed362e1af946de452ae094ca6bbda79a962f36a (14uFitjs...)) relay_parent=0x0de44d17c6edbb8e53ec9b9db0a7d9d320df780e18a4a3f3bfa28f95b5ac64f7 group_index=GroupIndex(33) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x3c1b8e4e9acdb12def48499ec9a577fc7ce6078eb114afc82155c09219e641d1 err=NotConnected traceID=79896758092145145381738482752108722172
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(c2da7fa88bd05ef1c881c39e227c1dd83a15b3f6b731fa50f012c9115eef9058 (15QVCShT...)) relay_parent=0x0de44d17c6edbb8e53ec9b9db0a7d9d320df780e18a4a3f3bfa28f95b5ac64f7 group_index=GroupIndex(47) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x555b6913dc011d65c2d53a55afcd9886c534969457f823ce79e99f8e68c68eb8 err=NotConnected traceID=113459009882276039619740679994604361862
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(8a2a3d2aa8595a2b4e60b5e89ebdd0ea864ca2a68d640c67334cca91d6d74e4f (148AA2gf...)) relay_parent=0x4023781a0abb9c617153737f6f7b697b48c38c446fef307064dd76dc98e9d52b group_index=GroupIndex(56) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x01a04faabf566908ba9a874c0e4ce65864cfb8d2c2a1e3c10d7971bbbf2443ca err=NotConnected traceID=2161611331512904706234197744355698264
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(74cbe6e914ea9faeb93af966ba5e96936a08948548f2924b7a143169e52e730b (13e98KSH...)) relay_parent=0x0de44d17c6edbb8e53ec9b9db0a7d9d320df780e18a4a3f3bfa28f95b5ac64f7 group_index=GroupIndex(54) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x6ff916d8cdfb8ad0b3e710441185db077730384145c936c9cfda3f7f6e4c616c err=NotConnected traceID=148837652839943740320232686613120277255
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(76af898d3856bbaca4a9954568509af53603d1a5afcc1e555c05328c2e874e44 (13gcoAJW...)) relay_parent=0x8aa95da893954dedd55feaa06aefd037f08274ad01bd14979c30530f428db594 group_index=GroupIndex(16) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xddfe2f9e5730173f9ad8a7e8bcbc556546c59f2434c2ac414090e1396efc3b92 err=NotConnected traceID=295079196288818637093193730210167280997
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(0849872804f7d5257fe7fa5a4293df4f9e4f900d2046fb5289cb48c97963155a (1BsE5PXs...)) relay_parent=0x0de44d17c6edbb8e53ec9b9db0a7d9d320df780e18a4a3f3bfa28f95b5ac64f7 group_index=GroupIndex(2) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x965d69a34d3bcb6fbf4efb1874ff36d232388e7c12640bbf4826e625410e3e26 err=NotConnected traceID=199869225566682626236876617134460843730
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(7e595af665c821e4a7f584efe325c58f9f900f2c22f856723dd959f25a97f17a (13rfaVsY...)) relay_parent=0x0de44d17c6edbb8e53ec9b9db0a7d9d320df780e18a4a3f3bfa28f95b5ac64f7 group_index=GroupIndex(17) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x7d3997e9102fdd14931fdd31629f1f127f601267763faf39550b51cb8c6a2f2d err=NotConnected traceID=166452541503072610206819235745115348754
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(8a2a3d2aa8595a2b4e60b5e89ebdd0ea864ca2a68d640c67334cca91d6d74e4f (148AA2gf...)) relay_parent=0x21fd5ae0cafae248b0127c35430710ed1f8b589b8999ae47d6773f1ddd47640b group_index=GroupIndex(56) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x1ffdf927b1b78b6e03a5e63cab91c0b1906dd66bbd9399cd7dc517ed43e61583 err=NotConnected traceID=42524772439432089373994354047282430129
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(8c33cececa85ff1b31709f042ee84fd70463bee65651bce9824e44db8345175a (14Aq6TE3...)) relay_parent=0x0de44d17c6edbb8e53ec9b9db0a7d9d320df780e18a4a3f3bfa28f95b5ac64f7 group_index=GroupIndex(14) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x4d6c3adbed97ddf0be52ef89259c73bace12913cd74e2d1ae2f7d34df7eb7b74 err=NotConnected traceID=102912517540416429228595128534497522618
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(7433f7efa4fd520b225629edd3bb4b0a1c8a79ba00198d81b2838fcb883ed464 (13dMzZkg...)) relay_parent=0x4023781a0abb9c617153737f6f7b697b48c38c446fef307064dd76dc98e9d52b group_index=GroupIndex(11) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x78027e9f3c75da06bc867e019a7a160e65efe7a9e4af3d990caf51996ce4519a err=NotConnected traceID=159520312287506448792551032586652751374
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(ac8f055f6c7f695cad037a0a0ed362e1af946de452ae094ca6bbda79a962f36a (14uFitjs...)) relay_parent=0x21fd5ae0cafae248b0127c35430710ed1f8b589b8999ae47d6773f1ddd47640b group_index=GroupIndex(33) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xfdece39f38d0d305bdae82cad23218ea1a3d41583f3855e8c93211bf34d00cf5 err=NotConnected traceID=337524681714039417507917075091505158378
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(c2da7fa88bd05ef1c881c39e227c1dd83a15b3f6b731fa50f012c9115eef9058 (15QVCShT...)) relay_parent=0x21fd5ae0cafae248b0127c35430710ed1f8b589b8999ae47d6773f1ddd47640b group_index=GroupIndex(47) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x84486d08358ae98bf23c8bdd3feb3dbcbd9cabf21b03ddb53440af5befaf053a err=NotConnected traceID=175834152250466141135328661071094037948
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(74cbe6e914ea9faeb93af966ba5e96936a08948548f2924b7a143169e52e730b (13e98KSH...)) relay_parent=0x21fd5ae0cafae248b0127c35430710ed1f8b589b8999ae47d6773f1ddd47640b group_index=GroupIndex(54) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x30ecef76c178b31c7c498da2820bced2070ae703208dd4fcb5a11cb1a2371089 err=NotConnected traceID=65033182760985154475077562016713133778
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(8a2a3d2aa8595a2b4e60b5e89ebdd0ea864ca2a68d640c67334cca91d6d74e4f (148AA2gf...)) relay_parent=0x8e92ea91c8dfb19bb314b66220707d022bcdda916b7a7f30812d6092127202a5 group_index=GroupIndex(56) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xedb7e75aee0e9d317b2b235d622598fe0f964a06eb91fda3472e6a7e2658bb02 err=NotConnected traceID=315981917766965104777139897034351155454
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(0849872804f7d5257fe7fa5a4293df4f9e4f900d2046fb5289cb48c97963155a (1BsE5PXs...)) relay_parent=0x21fd5ae0cafae248b0127c35430710ed1f8b589b8999ae47d6773f1ddd47640b group_index=GroupIndex(2) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x9821368f7ecd0b9190c0658eb0e95217a78333787d9bc95b4e91e6c8c438cd50 err=NotConnected traceID=202215107774627694384873476864468406807
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(76af898d3856bbaca4a9954568509af53603d1a5afcc1e555c05328c2e874e44 (13gcoAJW...)) relay_parent=0x4023781a0abb9c617153737f6f7b697b48c38c446fef307064dd76dc98e9d52b group_index=GroupIndex(16) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x8325bc21d335709a78dfd76690f5f1ac081f6cc4f1809d65d04fff11fb96e7a5 err=NotConnected traceID=174324798204490559352715166569698029996
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(7e595af665c821e4a7f584efe325c58f9f900f2c22f856723dd959f25a97f17a (13rfaVsY...)) relay_parent=0x21fd5ae0cafae248b0127c35430710ed1f8b589b8999ae47d6773f1ddd47640b group_index=GroupIndex(17) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x4f6766721b85c11930a9b0b2206d2f1947cd6bf3144ea2afd1cd1061d485d476 err=NotConnected traceID=105545896089745334582248513451747782425
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(5cfac09f57b19f42da147304cdc6bd26ae9fd4e292e53081fe77fb4114882d0d (136utsqs...)) relay_parent=0x8aa95da893954dedd55feaa06aefd037f08274ad01bd14979c30530f428db594 group_index=GroupIndex(11) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x529efe76398e7331025066f44dfe615e5ae2acb5e8e813869ad7d2bc17ba8cef err=NotConnected traceID=109822239656786965553304198269785497950
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(6ab9c89ba84d98e4ef3b3a71ce5773f373624abfb01e6ae5c2920f45d4724d1a (13QwGNc6...)) relay_parent=0x0de44d17c6edbb8e53ec9b9db0a7d9d320df780e18a4a3f3bfa28f95b5ac64f7 group_index=GroupIndex(56) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xbbfce123ae335f5d2762fcc43e9568653b69be2c4067334677303edc196b66c3 err=NotConnected traceID=249878660389189051689414377405261637733
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(e29c2b033f77fce6a102fbb891ea074b60774992a966a353753c66ccb6621661 (1688DPLD...)) relay_parent=0x0de44d17c6edbb8e53ec9b9db0a7d9d320df780e18a4a3f3bfa28f95b5ac64f7 group_index=GroupIndex(47) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x555b6913dc011d65c2d53a55afcd9886c534969457f823ce79e99f8e68c68eb8 err=NotConnected traceID=113459009882276039619740679994604361862
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(6ab9c89ba84d98e4ef3b3a71ce5773f373624abfb01e6ae5c2920f45d4724d1a (13QwGNc6...)) relay_parent=0x4023781a0abb9c617153737f6f7b697b48c38c446fef307064dd76dc98e9d52b group_index=GroupIndex(56) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x01a04faabf566908ba9a874c0e4ce65864cfb8d2c2a1e3c10d7971bbbf2443ca err=NotConnected traceID=2161611331512904706234197744355698264
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(c8fe387494182df44d7c718be506a7ce52316094a9228c24934bf6eb06936d19 (15YY6afA...)) relay_parent=0x0de44d17c6edbb8e53ec9b9db0a7d9d320df780e18a4a3f3bfa28f95b5ac64f7 group_index=GroupIndex(54) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x6ff916d8cdfb8ad0b3e710441185db077730384145c936c9cfda3f7f6e4c616c err=NotConnected traceID=148837652839943740320232686613120277255
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(c6c4e1328a4a6f7bb429aa6aa01813462c689c65dae0244e3bc3d7e6329ae219 (15Vcy4pQ...)) relay_parent=0x0de44d17c6edbb8e53ec9b9db0a7d9d320df780e18a4a3f3bfa28f95b5ac64f7 group_index=GroupIndex(17) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x7d3997e9102fdd14931fdd31629f1f127f601267763faf39550b51cb8c6a2f2d err=NotConnected traceID=166452541503072610206819235745115348754
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(6ab9c89ba84d98e4ef3b3a71ce5773f373624abfb01e6ae5c2920f45d4724d1a (13QwGNc6...)) relay_parent=0x21fd5ae0cafae248b0127c35430710ed1f8b589b8999ae47d6773f1ddd47640b group_index=GroupIndex(56) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x1ffdf927b1b78b6e03a5e63cab91c0b1906dd66bbd9399cd7dc517ed43e61583 err=NotConnected traceID=42524772439432089373994354047282430129
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(94e02c728aa186e3e3d14808b24781d46078618655940cbea7f9964fbbfe676f (14NCgTua...)) relay_parent=0x0de44d17c6edbb8e53ec9b9db0a7d9d320df780e18a4a3f3bfa28f95b5ac64f7 group_index=GroupIndex(14) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x4d6c3adbed97ddf0be52ef89259c73bace12913cd74e2d1ae2f7d34df7eb7b74 err=NotConnected traceID=102912517540416429228595128534497522618
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(5cfac09f57b19f42da147304cdc6bd26ae9fd4e292e53081fe77fb4114882d0d (136utsqs...)) relay_parent=0x4023781a0abb9c617153737f6f7b697b48c38c446fef307064dd76dc98e9d52b group_index=GroupIndex(11) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x78027e9f3c75da06bc867e019a7a160e65efe7a9e4af3d990caf51996ce4519a err=NotConnected traceID=159520312287506448792551032586652751374
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(e29c2b033f77fce6a102fbb891ea074b60774992a966a353753c66ccb6621661 (1688DPLD...)) relay_parent=0x21fd5ae0cafae248b0127c35430710ed1f8b589b8999ae47d6773f1ddd47640b group_index=GroupIndex(47) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x84486d08358ae98bf23c8bdd3feb3dbcbd9cabf21b03ddb53440af5befaf053a err=NotConnected traceID=175834152250466141135328661071094037948
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(c8fe387494182df44d7c718be506a7ce52316094a9228c24934bf6eb06936d19 (15YY6afA...)) relay_parent=0x21fd5ae0cafae248b0127c35430710ed1f8b589b8999ae47d6773f1ddd47640b group_index=GroupIndex(54) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x30ecef76c178b31c7c498da2820bced2070ae703208dd4fcb5a11cb1a2371089 err=NotConnected traceID=65033182760985154475077562016713133778
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(c6c4e1328a4a6f7bb429aa6aa01813462c689c65dae0244e3bc3d7e6329ae219 (15Vcy4pQ...)) relay_parent=0x21fd5ae0cafae248b0127c35430710ed1f8b589b8999ae47d6773f1ddd47640b group_index=GroupIndex(17) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x4f6766721b85c11930a9b0b2206d2f1947cd6bf3144ea2afd1cd1061d485d476 err=NotConnected traceID=105545896089745334582248513451747782425
2024-06-30 21:07:24 Some network error occurred when fetching erasure chunk origin=Public(6ab9c89ba84d98e4ef3b3a71ce5773f373624abfb01e6ae5c2920f45d4724d1a (13QwGNc6...)) relay_parent=0x8e92ea91c8dfb19bb314b66220707d022bcdda916b7a7f30812d6092127202a5 group_index=GroupIndex(56) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xedb7e75aee0e9d317b2b235d622598fe0f964a06eb91fda3472e6a7e2658bb02 err=NotConnected traceID=315981917766965104777139897034351155454
2024-06-30 21:07:25 Node uses the following topology indices index_in_gossip_topology=Some(ValidatorIndex(263)) index_in_parachain_authorities=Some(Active(ValidatorIndex(263)))
🥩 BEEFY gadget waiting for BEEFY pallet to become available...
👮‍♀️ Running in Secure Validator Mode. It is highly recommended that you operate according to our security guidelines.
tps://wiki.polkadot.network/docs/maintain-guides-secure-validator#secure-validator-mode
🔍 Discovered new external address for our node: /ip4/91.90.166.5/tcp/30333/p2p/12D3KooWHRd8ANXyUvkRNJiXsSUzB2pSaE7QvVATdsTNRCjRvdiP
Received msg before first active leaves update. This is not expected - message will be dropped. msg=DetermineUndisputedChain { base: (21447704, 0xcdb616525111c91bb49c31548048108e28200d3a915286b9c6c0d0e6096e2414), block_descriptions: [], tx: Sender { complete: false } }
Call to `DetermineUndisputedChain` failed error=DetermineUndisputedChainCanceled(Canceled)
🥩 BEEFY pallet available: block 21447794 beefy genesis 20351234
Candidate included without being backed? candidate_hash=0xb79b4acd67157f83cbb3c49d130eb7e28c1e3ffdd540880fdac18dfbb0c00b95 traceID=244055046413699434297175036917091055586
Candidate included without being backed? candidate_hash=0xa433c80f1bc90f077a3c342337605976b06022796c31dfda6ff816c4730e6780 traceID=218262256127453807741680547729600698742
Candidate included without being backed? candidate_hash=0xb9574c2f716062cc589a427f23c098429691ab3db703c4442b5dd64bca29b296 traceID=246360454268843811738193144184400025666
Candidate included without being backed? candidate_hash=0x638edba914dec4b3a07a6eeaf23a3f767fcc3b1b4c3bcb26629552921010d84e traceID=132335332980340292192966984847807102838
Candidate included without being backed? candidate_hash=0x72be319301014a016187d6a5b58b46071d0f44f4149b8f05629a75a4640546af traceID=152519533407523538705398456716878366215
Candidate included without being backed? candidate_hash=0x2877e7e4a15c55ff8bc07b7bfcb806618bc3c5533023bfb2e61df9e5766aa609 traceID=53791706508140490063866383821251806817
Candidate included without being backed? candidate_hash=0xf8f0adcadaea06e0086ac8add1d9d134fbaaa0b1bdef920392d69afd01438218 traceID=330898219129408407981090142525293449524
Candidate included without being backed? candidate_hash=0xc7b10c4025cf6cece39571463854e7259e6cbfa3707fe794e43bf048fa9301a0 traceID=265435656176378275324531081463553386277
🥩 Parent of header number 21447456 not found. BEEFY gadget waiting for header sync to finish ...
⏩ Block history, #2075904 (8 peers), best: #21447798 (0x8aa9…b594), finalized #21447794 (0x8e92…02a5), ⬇ 1.6MiB/s ⬆ 188.4kiB/s
🏆 Imported #21447799 (0x8aa9…b594 → 0x86e1…e326)
Some network error occurred when fetching erasure chunk origin=Public(a4c1b6ec21b490241d3db57f8c42af5565c276376fab107127665b492295d137 (14j2Q6yy...)) relay_parent=0x0de44d17c6edbb8e53ec9b9db0a7d9d320df780e18a4a3f3bfa28f95b5ac64f7 group_index=GroupIndex(51) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x1b466918a416bba67bbb50c47de8f5c2be04133709cfd4221a6552f5ce5a0f45 err=Network(DialFailure) traceID=36254748271557474402478276741368968642
Some network error occurred when fetching erasure chunk origin=Public(288616b7548602fcfbefa7ee5bd4290940d0711b2e29fd26d8874f167297a366 (1v8kPUzT...)) relay_parent=0x8aa95da893954dedd55feaa06aefd037f08274ad01bd14979c30530f428db594 group_index=GroupIndex(16) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xddfe2f9e5730173f9ad8a7e8bcbc556546c59f2434c2ac414090e1396efc3b92 err=Network(DialFailure) traceID=295079196288818637093193730210167280997
Some network error occurred when fetching erasure chunk origin=Public(a4c1b6ec21b490241d3db57f8c42af5565c276376fab107127665b492295d137 (14j2Q6yy...)) relay_parent=0x21fd5ae0cafae248b0127c35430710ed1f8b589b8999ae47d6773f1ddd47640b group_index=GroupIndex(51) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xce1e90cde91fcd6b702675b947f396ebe3a761da91c86ed089a549d40d95fac8 err=Network(DialFailure) traceID=273979673018353409967563759089802778347
Some network error occurred when fetching erasure chunk origin=Public(288616b7548602fcfbefa7ee5bd4290940d0711b2e29fd26d8874f167297a366 (1v8kPUzT...)) relay_parent=0x4023781a0abb9c617153737f6f7b697b48c38c446fef307064dd76dc98e9d52b group_index=GroupIndex(16) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x8325bc21d335709a78dfd76690f5f1ac081f6cc4f1809d65d04fff11fb96e7a5 err=Network(DialFailure) traceID=174324798204490559352715166569698029996
Some network error occurred when fetching erasure chunk origin=Public(7433f7efa4fd520b225629edd3bb4b0a1c8a79ba00198d81b2838fcb883ed464 (13dMzZkg...)) relay_parent=0x8aa95da893954dedd55feaa06aefd037f08274ad01bd14979c30530f428db594 group_index=GroupIndex(11) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x529efe76398e7331025066f44dfe615e5ae2acb5e8e813869ad7d2bc17ba8cef err=NotConnected traceID=109822239656786965553304198269785497950
Some network error occurred when fetching erasure chunk origin=Public(8a2a3d2aa8595a2b4e60b5e89ebdd0ea864ca2a68d640c67334cca91d6d74e4f (148AA2gf...)) relay_parent=0x0de44d17c6edbb8e53ec9b9db0a7d9d320df780e18a4a3f3bfa28f95b5ac64f7 group_index=GroupIndex(56) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xbbfce123ae335f5d2762fcc43e9568653b69be2c4067334677303edc196b66c3 err=NotConnected traceID=249878660389189051689414377405261637733
Some network error occurred when fetching erasure chunk origin=Public(ac8f055f6c7f695cad037a0a0ed362e1af946de452ae094ca6bbda79a962f36a (14uFitjs...)) relay_parent=0x0de44d17c6edbb8e53ec9b9db0a7d9d320df780e18a4a3f3bfa28f95b5ac64f7 group_index=GroupIndex(33) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x3c1b8e4e9acdb12def48499ec9a577fc7ce6078eb114afc82155c09219e641d1 err=NotConnected traceID=79896758092145145381738482752108722172
Some network error occurred when fetching erasure chunk origin=Public(c2da7fa88bd05ef1c881c39e227c1dd83a15b3f6b731fa50f012c9115eef9058 (15QVCShT...)) relay_parent=0x0de44d17c6edbb8e53ec9b9db0a7d9d320df780e18a4a3f3bfa28f95b5ac64f7 group_index=GroupIndex(47) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x555b6913dc011d65c2d53a55afcd9886c534969457f823ce79e99f8e68c68eb8 err=NotConnected traceID=113459009882276039619740679994604361862
Some network error occurred when fetching erasure chunk origin=Public(8a2a3d2aa8595a2b4e60b5e89ebdd0ea864ca2a68d640c67334cca91d6d74e4f (148AA2gf...)) relay_parent=0x4023781a0abb9c617153737f6f7b697b48c38c446fef307064dd76dc98e9d52b group_index=GroupIndex(56) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x01a04faabf566908ba9a874c0e4ce65864cfb8d2c2a1e3c10d7971bbbf2443ca err=NotConnected traceID=2161611331512904706234197744355698264
Some network error occurred when fetching erasure chunk origin=Public(74cbe6e914ea9faeb93af966ba5e96936a08948548f2924b7a143169e52e730b (13e98KSH...)) relay_parent=0x0de44d17c6edbb8e53ec9b9db0a7d9d320df780e18a4a3f3bfa28f95b5ac64f7 group_index=GroupIndex(54) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x6ff916d8cdfb8ad0b3e710441185db077730384145c936c9cfda3f7f6e4c616c err=NotConnected traceID=148837652839943740320232686613120277255
Some network error occurred when fetching erasure chunk origin=Public(76af898d3856bbaca4a9954568509af53603d1a5afcc1e555c05328c2e874e44 (13gcoAJW...)) relay_parent=0x8aa95da893954dedd55feaa06aefd037f08274ad01bd14979c30530f428db594 group_index=GroupIndex(16) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xddfe2f9e5730173f9ad8a7e8bcbc556546c59f2434c2ac414090e1396efc3b92 err=NotConnected traceID=295079196288818637093193730210167280997
Some network error occurred when fetching erasure chunk origin=Public(0849872804f7d5257fe7fa5a4293df4f9e4f900d2046fb5289cb48c97963155a (1BsE5PXs...)) relay_parent=0x0de44d17c6edbb8e53ec9b9db0a7d9d320df780e18a4a3f3bfa28f95b5ac64f7 group_index=GroupIndex(2) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x965d69a34d3bcb6fbf4efb1874ff36d232388e7c12640bbf4826e625410e3e26 err=NotConnected traceID=199869225566682626236876617134460843730
Some network error occurred when fetching erasure chunk origin=Public(7e595af665c821e4a7f584efe325c58f9f900f2c22f856723dd959f25a97f17a (13rfaVsY...)) relay_parent=0x0de44d17c6edbb8e53ec9b9db0a7d9d320df780e18a4a3f3bfa28f95b5ac64f7 group_index=GroupIndex(17) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x7d3997e9102fdd14931fdd31629f1f127f601267763faf39550b51cb8c6a2f2d err=NotConnected traceID=166452541503072610206819235745115348754
Some network error occurred when fetching erasure chunk origin=Public(8a2a3d2aa8595a2b4e60b5e89ebdd0ea864ca2a68d640c67334cca91d6d74e4f (148AA2gf...)) relay_parent=0x21fd5ae0cafae248b0127c35430710ed1f8b589b8999ae47d6773f1ddd47640b group_index=GroupIndex(56) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x1ffdf927b1b78b6e03a5e63cab91c0b1906dd66bbd9399cd7dc517ed43e61583 err=NotConnected traceID=42524772439432089373994354047282430129
Some network error occurred when fetching erasure chunk origin=Public(8c33cececa85ff1b31709f042ee84fd70463bee65651bce9824e44db8345175a (14Aq6TE3...)) relay_parent=0x0de44d17c6edbb8e53ec9b9db0a7d9d320df780e18a4a3f3bfa28f95b5ac64f7 group_index=GroupIndex(14) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x4d6c3adbed97ddf0be52ef89259c73bace12913cd74e2d1ae2f7d34df7eb7b74 err=NotConnected traceID=102912517540416429228595128534497522618
Some network error occurred when fetching erasure chunk origin=Public(7433f7efa4fd520b225629edd3bb4b0a1c8a79ba00198d81b2838fcb883ed464 (13dMzZkg...)) relay_parent=0x4023781a0abb9c617153737f6f7b697b48c38c446fef307064dd76dc98e9d52b group_index=GroupIndex(11) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x78027e9f3c75da06bc867e019a7a160e65efe7a9e4af3d990caf51996ce4519a err=NotConnected traceID=159520312287506448792551032586652751374
Some network error occurred when fetching erasure chunk origin=Public(ac8f055f6c7f695cad037a0a0ed362e1af946de452ae094ca6bbda79a962f36a (14uFitjs...)) relay_parent=0x21fd5ae0cafae248b0127c35430710ed1f8b589b8999ae47d6773f1ddd47640b group_index=GroupIndex(33) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xfdece39f38d0d305bdae82cad23218ea1a3d41583f3855e8c93211bf34d00cf5 err=NotConnected traceID=337524681714039417507917075091505158378
Some network error occurred when fetching erasure chunk origin=Public(c2da7fa88bd05ef1c881c39e227c1dd83a15b3f6b731fa50f012c9115eef9058 (15QVCShT...)) relay_parent=0x21fd5ae0cafae248b0127c35430710ed1f8b589b8999ae47d6773f1ddd47640b group_index=GroupIndex(47) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x84486d08358ae98bf23c8bdd3feb3dbcbd9cabf21b03ddb53440af5befaf053a err=NotConnected traceID=175834152250466141135328661071094037948
Some network error occurred when fetching erasure chunk origin=Public(74cbe6e914ea9faeb93af966ba5e96936a08948548f2924b7a143169e52e730b (13e98KSH...)) relay_parent=0x21fd5ae0cafae248b0127c35430710ed1f8b589b8999ae47d6773f1ddd47640b group_index=GroupIndex(54) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x30ecef76c178b31c7c498da2820bced2070ae703208dd4fcb5a11cb1a2371089 err=NotConnected traceID=65033182760985154475077562016713133778
Some network error occurred when fetching erasure chunk origin=Public(8a2a3d2aa8595a2b4e60b5e89ebdd0ea864ca2a68d640c67334cca91d6d74e4f (148AA2gf...)) relay_parent=0x8e92ea91c8dfb19bb314b66220707d022bcdda916b7a7f30812d6092127202a5 group_index=GroupIndex(56) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xedb7e75aee0e9d317b2b235d622598fe0f964a06eb91fda3472e6a7e2658bb02 err=NotConnected traceID=315981917766965104777139897034351155454
Some network error occurred when fetching erasure chunk origin=Public(0849872804f7d5257fe7fa5a4293df4f9e4f900d2046fb5289cb48c97963155a (1BsE5PXs...)) relay_parent=0x21fd5ae0cafae248b0127c35430710ed1f8b589b8999ae47d6773f1ddd47640b group_index=GroupIndex(2) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x9821368f7ecd0b9190c0658eb0e95217a78333787d9bc95b4e91e6c8c438cd50 err=NotConnected traceID=202215107774627694384873476864468406807
Some network error occurred when fetching erasure chunk origin=Public(76af898d3856bbaca4a9954568509af53603d1a5afcc1e555c05328c2e874e44 (13gcoAJW...)) relay_parent=0x4023781a0abb9c617153737f6f7b697b48c38c446fef307064dd76dc98e9d52b group_index=GroupIndex(16) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x8325bc21d335709a78dfd76690f5f1ac081f6cc4f1809d65d04fff11fb96e7a5 err=NotConnected traceID=174324798204490559352715166569698029996
Some network error occurred when fetching erasure chunk origin=Public(7e595af665c821e4a7f584efe325c58f9f900f2c22f856723dd959f25a97f17a (13rfaVsY...)) relay_parent=0x21fd5ae0cafae248b0127c35430710ed1f8b589b8999ae47d6773f1ddd47640b group_index=GroupIndex(17) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x4f6766721b85c11930a9b0b2206d2f1947cd6bf3144ea2afd1cd1061d485d476 err=NotConnected traceID=105545896089745334582248513451747782425
Some network error occurred when fetching erasure chunk origin=Public(5cfac09f57b19f42da147304cdc6bd26ae9fd4e292e53081fe77fb4114882d0d (136utsqs...)) relay_parent=0x8aa95da893954dedd55feaa06aefd037f08274ad01bd14979c30530f428db594 group_index=GroupIndex(11) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x529efe76398e7331025066f44dfe615e5ae2acb5e8e813869ad7d2bc17ba8cef err=NotConnected traceID=109822239656786965553304198269785497950
Some network error occurred when fetching erasure chunk origin=Public(6ab9c89ba84d98e4ef3b3a71ce5773f373624abfb01e6ae5c2920f45d4724d1a (13QwGNc6...)) relay_parent=0x0de44d17c6edbb8e53ec9b9db0a7d9d320df780e18a4a3f3bfa28f95b5ac64f7 group_index=GroupIndex(56) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xbbfce123ae335f5d2762fcc43e9568653b69be2c4067334677303edc196b66c3 err=NotConnected traceID=249878660389189051689414377405261637733
Some network error occurred when fetching erasure chunk origin=Public(e29c2b033f77fce6a102fbb891ea074b60774992a966a353753c66ccb6621661 (1688DPLD...)) relay_parent=0x0de44d17c6edbb8e53ec9b9db0a7d9d320df780e18a4a3f3bfa28f95b5ac64f7 group_index=GroupIndex(47) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x555b6913dc011d65c2d53a55afcd9886c534969457f823ce79e99f8e68c68eb8 err=NotConnected traceID=113459009882276039619740679994604361862
Some network error occurred when fetching erasure chunk origin=Public(6ab9c89ba84d98e4ef3b3a71ce5773f373624abfb01e6ae5c2920f45d4724d1a (13QwGNc6...)) relay_parent=0x4023781a0abb9c617153737f6f7b697b48c38c446fef307064dd76dc98e9d52b group_index=GroupIndex(56) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x01a04faabf566908ba9a874c0e4ce65864cfb8d2c2a1e3c10d7971bbbf2443ca err=NotConnected traceID=2161611331512904706234197744355698264
Some network error occurred when fetching erasure chunk origin=Public(c8fe387494182df44d7c718be506a7ce52316094a9228c24934bf6eb06936d19 (15YY6afA...)) relay_parent=0x0de44d17c6edbb8e53ec9b9db0a7d9d320df780e18a4a3f3bfa28f95b5ac64f7 group_index=GroupIndex(54) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x6ff916d8cdfb8ad0b3e710441185db077730384145c936c9cfda3f7f6e4c616c err=NotConnected traceID=148837652839943740320232686613120277255
Some network error occurred when fetching erasure chunk origin=Public(c6c4e1328a4a6f7bb429aa6aa01813462c689c65dae0244e3bc3d7e6329ae219 (15Vcy4pQ...)) relay_parent=0x0de44d17c6edbb8e53ec9b9db0a7d9d320df780e18a4a3f3bfa28f95b5ac64f7 group_index=GroupIndex(17) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x7d3997e9102fdd14931fdd31629f1f127f601267763faf39550b51cb8c6a2f2d err=NotConnected traceID=166452541503072610206819235745115348754
Some network error occurred when fetching erasure chunk origin=Public(6ab9c89ba84d98e4ef3b3a71ce5773f373624abfb01e6ae5c2920f45d4724d1a (13QwGNc6...)) relay_parent=0x21fd5ae0cafae248b0127c35430710ed1f8b589b8999ae47d6773f1ddd47640b group_index=GroupIndex(56) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x1ffdf927b1b78b6e03a5e63cab91c0b1906dd66bbd9399cd7dc517ed43e61583 err=NotConnected traceID=42524772439432089373994354047282430129
Some network error occurred when fetching erasure chunk origin=Public(94e02c728aa186e3e3d14808b24781d46078618655940cbea7f9964fbbfe676f (14NCgTua...)) relay_parent=0x0de44d17c6edbb8e53ec9b9db0a7d9d320df780e18a4a3f3bfa28f95b5ac64f7 group_index=GroupIndex(14) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x4d6c3adbed97ddf0be52ef89259c73bace12913cd74e2d1ae2f7d34df7eb7b74 err=NotConnected traceID=102912517540416429228595128534497522618
Some network error occurred when fetching erasure chunk origin=Public(5cfac09f57b19f42da147304cdc6bd26ae9fd4e292e53081fe77fb4114882d0d (136utsqs...)) relay_parent=0x4023781a0abb9c617153737f6f7b697b48c38c446fef307064dd76dc98e9d52b group_index=GroupIndex(11) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x78027e9f3c75da06bc867e019a7a160e65efe7a9e4af3d990caf51996ce4519a err=NotConnected traceID=159520312287506448792551032586652751374
Some network error occurred when fetching erasure chunk origin=Public(e29c2b033f77fce6a102fbb891ea074b60774992a966a353753c66ccb6621661 (1688DPLD...)) relay_parent=0x21fd5ae0cafae248b0127c35430710ed1f8b589b8999ae47d6773f1ddd47640b group_index=GroupIndex(47) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x84486d08358ae98bf23c8bdd3feb3dbcbd9cabf21b03ddb53440af5befaf053a err=NotConnected traceID=175834152250466141135328661071094037948
Some network error occurred when fetching erasure chunk origin=Public(c8fe387494182df44d7c718be506a7ce52316094a9228c24934bf6eb06936d19 (15YY6afA...)) relay_parent=0x21fd5ae0cafae248b0127c35430710ed1f8b589b8999ae47d6773f1ddd47640b group_index=GroupIndex(54) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x30ecef76c178b31c7c498da2820bced2070ae703208dd4fcb5a11cb1a2371089 err=NotConnected traceID=65033182760985154475077562016713133778
Some network error occurred when fetching erasure chunk origin=Public(c6c4e1328a4a6f7bb429aa6aa01813462c689c65dae0244e3bc3d7e6329ae219 (15Vcy4pQ...)) relay_parent=0x21fd5ae0cafae248b0127c35430710ed1f8b589b8999ae47d6773f1ddd47640b group_index=GroupIndex(17) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x4f6766721b85c11930a9b0b2206d2f1947cd6bf3144ea2afd1cd1061d485d476 err=NotConnected traceID=105545896089745334582248513451747782425
Some network error occurred when fetching erasure chunk origin=Public(6ab9c89ba84d98e4ef3b3a71ce5773f373624abfb01e6ae5c2920f45d4724d1a (13QwGNc6...)) relay_parent=0x8e92ea91c8dfb19bb314b66220707d022bcdda916b7a7f30812d6092127202a5 group_index=GroupIndex(56) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xedb7e75aee0e9d317b2b235d622598fe0f964a06eb91fda3472e6a7e2658bb02 err=NotConnected traceID=315981917766965104777139897034351155454
Node uses the following topology indices index_in_gossip_topology=Some(ValidatorIndex(263)) index_in_parachain_authorities=Some(Active(ValidatorIndex(263)))
👮‍♀️ Running in Secure Validator Mode. It is highly recommended that you operate according to our security guidelines.
tps://wiki.polkadot.network/docs/maintain-guides-secure-validator#secure-validator-mode
Sending fatal alert BadCertificate
Sending fatal alert BadCertificate
Sending fatal alert BadCertificate
Sending fatal alert BadCertificate
🔍 Discovered new external address for our node: /ip4/91.90.166.5/tcp/30333/p2p/12D3KooWHRd8ANXyUvkRNJiXsSUzB2pSaE7QvVATdsTNRCjRvdiP
Received msg before first active leaves update. This is not expected - message will be dropped. msg=DetermineUndisputedChain { base: (21447857, 0xf699f639539ba536e48b17d8e4e1b6dedd3533320814e0d6f7a3b2c9ee992702), block_descriptions: [], tx: Sender { complete: false } }
Call to `DetermineUndisputedChain` failed error=DetermineUndisputedChainCanceled(Canceled)
Importing locally an already known assignment message_subject=MessageSubject(0xf699f639539ba536e48b17d8e4e1b6dedd3533320814e0d6f7a3b2c9ee992702, Bitfield(BitVec<u8, bitvec::order::Lsb0> { addr: 0x7e9d3080b0a8, head: 000, bits: 35, capacity: 40 } [0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1], PhantomData<u32>), ValidatorIndex(263))
Importing locally an already known assignment message_subject=MessageSubject(0xf699f639539ba536e48b17d8e4e1b6dedd3533320814e0d6f7a3b2c9ee992702, Bitfield(BitVec<u8, bitvec::order::Lsb0> { addr: 0x7e9d3080b078, head: 000, bits: 35, capacity: 40 } [0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1], PhantomData<u32>), ValidatorIndex(263))
Importing locally an already known assignment message_subject=MessageSubject(0xf699f639539ba536e48b17d8e4e1b6dedd3533320814e0d6f7a3b2c9ee992702, Bitfield(BitVec<u8, bitvec::order::Lsb0> { addr: 0x7e9d3080b110, head: 000, bits: 35, capacity: 40 } [0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1], PhantomData<u32>), ValidatorIndex(263))
Importing locally an already known assignment message_subject=MessageSubject(0xc3d8b211dda809b0797ef5515fa1eee1248419a965c5158a0572c931afa85397, Bitfield(BitVec<u8, bitvec::order::Lsb0> { addr: 0x7e9d3080b080, head: 000, bits: 3, capacity: 8 } [0, 1, 1], PhantomData<u32>), ValidatorIndex(263))
Importing locally an already known assignment message_subject=MessageSubject(0x03b944ecb4dbfee868617935460bf406cc93ee8a7e5263f81a8d5f62c20d8cc8, Bitfield(BitVec<u8, bitvec::order::Lsb0> { addr: 0x7e9d3080b0e8, head: 000, bits: 30, capacity: 32 } [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1], PhantomData<u32>), ValidatorIndex(263))
Importing locally an already known assignment message_subject=MessageSubject(0x03b944ecb4dbfee868617935460bf406cc93ee8a7e5263f81a8d5f62c20d8cc8, Bitfield(BitVec<u8, bitvec::order::Lsb0> { addr: 0x7e9d3080b090, head: 000, bits: 30, capacity: 32 } [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1], PhantomData<u32>), ValidatorIndex(263))
Importing locally an already known assignment message_subject=MessageSubject(0x03b944ecb4dbfee868617935460bf406cc93ee8a7e5263f81a8d5f62c20d8cc8, Bitfield(BitVec<u8, bitvec::order::Lsb0> { addr: 0x7e9d3080b1a8, head: 000, bits: 30, capacity: 32 } [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1], PhantomData<u32>), ValidatorIndex(263))
Importing locally an already known assignment message_subject=MessageSubject(0x03b944ecb4dbfee868617935460bf406cc93ee8a7e5263f81a8d5f62c20d8cc8, Bitfield(BitVec<u8, bitvec::order::Lsb0> { addr: 0x7e9d3080b198, head: 000, bits: 30, capacity: 32 } [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1], PhantomData<u32>), ValidatorIndex(263))
Importing locally an already known assignment message_subject=MessageSubject(0x03b944ecb4dbfee868617935460bf406cc93ee8a7e5263f81a8d5f62c20d8cc8, Bitfield(BitVec<u8, bitvec::order::Lsb0> { addr: 0x7e9d3080b1d0, head: 000, bits: 30, capacity: 32 } [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1], PhantomData<u32>), ValidatorIndex(263))
Sending fatal alert BadCertificate
Sending fatal alert BadCertificate
🏆 Imported #21447872 (0xccb7…ffd3 → 0xf51d…705a)
Candidate included without being backed? candidate_hash=0x8486297f52e2aaf968560f987d90c6045edd5c9a4da27c41852502746c8ec90c traceID=176154704889074746599226652480343426564
Candidate included without being backed? candidate_hash=0x4097f412fa55a7686df06545990e4d1500dc9cfc7b162f84903d5e54118f0c29 traceID=85859578967398392813682753268488228117
Candidate included without being backed? candidate_hash=0x2e1139c13496af000304b7af132528a72037419ec7c620e37aed38aed0aaed83 traceID=61233928257359381419495604540194236583
Candidate included without being backed? candidate_hash=0x651e707ecf0b5faea537bb7264d0cc42a1cfbf6356cf085fe4284a7b8c2cb1c5 traceID=134410078156733780840023168685551176770
Candidate included without being backed? candidate_hash=0xce8df1d924253f0b3ad5eee9b5abe93e5162a933d095fa405705c43f73077e78 traceID=274557986253158314486876810150559869246
Candidate included without being backed? candidate_hash=0x50da397cf8af2b9c420f2b7c01bdb0961ae2ed73e689de51492592c19ecb67c6 traceID=107471326376557472567653286800483266710
Candidate included without being backed? candidate_hash=0x95f63d3799dfe7143e8d0558f1bb0a2e1ec5984525ccc8995b6c6b55acf894c1 traceID=199333518031308681896259729370730138158
Candidate included without being backed? candidate_hash=0xe13365cded2f761a788962d653fcdeb8631cf03f94363ce7e283dccdc46a369b traceID=299343171029939956424040244647357636280
Candidate included without being backed? candidate_hash=0x234277057ae2017480e71639757e3eb079e96d17a283bab1c84f26df8a0469be traceID=46868085486049396600132614748538551984
Candidate included without being backed? candidate_hash=0x974a2aed85ca9c0d70bec4f488e75fda99a484bbc9806ae7455c17328379b6da traceID=201098528010738189562696135474099478490
Candidate included without being backed? candidate_hash=0xc9b820a7e1aaaf007b00bbe2266e2f228961c14dc1edd00403f9df1cebe27e1b traceID=268130872112789426495047695132141694754
Candidate included without being backed? candidate_hash=0x3613b6e788cea739056725a3a03534e3145c115ec1f1c5b8e021c19b41de9643 traceID=71880675155290814002426718860023772387
Some network error occurred when fetching erasure chunk origin=Public(6c7aa614ce9faa7901f55f64b0b0bf1c084f52e7069e5bf1097a9889ff977307 (13TEc93P...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(38) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x7d3be0504e85319e95dd7a9c72ecf976c78fb01689501e0285339d91545010ed err=Network(DialFailure) traceID=166464394610073110279796961079630559606
Some network error occurred when fetching erasure chunk origin=Public(7254c168993010ae11755affc59d95eb79a9c56eec44b2bf733dcc73bdb0e151 (13audvBh...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(39) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x001ee36bec3a0b989af91570661529683ce2cf642f646d9ee861309b8b2ef78a err=Network(DialFailure) traceID=160381563258097560804749229786737000
Some network error occurred when fetching erasure chunk origin=Public(869e7916ac057408f5fb2b6b97b875a3f7687b8ea8c0c1192221a05689c9e430 (143WVazv...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(40) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x1b5d6e109a6ae36632713f7528d7f5974ab6e75cdea38985b3ae23575dde65b1 err=Network(DialFailure) traceID=36374271874533380961592716831198016919
Some network error occurred when fetching erasure chunk origin=Public(a2e8b55d73a307954f761d76c1d89019e2cb7aa014ff45e3205b3e5baa67f34f (14gbtQ64...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(45) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x599b63325a482d7b3aeff67a8649183cc039ab8dfb168647dd558dd7b03e636f err=Network(DialFailure) traceID=119108109585830206522071349416906266684
Some network error occurred when fetching erasure chunk origin=Public(c44448fc05a2545c1cd24ee7cf5f6999dacedd9801a70ed7e9cc2a2038af073f (15SLfsw2...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(46) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xc9f0fd0b5a04fe0765700c3002985f35fcc56355f4d547447a4710cdc8f3ca5f err=Network(DialFailure) traceID=268426110747815646818760238122264977205
Some network error occurred when fetching erasure chunk origin=Public(70508bcd1902e7b9b5b2423ddaa995ceec5ad7eaeccbeea1670c2e87e94b7f1a (13YGHqVK...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(47) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xe6135b961b9f2c7310cdaac95774644146394ea680fd3bcd81a9ddff530a5dbd err=Network(DialFailure) traceID=305822950262889646311683590818094146625
Some network error occurred when fetching erasure chunk origin=Public(58c7a156ab25151065f3bcfad6ff9589d46a36f067e1f46c00a942b32a446452 (131QWxEX...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(48) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x0bde698eb551dd26fccc2b3b2bf8106bd90c1c62e4d84a9bc355688465441abe err=Network(DialFailure) traceID=15776338815752020917643314113635946603
Some network error occurred when fetching erasure chunk origin=Public(a4c1b6ec21b490241d3db57f8c42af5565c276376fab107127665b492295d137 (14j2Q6yy...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(51) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x18123d65fe9839c3a6ebe15a41cb87511ca075844f48d09547f676bedf7eddb6 err=Network(DialFailure) traceID=31996178550115066542834520152301340497
Some network error occurred when fetching erasure chunk origin=Public(4a78cfc4f9069a9a410d5222ce85d24f7b5db9bf1daa03b86f07c859f636d947 (12geS4HE...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(54) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x8b71cb9633ed80308115086854c18c5a14d6183a81afd1e737558c47f5824220 err=Network(DialFailure) traceID=185353550188562516562191966109383625818
Some network error occurred when fetching erasure chunk origin=Public(6ab9c89ba84d98e4ef3b3a71ce5773f373624abfb01e6ae5c2920f45d4724d1a (13QwGNc6...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(56) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xe856ee1e045eea1ca4a2e916bacd70f420083f85c48bc52a44db3feecc58eab8 err=NotConnected traceID=308832262143617706843686169551926292724
Some network error occurred when fetching erasure chunk origin=Public(14497fadd0ac2e7e520a4f40985914d8eb1c0e9c7abfa2582e51aa3c8c76d24a (1TbnuDAd...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(57) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x049b71d75c0fe4d77dae4622a8612c4cc1404bbc68a137cda53fb7564462f1fb err=Network(DialFailure) traceID=6124026971044550445150232725814324300
Some network error occurred when fetching erasure chunk origin=Public(94176052dce88ba60b48e3db95facaec024120bc8bc6590322f04380e5a2c817 (14MB2okG...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(53) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xc0a0389605c03b503ceaf079c96062b2d6c9f5f00cd5ab4313172efb67ed7ca9 err=Network(DialFailure) traceID=256043690389011420557514549607851385522
Some network error occurred when fetching erasure chunk origin=Public(c2b89cda6db6628508a57a315de9d78ca89d61713a8a21ab8a247b8a61cb026a (15QK8btn...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(55) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xea7b2957a9b549b401dfa38302b71858cb6777cc7f2dbfa24d54438472bebf7b err=Network(DialFailure) traceID=311678842051436116803744609173576816728
Some network error occurred when fetching erasure chunk origin=Public(7c125697d2722da808d130071a597858c1df4ae6c5ac43a87f4cff4376a34d71 (13ogPMAB...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(0) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xbffec3796f2c9e901b2ce1aa5eb9f4a09f0e7f12f74e54a837b0e93461148a3b err=Network(DialFailure) traceID=255205355287873931862417387504092705952
Some network error occurred when fetching erasure chunk origin=Public(aa56949cc5b86cea9edbc4a617d45442699b4a3700ff93065a87cf5240fa4f14 (14rLrtfY...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(1) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x9b44bb30de2690e32962eb2bf27347f6169de2040da04393987db7f7449113ae err=Network(DialFailure) traceID=206387212215342307698538254323716147190
Some network error occurred when fetching erasure chunk origin=Public(ee90d96f1009ca2f91ab039f0e66eddb2dfaf7e4b4aff21c3bf73feb75c7610b (16PoRhQc...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(8) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xa2a4e4082157db2c9f32c503b61c6fae845e8f0ddd91da7d78f143274a539b83 err=Network(DialFailure) traceID=216191097035490232728265901407288979374
Some network error occurred when fetching erasure chunk origin=Public(22d9a1a7853cb52dd217f6e9f3b87791e5dbec78c7b09dae9843f97bdafc373a (1nhH8dA9...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(2) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xf07fe19981e00bc7b26589f161ddb9519b5188878d4e77fb7f4c4a11459e63d4 err=NotConnected traceID=319678716393677840194361512861007460689
Some network error occurred when fetching erasure chunk origin=Public(9aa2ee807cb7b71ce4f1425535ec895d4b6f786cf8f108d055e76833a6103b0e (14VkmxHJ...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(11) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xcef780a4dabcb49d97560062889b591bea8cf44cb23ab20819fe015d3ae725af err=Network(DialFailure) traceID=275106073665294555123748332946748823835
Some network error occurred when fetching erasure chunk origin=Public(3cb7b6197fcd83df725f729c1597a1c3e45eabd7aeab82d59497c185c274b144 (12NcSurQ...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(12) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x5e4faddf87b7a673f2b7acbf8ad2f04f33d20774cc3b3a5cf465b9093e855e7c err=Network(DialFailure) traceID=125361149622350511658114822982239187023
Some network error occurred when fetching erasure chunk origin=Public(26bb23c6bed870ea1785b8a3de686d04a2460d85c3d9dfb11aea0f7c38aeea2f (1snQt5om...)) relay_parent=0x35895d1fbb335fc39c239e0a6e50941d2e89dfaf4d4ea914f8f0234779109e49 group_index=GroupIndex(4) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x7fb7049905a1bd977db7e7609fcd18cb90664cdf97d1833f8d5d9ff3921bba3d err=Network(DialFailure) traceID=169762239043086426031777186796582803659
Some network error occurred when fetching erasure chunk origin=Public(280d84ed3e2333c7e336f1a9c1999651a43580aa8fc84afdd18c1494ca4a3b0b (1uWw2Nfz...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(19) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x9cc3c423a9de07d55c41c8999e6348c28a24277dc8baa9c25c07d3cddf7f5ec1 err=Network(DialFailure) traceID=208376043407700556471343704789234174146
Some network error occurred when fetching erasure chunk origin=Public(8a8c95e7e1ff8eb04a2da78a216a9486069c1538be959b7e7c0168902f04e617 (148fNW8D...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(26) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xd5b307f52643500ee2907ab52b9bc64cb7b2aafaf3865d54002bf400ad631950 err=NotConnected traceID=284055145639473663097497563669225784908
Some network error occurred when fetching erasure chunk origin=Public(86acb09806dbaba3f33bab6c954127b10397f36c4083920afb95f4fe51c62f62 (143aiY6w...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(13) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x875d83d41e808c64af99044494de4c5e0ffed98217cba481dd3fe5fdc94292ef err=Network(DialFailure) traceID=179931336840275868992518579255766830174
Some network error occurred when fetching erasure chunk origin=Public(e88648905e5c418ef73c2afaed085dbea8f78268e187ae8335c28391d4745d2e (16FszzDy...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(15) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x92ef4638078ae2ace8353e3dfd203e309998e1c7883df71b0a8cde34e3072431 err=Network(DialFailure) traceID=195309670541571194939160892921234865712
Some network error occurred when fetching erasure chunk origin=Public(c2b109d63e840e91a667b8abff4c9ef51485690694ecf74c052ef9cf408d8d2b (15QGt6nZ...)) relay_parent=0x35895d1fbb335fc39c239e0a6e50941d2e89dfaf4d4ea914f8f0234779109e49 group_index=GroupIndex(27) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x74e07b87bd5e02920f0f6477cfd3e4f13c58fdb8a5b39a17d6f725a0320bfb89 err=Network(DialFailure) traceID=155356027498151549249100960943867946225
Some network error occurred when fetching erasure chunk origin=Public(b650cf1e3fa031e6afeb99ddcf58693d934063ca1d211795995d056b76c22344 (1583inA3...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(20) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x6016f47293b3e2951eda81b7fb873cb80b1be442fa841bba5f0397dce2de7d2a err=NotConnected traceID=127725076111905271811373048740096457912
Some network error occurred when fetching erasure chunk origin=Public(56ce074af80277013fb5641e6db709854366a16f591819c7543abc9bff8a1c5e (12xpKdxE...)) relay_parent=0x35895d1fbb335fc39c239e0a6e50941d2e89dfaf4d4ea914f8f0234779109e49 group_index=GroupIndex(42) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x8486297f52e2aaf968560f987d90c6045edd5c9a4da27c41852502746c8ec90c err=Network(DialFailure) traceID=176154704889074746599226652480343426564
Some network error occurred when fetching erasure chunk origin=Public(4ada71ddd435ad8f98d1f49861d8ec7fc09fc3a6dddb061562a240522416e936 (12h9SEmD...)) relay_parent=0x35895d1fbb335fc39c239e0a6e50941d2e89dfaf4d4ea914f8f0234779109e49 group_index=GroupIndex(58) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x4097f412fa55a7686df06545990e4d1500dc9cfc7b162f84903d5e54118f0c29 err=Network(DialFailure) traceID=85859578967398392813682753268488228117
Some network error occurred when fetching erasure chunk origin=Public(28e4f64306286309157a42f85a3386b2d9432030cbe651e09199059a841c3a4f (1vcw1tHx...)) relay_parent=0x35895d1fbb335fc39c239e0a6e50941d2e89dfaf4d4ea914f8f0234779109e49 group_index=GroupIndex(28) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xdc83026c22c648e47772bf89166d935431c2749cf60c1792639ba4032460a659 err=Network(DialFailure) traceID=293110399093372515173125353044886524756
Some network error occurred when fetching erasure chunk origin=Public(50a56ddde1c7d26cf746de885de060e97bbe1a5f4518b63de31c173446712c03 (12pjyTsG...)) relay_parent=0x35895d1fbb335fc39c239e0a6e50941d2e89dfaf4d4ea914f8f0234779109e49 group_index=GroupIndex(7) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xce8df1d924253f0b3ad5eee9b5abe93e5162a933d095fa405705c43f73077e78 err=Network(DialFailure) traceID=274557986253158314486876810150559869246
Some network error occurred when fetching erasure chunk origin=Public(26bb23c6bed870ea1785b8a3de686d04a2460d85c3d9dfb11aea0f7c38aeea2f (1snQt5om...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(4) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x651e707ecf0b5faea537bb7264d0cc42a1cfbf6356cf085fe4284a7b8c2cb1c5 err=Network(DialFailure) traceID=134410078156733780840023168685551176770
Some network error occurred when fetching erasure chunk origin=Public(08a75be8e3353bc81b27f24377bd8a3e1c8eb661f7234437f8d4d8274f36f24e (1CM6kM1M...)) relay_parent=0x35895d1fbb335fc39c239e0a6e50941d2e89dfaf4d4ea914f8f0234779109e49 group_index=GroupIndex(23) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x234277057ae2017480e71639757e3eb079e96d17a283bab1c84f26df8a0469be err=Network(DialFailure) traceID=46868085486049396600132614748538551984
Some network error occurred when fetching erasure chunk origin=Public(9e3c240cf3f6c63f8ca7db4f6d97de8c2e9c8191f99eb6207a546eac1d36bc1d (14aUS22V...)) relay_parent=0x35895d1fbb335fc39c239e0a6e50941d2e89dfaf4d4ea914f8f0234779109e49 group_index=GroupIndex(14) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x50da397cf8af2b9c420f2b7c01bdb0961ae2ed73e689de51492592c19ecb67c6 err=Network(DialFailure) traceID=107471326376557472567653286800483266710
Some network error occurred when fetching erasure chunk origin=Public(c2b109d63e840e91a667b8abff4c9ef51485690694ecf74c052ef9cf408d8d2b (15QGt6nZ...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(27) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xc9b820a7e1aaaf007b00bbe2266e2f228961c14dc1edd00403f9df1cebe27e1b err=Network(DialFailure) traceID=268130872112789426495047695132141694754
Some network error occurred when fetching erasure chunk origin=Public(f86c289cff15b99d17f3006eba6efc378a3bcfae0f019666a47f89298c13ff71 (16cj1JcY...)) relay_parent=0x35895d1fbb335fc39c239e0a6e50941d2e89dfaf4d4ea914f8f0234779109e49 group_index=GroupIndex(24) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x974a2aed85ca9c0d70bec4f488e75fda99a484bbc9806ae7455c17328379b6da err=Network(DialFailure) traceID=201098528010738189562696135474099478490
Some network error occurred when fetching erasure chunk origin=Public(28e4f64306286309157a42f85a3386b2d9432030cbe651e09199059a841c3a4f (1vcw1tHx...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(28) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x3613b6e788cea739056725a3a03534e3145c115ec1f1c5b8e021c19b41de9643 err=Network(DialFailure) traceID=71880675155290814002426718860023772387
Some network error occurred when fetching erasure chunk origin=Public(6c7aa614ce9faa7901f55f64b0b0bf1c084f52e7069e5bf1097a9889ff977307 (13TEc93P...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(38) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x13329b448cf9e57d634ed8a0276f06a585a8d1c7aee6a1098e1b85ee6f9ee18f err=Network(DialFailure) traceID=25518095967473767584485360835450832549
Some network error occurred when fetching erasure chunk origin=Public(7254c168993010ae11755affc59d95eb79a9c56eec44b2bf733dcc73bdb0e151 (13audvBh...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(39) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xae1522cf54f9b10a6ef4de457ece3df324f4bbd0810b758ef985d3184f1ec550 err=Network(DialFailure) traceID=231395415529059357280381061712067837427
Some network error occurred when fetching erasure chunk origin=Public(869e7916ac057408f5fb2b6b97b875a3f7687b8ea8c0c1192221a05689c9e430 (143WVazv...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(40) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x36290213a99be04a71817de91df9ba062abc98bee9ed6493a929d31201e9db63 err=Network(DialFailure) traceID=71991238066231089489116359006541691398
Some network error occurred when fetching erasure chunk origin=Public(a2e8b55d73a307954f761d76c1d89019e2cb7aa014ff45e3205b3e5baa67f34f (14gbtQ64...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(45) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xd4b0a456094f6b8afdee4971f7c1e4a6f45aebd8b26d2933b10f3fdfb8fca0bf err=Network(DialFailure) traceID=282713512485182647904214678164536943782
Some network error occurred when fetching erasure chunk origin=Public(c44448fc05a2545c1cd24ee7cf5f6999dacedd9801a70ed7e9cc2a2038af073f (15SLfsw2...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(46) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xf9360aecb715614a77e6a5682c4c77e5f4e57ef43ba3f9c814e89d7f2c2d885a err=Network(DialFailure) traceID=331258376559408926610842245761860401125
Some network error occurred when fetching erasure chunk origin=Public(70508bcd1902e7b9b5b2423ddaa995ceec5ad7eaeccbeea1670c2e87e94b7f1a (13YGHqVK...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(47) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xc217d2e745af277eb1fdfe619c3015e7142b92318657e767a9b7e3bf9f1b1371 err=Network(DialFailure) traceID=257993931639308502230159426201538532839
Some network error occurred when fetching erasure chunk origin=Public(a4c1b6ec21b490241d3db57f8c42af5565c276376fab107127665b492295d137 (14j2Q6yy...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(51) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xc4805015465a0050d23a3d821258b574c9d28add1504a97665d514b2694e851d err=Network(DialFailure) traceID=261194925450068429460930577848312837492
Some network error occurred when fetching erasure chunk origin=Public(4a78cfc4f9069a9a410d5222ce85d24f7b5db9bf1daa03b86f07c859f636d947 (12geS4HE...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(54) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x00a6f589d8a5bbde55affd5f81a3f68740b332f530122259c54885f81a3d8570 err=Network(DialFailure) traceID=866901390177062565335041629078419079
Some network error occurred when fetching erasure chunk origin=Public(6ab9c89ba84d98e4ef3b3a71ce5773f373624abfb01e6ae5c2920f45d4724d1a (13QwGNc6...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(56) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x3e5ae837f72fdcbdb442527f9cc8880d9bfa05ecf12858310835b0d1254de4b0 err=NotConnected traceID=82884152409010563438124721596845819917
Some network error occurred when fetching erasure chunk origin=Public(c2b89cda6db6628508a57a315de9d78ca89d61713a8a21ab8a247b8a61cb026a (15QK8btn...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(55) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x6321d355a73ee32b247435cc9bd06d0d52936479911a5b906813dfd38cfeb045 err=Network(DialFailure) traceID=131769203753618527758155677368287522061
Some network error occurred when fetching erasure chunk origin=Public(7c125697d2722da808d130071a597858c1df4ae6c5ac43a87f4cff4376a34d71 (13ogPMAB...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(0) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xe7fc5fa0f6a753f86867ad591911e016fc2a8031e890af075174f26fa79d300e err=Network(DialFailure) traceID=308362065416420291960813716695350894614
Some network error occurred when fetching erasure chunk origin=Public(58c7a156ab25151065f3bcfad6ff9589d46a36f067e1f46c00a942b32a446452 (131QWxEX...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(48) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x71d126d64df426eb8c2c40cbd491854eb86119daf264ef304efe4f1a12678966 err=Network(DialFailure) traceID=151288741277646496879778692045613401422
Some network error occurred when fetching erasure chunk origin=Public(14497fadd0ac2e7e520a4f40985914d8eb1c0e9c7abfa2582e51aa3c8c76d24a (1TbnuDAd...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(57) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xb7570bee284f7d22cf3f64377c1859d970aa0cb3b3e3289e0ccda52bee606d4e err=Network(DialFailure) traceID=243700695030615949778902636639590832601
Some network error occurred when fetching erasure chunk origin=Public(aa56949cc5b86cea9edbc4a617d45442699b4a3700ff93065a87cf5240fa4f14 (14rLrtfY...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(1) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x9b232e0116a5ab512047b8fd94b94b11231fcc58af1a51cbdfbba9ed630b37b3 err=Network(DialFailure) traceID=206213002813789561755679378019477703441
Some network error occurred when fetching erasure chunk origin=Public(26bb23c6bed870ea1785b8a3de686d04a2460d85c3d9dfb11aea0f7c38aeea2f (1snQt5om...)) relay_parent=0x313405c8380fc4210709e1610803f3cb20080cdb58d8fbb7cda2eb1b2fb26f07 group_index=GroupIndex(4) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xfd60c4cfb1c0bedab43d93f5328020dd300201b925fb63097f4c4ac6ccc74362 err=Network(DialFailure) traceID=336797135239526877256876828272214679773
Some network error occurred when fetching erasure chunk origin=Public(ee90d96f1009ca2f91ab039f0e66eddb2dfaf7e4b4aff21c3bf73feb75c7610b (16PoRhQc...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(8) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x15ba4b9d9ab3c5e826fdce2fd306727046a51463dfecd3c7c5077258fd524f61 err=Network(DialFailure) traceID=28881088794590523707518956887328846448
Some network error occurred when fetching erasure chunk origin=Public(22d9a1a7853cb52dd217f6e9f3b87791e5dbec78c7b09dae9843f97bdafc373a (1nhH8dA9...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(2) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x40cc33cc8c9cc7c9f4a41b3c100f421e2433fa5c438aaf4a702f49b0acf3e081 err=NotConnected traceID=86130870898328097126382449935841706526
Some network error occurred when fetching erasure chunk origin=Public(9aa2ee807cb7b71ce4f1425535ec895d4b6f786cf8f108d055e76833a6103b0e (14VkmxHJ...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(11) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xfeb6807b9ed237cd7045ca43bc47b81645acbba8447db91ccd323b85d94cbb08 err=Network(DialFailure) traceID=338571514900267996508008113521256019990
Some network error occurred when fetching erasure chunk origin=Public(3cb7b6197fcd83df725f729c1597a1c3e45eabd7aeab82d59497c185c274b144 (12NcSurQ...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(12) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x61b436247fd355c90cb8ad5c1836115934b7921b4b2a6b5c4322296b1fbaa5bb err=Network(DialFailure) traceID=129870827167565641053636607060202623321
Some network error occurred when fetching erasure chunk origin=Public(86acb09806dbaba3f33bab6c954127b10397f36c4083920afb95f4fe51c62f62 (143aiY6w...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(13) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xb519993107aeeae306103fc10ad00a8d384eccdae830408a8d907a7705b9d274 err=Network(DialFailure) traceID=240723181751760322973796922780791343757
Some network error occurred when fetching erasure chunk origin=Public(e88648905e5c418ef73c2afaed085dbea8f78268e187ae8335c28391d4745d2e (16FszzDy...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(15) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x5724031051c1d45e8127b9567a44a370fc2ef351b3348dc1e542a0ce36b9f893 err=Network(DialFailure) traceID=115829820460376957284208481677661741936
Some network error occurred when fetching erasure chunk origin=Public(c2b109d63e840e91a667b8abff4c9ef51485690694ecf74c052ef9cf408d8d2b (15QGt6nZ...)) relay_parent=0x313405c8380fc4210709e1610803f3cb20080cdb58d8fbb7cda2eb1b2fb26f07 group_index=GroupIndex(27) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x680c91faa13c8ea3de5100dc08b4df0fa7a0f563f9fa6c9c8007f4e1f6d47274 err=Network(DialFailure) traceID=138304979930267122501127042576552353551
Some network error occurred when fetching erasure chunk origin=Public(280d84ed3e2333c7e336f1a9c1999651a43580aa8fc84afdd18c1494ca4a3b0b (1uWw2Nfz...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(19) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x3012fb10b12c7619f98a8290876912f5e18157fa217d26d8fcc893a358a6c722 err=Network(DialFailure) traceID=63901497348423302787038129412494660341
Some network error occurred when fetching erasure chunk origin=Public(56ce074af80277013fb5641e6db709854366a16f591819c7543abc9bff8a1c5e (12xpKdxE...)) relay_parent=0x313405c8380fc4210709e1610803f3cb20080cdb58d8fbb7cda2eb1b2fb26f07 group_index=GroupIndex(42) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x25a25533e0fd1b15de44976600096eb745c881f127fb05680c4223e326f4faba err=Network(DialFailure) traceID=50024316050207756332366606665859559095
Some network error occurred when fetching erasure chunk origin=Public(b650cf1e3fa031e6afeb99ddcf58693d934063ca1d211795995d056b76c22344 (1583inA3...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(20) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x8a8b34426829bb0492fe98f00fba1c406ef998215c1fa8b0c97973380b14809c err=NotConnected traceID=184156252628249737476334871131895503936
Some network error occurred when fetching erasure chunk origin=Public(28e4f64306286309157a42f85a3386b2d9432030cbe651e09199059a841c3a4f (1vcw1tHx...)) relay_parent=0x313405c8380fc4210709e1610803f3cb20080cdb58d8fbb7cda2eb1b2fb26f07 group_index=GroupIndex(28) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x0ecedf5a711182a7ca952eb1de5d79240e4b8c559b7d86f2ce56af9824471e7a err=Network(DialFailure) traceID=19683335236716212170017604677645728036
Some network error occurred when fetching erasure chunk origin=Public(94176052dce88ba60b48e3db95facaec024120bc8bc6590322f04380e5a2c817 (14MB2okG...)) relay_parent=0x313405c8380fc4210709e1610803f3cb20080cdb58d8fbb7cda2eb1b2fb26f07 group_index=GroupIndex(53) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x97f6e95dfbb9329cb11fc74240331997cda54e4626018eb8c0b7562b25d78a03 err=Network(DialFailure) traceID=201995465638283256821451220692998756759
Some network error occurred when fetching erasure chunk origin=Public(4ada71ddd435ad8f98d1f49861d8ec7fc09fc3a6dddb061562a240522416e936 (12h9SEmD...)) relay_parent=0x313405c8380fc4210709e1610803f3cb20080cdb58d8fbb7cda2eb1b2fb26f07 group_index=GroupIndex(58) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x9edbadead192cb6c020889afcabcf17a8d47083b4e6e7b69cf43451bc1aa4104 err=Network(DialFailure) traceID=211158663807147126101403790081750004090
Some network error occurred when fetching erasure chunk origin=Public(8a8c95e7e1ff8eb04a2da78a216a9486069c1538be959b7e7c0168902f04e617 (148fNW8D...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(26) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x78b755d91ff0ba0f33e05e37f2afb16983bb450730940a633063a89f9b2202f3 err=NotConnected traceID=160459291026514410333599495654441202025
Some network error occurred when fetching erasure chunk origin=Public(26bb23c6bed870ea1785b8a3de686d04a2460d85c3d9dfb11aea0f7c38aeea2f (1snQt5om...)) relay_parent=0x2a2c53231262a65294d3627ff1efbaa1a6811e5eaa7aef68fabeb2f65508190f group_index=GroupIndex(4) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x89dffba9afdade146a048ae4750d8b9959e0d996f5e14a1135925996f62a421d err=Network(DialFailure) traceID=183267221950781193918278170312922991513
Some network error occurred when fetching erasure chunk origin=Public(50a56ddde1c7d26cf746de885de060e97bbe1a5f4518b63de31c173446712c03 (12pjyTsG...)) relay_parent=0x313405c8380fc4210709e1610803f3cb20080cdb58d8fbb7cda2eb1b2fb26f07 group_index=GroupIndex(7) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xa1153130a84d4f6b49df6d012d84fda1c8b5ffdacc631163be0033c4352d2d92 err=Network(DialFailure) traceID=214115743248510010518424208750982528417
Some network error occurred when fetching erasure chunk origin=Public(c2b109d63e840e91a667b8abff4c9ef51485690694ecf74c052ef9cf408d8d2b (15QGt6nZ...)) relay_parent=0x2a2c53231262a65294d3627ff1efbaa1a6811e5eaa7aef68fabeb2f65508190f group_index=GroupIndex(27) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xffe4f246f9b3ca4d3354bb4c16cfb0d4c9b8161e3d258a764d6b40fca023c626 err=Network(DialFailure) traceID=340141896575274068501875541525601956052
Some network error occurred when fetching erasure chunk origin=Public(08a75be8e3353bc81b27f24377bd8a3e1c8eb661f7234437f8d4d8274f36f24e (1CM6kM1M...)) relay_parent=0x313405c8380fc4210709e1610803f3cb20080cdb58d8fbb7cda2eb1b2fb26f07 group_index=GroupIndex(23) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x0fb38485faf9198ed40e762ea045722887c4d7521046eab220d2e8fd81165780 err=Network(DialFailure) traceID=20870528967537164335017246307578638888
Some network error occurred when fetching erasure chunk origin=Public(a26bec03369eeaa592431cea6971af771739a8fd02d16d87f5f82fce1841826d (14fxpNpM...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(38) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x7d3be0504e85319e95dd7a9c72ecf976c78fb01689501e0285339d91545010ed err=NotConnected traceID=166464394610073110279796961079630559606
Some network error occurred when fetching erasure chunk origin=Public(9c8adc305a080e5fb7c2eddcdff24d3695084af62694f139d59aca72d957bd32 (14YFimMx...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(39) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x001ee36bec3a0b989af91570661529683ce2cf642f646d9ee861309b8b2ef78a err=Network(DialFailure) traceID=160381563258097560804749229786737000
Some network error occurred when fetching erasure chunk origin=Public(8c4e50094346465878b69416bcb22e751a9bf159a6430242cd0a1bcc28176b03 (14Axy7uz...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(40) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x1b5d6e109a6ae36632713f7528d7f5974ab6e75cdea38985b3ae23575dde65b1 err=Network(DialFailure) traceID=36374271874533380961592716831198016919
Some network error occurred when fetching erasure chunk origin=Public(28e4f64306286309157a42f85a3386b2d9432030cbe651e09199059a841c3a4f (1vcw1tHx...)) relay_parent=0x2a2c53231262a65294d3627ff1efbaa1a6811e5eaa7aef68fabeb2f65508190f group_index=GroupIndex(28) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x531609025602fae233d5fdacbc36c540b9c16364bbfe3310386b5a0393f99e94 err=Network(DialFailure) traceID=110440336907797855013001101257511191872
Some network error occurred when fetching erasure chunk origin=Public(3200bdf6ef0b2996121eec3b6305fa98a959eeb715389e1526c53c2df6043671 (128ZcekU...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(45) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x599b63325a482d7b3aeff67a8649183cc039ab8dfb168647dd558dd7b03e636f err=Network(DialFailure) traceID=119108109585830206522071349416906266684
Some network error occurred when fetching erasure chunk origin=Public(705675abc0fd197435e23a2962f73fd2131f30281fe3d27b6d0d3b07eda75939 (13YJ3j16...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(46) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xc9f0fd0b5a04fe0765700c3002985f35fcc56355f4d547447a4710cdc8f3ca5f err=Network(DialFailure) traceID=268426110747815646818760238122264977205
Some network error occurred when fetching erasure chunk origin=Public(e29c2b033f77fce6a102fbb891ea074b60774992a966a353753c66ccb6621661 (1688DPLD...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(47) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xe6135b961b9f2c7310cdaac95774644146394ea680fd3bcd81a9ddff530a5dbd err=Network(DialFailure) traceID=305822950262889646311683590818094146625
Some network error occurred when fetching erasure chunk origin=Public(662326567f87037455978244636309d24302b5f8efb62ab76ff8a020d55b1732 (13JvKuKm...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(48) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x0bde698eb551dd26fccc2b3b2bf8106bd90c1c62e4d84a9bc355688465441abe err=Network(DialFailure) traceID=15776338815752020917643314113635946603
Some network error occurred when fetching erasure chunk origin=Public(52b33dc90d7a7032e235c4e484eacd53ca5fc6a38ce93cc062d1890200f9cc70 (12sSAznD...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(51) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x18123d65fe9839c3a6ebe15a41cb87511ca075844f48d09547f676bedf7eddb6 err=Network(DialFailure) traceID=31996178550115066542834520152301340497
Some network error occurred when fetching erasure chunk origin=Public(74cbe6e914ea9faeb93af966ba5e96936a08948548f2924b7a143169e52e730b (13e98KSH...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(54) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x8b71cb9633ed80308115086854c18c5a14d6183a81afd1e737558c47f5824220 err=Network(DialFailure) traceID=185353550188562516562191966109383625818
Some network error occurred when fetching erasure chunk origin=Public(06f32ceccfb0945e11606f4d40055d5ec977079f6f334030e6768e428a8cb932 (1A7XUqip...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(56) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xe856ee1e045eea1ca4a2e916bacd70f420083f85c48bc52a44db3feecc58eab8 err=Network(DialFailure) traceID=308832262143617706843686169551926292724
Some network error occurred when fetching erasure chunk origin=Public(ca97c357c3af258b5a17f06470aee89175d6834e5cd705ebb6ecabe3c6c3a81f (15adkqMw...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(57) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x049b71d75c0fe4d77dae4622a8612c4cc1404bbc68a137cda53fb7564462f1fb err=Network(DialFailure) traceID=6124026971044550445150232725814324300
Some network error occurred when fetching erasure chunk origin=Public(00c82090e61c34b4675c01815b7e25a1fd09d2fc60f60ec1f3f435096790681c (1122T7UK...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(53) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xc0a0389605c03b503ceaf079c96062b2d6c9f5f00cd5ab4313172efb67ed7ca9 err=Network(DialFailure) traceID=256043690389011420557514549607851385522
Some network error occurred when fetching erasure chunk origin=Public(f4bc1b9c425884a67ac27673e0ed56437eec77bb7d34803df0a5b94d8924be5e (16XtZgri...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(55) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xea7b2957a9b549b401dfa38302b71858cb6777cc7f2dbfa24d54438472bebf7b err=Network(DialFailure) traceID=311678842051436116803744609173576816728
Some network error occurred when fetching erasure chunk origin=Public(d4438a5465496aecd6bbf65dbf90b48e32368969e2b2180508e142d18b740431 (15oKDV3P...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(0) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xbffec3796f2c9e901b2ce1aa5eb9f4a09f0e7f12f74e54a837b0e93461148a3b err=Network(DialFailure) traceID=255205355287873931862417387504092705952
Some network error occurred when fetching erasure chunk origin=Public(8889c38e92b6f1819fa1ea474c1364beeac14d1bcea9f8c292aec350eaa15825 (1462SLD7...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(1) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x9b44bb30de2690e32962eb2bf27347f6169de2040da04393987db7f7449113ae err=Network(DialFailure) traceID=206387212215342307698538254323716147190
Some network error occurred when fetching erasure chunk origin=Public(2c596de56fe273fe68d079f638ce81c412dc6451f18e90d9b2d868bacdecaf13 (1219g2t5...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(8) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xa2a4e4082157db2c9f32c503b61c6fae845e8f0ddd91da7d78f143274a539b83 err=Network(DialFailure) traceID=216191097035490232728265901407288979374
Some network error occurred when fetching erasure chunk origin=Public(0849872804f7d5257fe7fa5a4293df4f9e4f900d2046fb5289cb48c97963155a (1BsE5PXs...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(2) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xf07fe19981e00bc7b26589f161ddb9519b5188878d4e77fb7f4c4a11459e63d4 err=Network(DialFailure) traceID=319678716393677840194361512861007460689
Some network error occurred when fetching erasure chunk origin=Public(5cfac09f57b19f42da147304cdc6bd26ae9fd4e292e53081fe77fb4114882d0d (136utsqs...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(11) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xcef780a4dabcb49d97560062889b591bea8cf44cb23ab20819fe015d3ae725af err=Network(DialFailure) traceID=275106073665294555123748332946748823835
Some network error occurred when fetching erasure chunk origin=Public(20a35d0b24146651f788f040bd82d0906f51d9c851f67af35bb9c53d914f1d47 (1jo4ZXMg...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(12) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x5e4faddf87b7a673f2b7acbf8ad2f04f33d20774cc3b3a5cf465b9093e855e7c err=Network(DialFailure) traceID=125361149622350511658114822982239187023
Some network error occurred when fetching erasure chunk origin=Public(ce37de7be55a1d749ccf7592f017656ca7eede86aeb7cdc78ce5f132d363a967 (15fPTjEr...)) relay_parent=0x35895d1fbb335fc39c239e0a6e50941d2e89dfaf4d4ea914f8f0234779109e49 group_index=GroupIndex(4) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x7fb7049905a1bd977db7e7609fcd18cb90664cdf97d1833f8d5d9ff3921bba3d err=Network(DialFailure) traceID=169762239043086426031777186796582803659
Some network error occurred when fetching erasure chunk origin=Public(ac35b0f0a9500227a1e1220d3db5a0ec6c3fa21b585097836a3addd58576dc43 (14toBmxg...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(26) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xd5b307f52643500ee2907ab52b9bc64cb7b2aafaf3865d54002bf400ad631950 err=Network(DialFailure) traceID=284055145639473663097497563669225784908
Some network error occurred when fetching erasure chunk origin=Public(be512d17b3071eba0938be174d5db0d38ddfaa17ff65278fd036874ad033507a (15JYDKfc...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(13) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x875d83d41e808c64af99044494de4c5e0ffed98217cba481dd3fe5fdc94292ef err=Network(DialFailure) traceID=179931336840275868992518579255766830174
Some network error occurred when fetching erasure chunk origin=Public(965eb1f5d9f3ebf949e1a8faf9706205890be6357fe19681cb29f23301c78855 (14QAKA5W...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(19) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x9cc3c423a9de07d55c41c8999e6348c28a24277dc8baa9c25c07d3cddf7f5ec1 err=NotConnected traceID=208376043407700556471343704789234174146
Some network error occurred when fetching erasure chunk origin=Public(e2cc82a1ccf4df91a4ec36a8dd0614c3f8ac285833c03435783f8170db2dbc2a (168NaJZh...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(15) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x92ef4638078ae2ace8353e3dfd203e309998e1c7883df71b0a8cde34e3072431 err=NotConnected traceID=195309670541571194939160892921234865712
Some network error occurred when fetching erasure chunk origin=Public(429396b5a4246ab31f6cfed3dd993d402576e93dfbd3268d6c99f7ce852b1a0d (12WJ1CRP...)) relay_parent=0x35895d1fbb335fc39c239e0a6e50941d2e89dfaf4d4ea914f8f0234779109e49 group_index=GroupIndex(27) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x74e07b87bd5e02920f0f6477cfd3e4f13c58fdb8a5b39a17d6f725a0320bfb89 err=Network(DialFailure) traceID=155356027498151549249100960943867946225
Some network error occurred when fetching erasure chunk origin=Public(f69b5dee72c16fe8087cf2004c774d4e9b838b274eaa5b28669fbba2edda6a76 (16aLw8Tr...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(20) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x6016f47293b3e2951eda81b7fb873cb80b1be442fa841bba5f0397dce2de7d2a err=Network(DialFailure) traceID=127725076111905271811373048740096457912
Some network error occurred when fetching erasure chunk origin=Public(e09798083b9dcdf5bdf0bfed26cf73aed63ea6e5ab81594907043b4438eb941b (165Um29S...)) relay_parent=0x35895d1fbb335fc39c239e0a6e50941d2e89dfaf4d4ea914f8f0234779109e49 group_index=GroupIndex(42) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x8486297f52e2aaf968560f987d90c6045edd5c9a4da27c41852502746c8ec90c err=Network(DialFailure) traceID=176154704889074746599226652480343426564
Some network error occurred when fetching erasure chunk origin=Public(1acd78bd3af19f23d9ef21ac20d72bcdb38e209b55638606b69393d9c7bad50e (1c9HQS9k...)) relay_parent=0x35895d1fbb335fc39c239e0a6e50941d2e89dfaf4d4ea914f8f0234779109e49 group_index=GroupIndex(28) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xdc83026c22c648e47772bf89166d935431c2749cf60c1792639ba4032460a659 err=Network(DialFailure) traceID=293110399093372515173125353044886524756
Some network error occurred when fetching erasure chunk origin=Public(b69a299d39ea23163447601bfb62d5066c7aa64744e12a1c4671aac7cfe94342 (158RWdY6...)) relay_parent=0x35895d1fbb335fc39c239e0a6e50941d2e89dfaf4d4ea914f8f0234779109e49 group_index=GroupIndex(58) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x4097f412fa55a7686df06545990e4d1500dc9cfc7b162f84903d5e54118f0c29 err=NotConnected traceID=85859578967398392813682753268488228117
Some network error occurred when fetching erasure chunk origin=Public(36401053949d148a8c6eb22f390661ac3a3d648f2ac89bde4ab3b97466db3439 (12E8cmYi...)) relay_parent=0x35895d1fbb335fc39c239e0a6e50941d2e89dfaf4d4ea914f8f0234779109e49 group_index=GroupIndex(7) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xce8df1d924253f0b3ad5eee9b5abe93e5162a933d095fa405705c43f73077e78 err=Network(DialFailure) traceID=274557986253158314486876810150559869246
Some network error occurred when fetching erasure chunk origin=Public(ce37de7be55a1d749ccf7592f017656ca7eede86aeb7cdc78ce5f132d363a967 (15fPTjEr...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(4) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x651e707ecf0b5faea537bb7264d0cc42a1cfbf6356cf085fe4284a7b8c2cb1c5 err=Network(DialFailure) traceID=134410078156733780840023168685551176770
Some network error occurred when fetching erasure chunk origin=Public(82cf56b553fbf198cdc2ce5a13ecac83f5be74fca9f91cacf3bcca9fa091054c (13xWpQyc...)) relay_parent=0x35895d1fbb335fc39c239e0a6e50941d2e89dfaf4d4ea914f8f0234779109e49 group_index=GroupIndex(23) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x234277057ae2017480e71639757e3eb079e96d17a283bab1c84f26df8a0469be err=Network(DialFailure) traceID=46868085486049396600132614748538551984
Some network error occurred when fetching erasure chunk origin=Public(94e02c728aa186e3e3d14808b24781d46078618655940cbea7f9964fbbfe676f (14NCgTua...)) relay_parent=0x35895d1fbb335fc39c239e0a6e50941d2e89dfaf4d4ea914f8f0234779109e49 group_index=GroupIndex(14) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x50da397cf8af2b9c420f2b7c01bdb0961ae2ed73e689de51492592c19ecb67c6 err=Network(DialFailure) traceID=107471326376557472567653286800483266710
Some network error occurred when fetching erasure chunk origin=Public(429396b5a4246ab31f6cfed3dd993d402576e93dfbd3268d6c99f7ce852b1a0d (12WJ1CRP...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(27) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xc9b820a7e1aaaf007b00bbe2266e2f228961c14dc1edd00403f9df1cebe27e1b err=Network(DialFailure) traceID=268130872112789426495047695132141694754
Some network error occurred when fetching erasure chunk origin=Public(3e926927e11612453e0ea23ec75ae01f22c2bd27151932e30b8276ea8b406831 (12R3Tnwn...)) relay_parent=0x35895d1fbb335fc39c239e0a6e50941d2e89dfaf4d4ea914f8f0234779109e49 group_index=GroupIndex(24) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x974a2aed85ca9c0d70bec4f488e75fda99a484bbc9806ae7455c17328379b6da err=Network(DialFailure) traceID=201098528010738189562696135474099478490
Some network error occurred when fetching erasure chunk origin=Public(1acd78bd3af19f23d9ef21ac20d72bcdb38e209b55638606b69393d9c7bad50e (1c9HQS9k...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(28) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x3613b6e788cea739056725a3a03534e3145c115ec1f1c5b8e021c19b41de9643 err=Network(DialFailure) traceID=71880675155290814002426718860023772387
Some network error occurred when fetching erasure chunk origin=Public(a26bec03369eeaa592431cea6971af771739a8fd02d16d87f5f82fce1841826d (14fxpNpM...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(38) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x13329b448cf9e57d634ed8a0276f06a585a8d1c7aee6a1098e1b85ee6f9ee18f err=NotConnected traceID=25518095967473767584485360835450832549
Some network error occurred when fetching erasure chunk origin=Public(9c8adc305a080e5fb7c2eddcdff24d3695084af62694f139d59aca72d957bd32 (14YFimMx...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(39) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xae1522cf54f9b10a6ef4de457ece3df324f4bbd0810b758ef985d3184f1ec550 err=Network(DialFailure) traceID=231395415529059357280381061712067837427
Some network error occurred when fetching erasure chunk origin=Public(8c4e50094346465878b69416bcb22e751a9bf159a6430242cd0a1bcc28176b03 (14Axy7uz...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(40) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x36290213a99be04a71817de91df9ba062abc98bee9ed6493a929d31201e9db63 err=Network(DialFailure) traceID=71991238066231089489116359006541691398
Some network error occurred when fetching erasure chunk origin=Public(3200bdf6ef0b2996121eec3b6305fa98a959eeb715389e1526c53c2df6043671 (128ZcekU...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(45) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xd4b0a456094f6b8afdee4971f7c1e4a6f45aebd8b26d2933b10f3fdfb8fca0bf err=Network(DialFailure) traceID=282713512485182647904214678164536943782
Some network error occurred when fetching erasure chunk origin=Public(705675abc0fd197435e23a2962f73fd2131f30281fe3d27b6d0d3b07eda75939 (13YJ3j16...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(46) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xf9360aecb715614a77e6a5682c4c77e5f4e57ef43ba3f9c814e89d7f2c2d885a err=Network(DialFailure) traceID=331258376559408926610842245761860401125
Some network error occurred when fetching erasure chunk origin=Public(e29c2b033f77fce6a102fbb891ea074b60774992a966a353753c66ccb6621661 (1688DPLD...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(47) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xc217d2e745af277eb1fdfe619c3015e7142b92318657e767a9b7e3bf9f1b1371 err=Network(DialFailure) traceID=257993931639308502230159426201538532839
Some network error occurred when fetching erasure chunk origin=Public(52b33dc90d7a7032e235c4e484eacd53ca5fc6a38ce93cc062d1890200f9cc70 (12sSAznD...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(51) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xc4805015465a0050d23a3d821258b574c9d28add1504a97665d514b2694e851d err=Network(DialFailure) traceID=261194925450068429460930577848312837492
Some network error occurred when fetching erasure chunk origin=Public(74cbe6e914ea9faeb93af966ba5e96936a08948548f2924b7a143169e52e730b (13e98KSH...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(54) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x00a6f589d8a5bbde55affd5f81a3f68740b332f530122259c54885f81a3d8570 err=Network(DialFailure) traceID=866901390177062565335041629078419079
Some network error occurred when fetching erasure chunk origin=Public(06f32ceccfb0945e11606f4d40055d5ec977079f6f334030e6768e428a8cb932 (1A7XUqip...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(56) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x3e5ae837f72fdcbdb442527f9cc8880d9bfa05ecf12858310835b0d1254de4b0 err=Network(DialFailure) traceID=82884152409010563438124721596845819917
Some network error occurred when fetching erasure chunk origin=Public(f4bc1b9c425884a67ac27673e0ed56437eec77bb7d34803df0a5b94d8924be5e (16XtZgri...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(55) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x6321d355a73ee32b247435cc9bd06d0d52936479911a5b906813dfd38cfeb045 err=Network(DialFailure) traceID=131769203753618527758155677368287522061
Some network error occurred when fetching erasure chunk origin=Public(d4438a5465496aecd6bbf65dbf90b48e32368969e2b2180508e142d18b740431 (15oKDV3P...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(0) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xe7fc5fa0f6a753f86867ad591911e016fc2a8031e890af075174f26fa79d300e err=Network(DialFailure) traceID=308362065416420291960813716695350894614
Some network error occurred when fetching erasure chunk origin=Public(662326567f87037455978244636309d24302b5f8efb62ab76ff8a020d55b1732 (13JvKuKm...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(48) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x71d126d64df426eb8c2c40cbd491854eb86119daf264ef304efe4f1a12678966 err=Network(DialFailure) traceID=151288741277646496879778692045613401422
Some network error occurred when fetching erasure chunk origin=Public(ca97c357c3af258b5a17f06470aee89175d6834e5cd705ebb6ecabe3c6c3a81f (15adkqMw...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(57) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xb7570bee284f7d22cf3f64377c1859d970aa0cb3b3e3289e0ccda52bee606d4e err=Network(DialFailure) traceID=243700695030615949778902636639590832601
Some network error occurred when fetching erasure chunk origin=Public(8889c38e92b6f1819fa1ea474c1364beeac14d1bcea9f8c292aec350eaa15825 (1462SLD7...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(1) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x9b232e0116a5ab512047b8fd94b94b11231fcc58af1a51cbdfbba9ed630b37b3 err=Network(DialFailure) traceID=206213002813789561755679378019477703441
Some network error occurred when fetching erasure chunk origin=Public(ce37de7be55a1d749ccf7592f017656ca7eede86aeb7cdc78ce5f132d363a967 (15fPTjEr...)) relay_parent=0x313405c8380fc4210709e1610803f3cb20080cdb58d8fbb7cda2eb1b2fb26f07 group_index=GroupIndex(4) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xfd60c4cfb1c0bedab43d93f5328020dd300201b925fb63097f4c4ac6ccc74362 err=Network(DialFailure) traceID=336797135239526877256876828272214679773
Some network error occurred when fetching erasure chunk origin=Public(2c596de56fe273fe68d079f638ce81c412dc6451f18e90d9b2d868bacdecaf13 (1219g2t5...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(8) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x15ba4b9d9ab3c5e826fdce2fd306727046a51463dfecd3c7c5077258fd524f61 err=Network(DialFailure) traceID=28881088794590523707518956887328846448
Some network error occurred when fetching erasure chunk origin=Public(0849872804f7d5257fe7fa5a4293df4f9e4f900d2046fb5289cb48c97963155a (1BsE5PXs...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(2) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x40cc33cc8c9cc7c9f4a41b3c100f421e2433fa5c438aaf4a702f49b0acf3e081 err=Network(DialFailure) traceID=86130870898328097126382449935841706526
Some network error occurred when fetching erasure chunk origin=Public(5cfac09f57b19f42da147304cdc6bd26ae9fd4e292e53081fe77fb4114882d0d (136utsqs...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(11) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xfeb6807b9ed237cd7045ca43bc47b81645acbba8447db91ccd323b85d94cbb08 err=Network(DialFailure) traceID=338571514900267996508008113521256019990
Some network error occurred when fetching erasure chunk origin=Public(20a35d0b24146651f788f040bd82d0906f51d9c851f67af35bb9c53d914f1d47 (1jo4ZXMg...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(12) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x61b436247fd355c90cb8ad5c1836115934b7921b4b2a6b5c4322296b1fbaa5bb err=Network(DialFailure) traceID=129870827167565641053636607060202623321
Some network error occurred when fetching erasure chunk origin=Public(be512d17b3071eba0938be174d5db0d38ddfaa17ff65278fd036874ad033507a (15JYDKfc...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(13) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xb519993107aeeae306103fc10ad00a8d384eccdae830408a8d907a7705b9d274 err=Network(DialFailure) traceID=240723181751760322973796922780791343757
Some network error occurred when fetching erasure chunk origin=Public(e2cc82a1ccf4df91a4ec36a8dd0614c3f8ac285833c03435783f8170db2dbc2a (168NaJZh...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(15) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x5724031051c1d45e8127b9567a44a370fc2ef351b3348dc1e542a0ce36b9f893 err=NotConnected traceID=115829820460376957284208481677661741936
Some network error occurred when fetching erasure chunk origin=Public(429396b5a4246ab31f6cfed3dd993d402576e93dfbd3268d6c99f7ce852b1a0d (12WJ1CRP...)) relay_parent=0x313405c8380fc4210709e1610803f3cb20080cdb58d8fbb7cda2eb1b2fb26f07 group_index=GroupIndex(27) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x680c91faa13c8ea3de5100dc08b4df0fa7a0f563f9fa6c9c8007f4e1f6d47274 err=Network(DialFailure) traceID=138304979930267122501127042576552353551
Some network error occurred when fetching erasure chunk origin=Public(965eb1f5d9f3ebf949e1a8faf9706205890be6357fe19681cb29f23301c78855 (14QAKA5W...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(19) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x3012fb10b12c7619f98a8290876912f5e18157fa217d26d8fcc893a358a6c722 err=NotConnected traceID=63901497348423302787038129412494660341
Some network error occurred when fetching erasure chunk origin=Public(e09798083b9dcdf5bdf0bfed26cf73aed63ea6e5ab81594907043b4438eb941b (165Um29S...)) relay_parent=0x313405c8380fc4210709e1610803f3cb20080cdb58d8fbb7cda2eb1b2fb26f07 group_index=GroupIndex(42) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x25a25533e0fd1b15de44976600096eb745c881f127fb05680c4223e326f4faba err=Network(DialFailure) traceID=50024316050207756332366606665859559095
Some network error occurred when fetching erasure chunk origin=Public(f69b5dee72c16fe8087cf2004c774d4e9b838b274eaa5b28669fbba2edda6a76 (16aLw8Tr...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(20) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x8a8b34426829bb0492fe98f00fba1c406ef998215c1fa8b0c97973380b14809c err=Network(DialFailure) traceID=184156252628249737476334871131895503936
Some network error occurred when fetching erasure chunk origin=Public(1acd78bd3af19f23d9ef21ac20d72bcdb38e209b55638606b69393d9c7bad50e (1c9HQS9k...)) relay_parent=0x313405c8380fc4210709e1610803f3cb20080cdb58d8fbb7cda2eb1b2fb26f07 group_index=GroupIndex(28) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x0ecedf5a711182a7ca952eb1de5d79240e4b8c559b7d86f2ce56af9824471e7a err=Network(DialFailure) traceID=19683335236716212170017604677645728036
Some network error occurred when fetching erasure chunk origin=Public(00c82090e61c34b4675c01815b7e25a1fd09d2fc60f60ec1f3f435096790681c (1122T7UK...)) relay_parent=0x313405c8380fc4210709e1610803f3cb20080cdb58d8fbb7cda2eb1b2fb26f07 group_index=GroupIndex(53) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x97f6e95dfbb9329cb11fc74240331997cda54e4626018eb8c0b7562b25d78a03 err=Network(DialFailure) traceID=201995465638283256821451220692998756759
Some network error occurred when fetching erasure chunk origin=Public(ac35b0f0a9500227a1e1220d3db5a0ec6c3fa21b585097836a3addd58576dc43 (14toBmxg...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(26) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x78b755d91ff0ba0f33e05e37f2afb16983bb450730940a633063a89f9b2202f3 err=Network(DialFailure) traceID=160459291026514410333599495654441202025
Some network error occurred when fetching erasure chunk origin=Public(b69a299d39ea23163447601bfb62d5066c7aa64744e12a1c4671aac7cfe94342 (158RWdY6...)) relay_parent=0x313405c8380fc4210709e1610803f3cb20080cdb58d8fbb7cda2eb1b2fb26f07 group_index=GroupIndex(58) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x9edbadead192cb6c020889afcabcf17a8d47083b4e6e7b69cf43451bc1aa4104 err=NotConnected traceID=211158663807147126101403790081750004090
Some network error occurred when fetching erasure chunk origin=Public(ce37de7be55a1d749ccf7592f017656ca7eede86aeb7cdc78ce5f132d363a967 (15fPTjEr...)) relay_parent=0x2a2c53231262a65294d3627ff1efbaa1a6811e5eaa7aef68fabeb2f65508190f group_index=GroupIndex(4) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x89dffba9afdade146a048ae4750d8b9959e0d996f5e14a1135925996f62a421d err=Network(DialFailure) traceID=183267221950781193918278170312922991513
Some network error occurred when fetching erasure chunk origin=Public(36401053949d148a8c6eb22f390661ac3a3d648f2ac89bde4ab3b97466db3439 (12E8cmYi...)) relay_parent=0x313405c8380fc4210709e1610803f3cb20080cdb58d8fbb7cda2eb1b2fb26f07 group_index=GroupIndex(7) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xa1153130a84d4f6b49df6d012d84fda1c8b5ffdacc631163be0033c4352d2d92 err=Network(DialFailure) traceID=214115743248510010518424208750982528417
Some network error occurred when fetching erasure chunk origin=Public(429396b5a4246ab31f6cfed3dd993d402576e93dfbd3268d6c99f7ce852b1a0d (12WJ1CRP...)) relay_parent=0x2a2c53231262a65294d3627ff1efbaa1a6811e5eaa7aef68fabeb2f65508190f group_index=GroupIndex(27) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xffe4f246f9b3ca4d3354bb4c16cfb0d4c9b8161e3d258a764d6b40fca023c626 err=Network(DialFailure) traceID=340141896575274068501875541525601956052
Some network error occurred when fetching erasure chunk origin=Public(82cf56b553fbf198cdc2ce5a13ecac83f5be74fca9f91cacf3bcca9fa091054c (13xWpQyc...)) relay_parent=0x313405c8380fc4210709e1610803f3cb20080cdb58d8fbb7cda2eb1b2fb26f07 group_index=GroupIndex(23) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x0fb38485faf9198ed40e762ea045722887c4d7521046eab220d2e8fd81165780 err=Network(DialFailure) traceID=20870528967537164335017246307578638888
Some network error occurred when fetching erasure chunk origin=Public(1acd78bd3af19f23d9ef21ac20d72bcdb38e209b55638606b69393d9c7bad50e (1c9HQS9k...)) relay_parent=0x2a2c53231262a65294d3627ff1efbaa1a6811e5eaa7aef68fabeb2f65508190f group_index=GroupIndex(28) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x531609025602fae233d5fdacbc36c540b9c16364bbfe3310386b5a0393f99e94 err=Network(DialFailure) traceID=110440336907797855013001101257511191872
Some network error occurred when fetching erasure chunk origin=Public(1ccca873bed5bccb40ebc740f2d646c3f23bc649c3c062e2ee02447f5e2c1c76 (1em8wY7z...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(0) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xbffec3796f2c9e901b2ce1aa5eb9f4a09f0e7f12f74e54a837b0e93461148a3b err=NotConnected traceID=255205355287873931862417387504092705952
Some network error occurred when fetching erasure chunk origin=Public(be45a812ff9897601def6b8a669e2b4ce91c2d3a500a15171511c1bc2102653b (15JUnqwA...)) relay_parent=0xccb73c963b0775f5ba2750080380f83ca58b082f87c3f295cbbf87627535ffd3 group_index=GroupIndex(1) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x9b44bb30de2690e32962eb2bf27347f6169de2040da04393987db7f7449113ae err=NotConnected traceID=206387212215342307698538254323716147190
Some network error occurred when fetching erasure chunk origin=Public(1ccca873bed5bccb40ebc740f2d646c3f23bc649c3c062e2ee02447f5e2c1c76 (1em8wY7z...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(0) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0xe7fc5fa0f6a753f86867ad591911e016fc2a8031e890af075174f26fa79d300e err=NotConnected traceID=308362065416420291960813716695350894614
Some network error occurred when fetching erasure chunk origin=Public(be45a812ff9897601def6b8a669e2b4ce91c2d3a500a15171511c1bc2102653b (15JUnqwA...)) relay_parent=0xb9e2544d325ce9c2a36c1468942145a3701e77d5579dfb487e7dcf6fc396d714 group_index=GroupIndex(1) session_index=8977 chunk_index=ValidatorIndex(263) candidate_hash=0x9b232e0116a5ab512047b8fd94b94b11231fcc58af1a51cbdfbba9ed630b37b3 err=NotConnected traceID=206213002813789561755679378019477703441
Node uses the following topology indices index_in_gossip_topology=Some(ValidatorIndex(263)) index_in_parachain_authorities=Some(Active(ValidatorIndex(263)))
Sending fatal alert BadCertificate
Sending fatal alert BadCertificate
⏩ Block history, #2633152 (6 peers), best: #21447872 (0xf51d…705a), finalized #21447857 (0xf699…2702), ⬇ 2.4MiB/s ⬆ 364.0kiB/s
🥩 BEEFY pallet available: block 21447869 beefy genesis 20351234
Candidate included without being backed? candidate_hash=0x9fc4e97bfa04f9ccd1825b6fad0a9a09501d9b6d5ff3136a68af36e7726019d3 traceID=21236967713795335790672108429296026266
Candidate included without being backed? candidate_hash=0xb8b94c54aeb40c972822392843436a920b2a65e14d35861f9c0120ef92dbf945 traceID=24554007431561705026392798029253780545
Candidate included without being backed? candidate_hash=0x9e4a06e39ea56e55fb5935f3dc40224bbef4d97cc52be6ced2892b42f261b3f6 traceID=21040239302989742316974681296720416826
Candidate included without being backed? candidate_hash=0x901f97bf79781b5db6eeffe3a99855bfa1860d2f193211e65e57f074fe88b567 traceID=19157287040966454350071565813311447596
Candidate included without being backed? candidate_hash=0x516d1feda12bd793ced3481e5228c6ca0e857008f9150c20954fe4b03db086ba traceID=10823407559781079942599786383248987105
Candidate included without being backed? candidate_hash=0x1c71a55724cf9f05bea7595dc4e987ff48b3a29c4c2ffcdc1ef8101dcfba6920 traceID=37808466928819280190419060510817880063
Candidate included without being backed? candidate_hash=0xc49c30e3fbd041140c3ac91509fb664ec9930224b5e9cfb2f62b1e80d006b106 traceID=26133967710216131157665688533353649927
Candidate included without being backed? candidate_hash=0x4efdd92ae0458f25fa251d77052bac2cc431dcc4636335073d917a64b7543c7c traceID=10499783945630830156859547313010453201
Candidate included without being backed? candidate_hash=0x60fb8e61421335f4d19fa991b4bf356ae7cfaa7157f657cd25e824ff8d79b1ff traceID=12891204191458888299468315613396672241
Candidate included without being backed? candidate_hash=0x62bf0d2bae93ea70318916731d18ac420e991cadb39f35e3bdd10fe30c9b7ec8 traceID=13125633941906695411446357360558792198
Candidate included without being backed? candidate_hash=0x604209603aa0766c403a9774c8a113c553be5d783c9b9a954a6b94cc8f851aed traceID=12794876935374937459355658637779409811
Candidate included without being backed? candidate_hash=0x912eec782dc23d9266cf17d258df4cfa3fd6e101237d9fc383aa0bd7f46e94aa traceID=19298170121451301496385901141508170060
Candidate included without being backed? candidate_hash=0x3f1ff6e50a6823670c4c769e06f0e71cd2ca02a69a5e4de24a55bd9e9bc3153c traceID=83907332556296739294063015928110573340
Candidate included without being backed? candidate_hash=0x725c841073af747c02b5f8eaa116b1e0d28ebdfa606681de450de92e63522476 traceID=15201236141198678413157502994134222076
Candidate included without being backed? candidate_hash=0x215281bf3b42e418156972bdaff4b6443ac041eb178beb4f2c0dd3a3019e135a traceID=44292923785060472788715070385366021700
Candidate included without being backed? candidate_hash=0xed116bc350f886af2a4119dae199b03a3ed84965cdcb25756c01a849d98699c0 traceID=31511748973999868537547417903689378617
Candidate included without being backed? candidate_hash=0x65b1a2c5d26a3ed65611b3c78b5b3966d37d11bc9c02d71cc55a0f0288d6a9c0 traceID=13517436554166126947410631696950724029
Candidate included without being backed? candidate_hash=0x3f99a299dec1dcfb1b77bd044265f650f6a8b2c1b243e1ddd6156c0aa7121e17 traceID=84539083095010223289419822900873393744
Candidate included without being backed? candidate_hash=0x756e404c8c432cb4715daaa35fdf98f8693cf8dfda9f4ef5262a39268ff0479c traceID=15609213230023808347749111260850523980
Candidate included without being backed? candidate_hash=0x3e3337493502f1b274a19f47057a1fb0a10094b1f5ca0ef168bf5cbc6f155be0 traceID=82678064211040389466678562590293630896
Candidate included without being backed? candidate_hash=0xd7900ec2f1c5f209a7a12ad4798e273783c7097bcae167366c8b902db54c6451 traceID=28653200924020909548769111523598835077
Candidate included without being backed? candidate_hash=0x3e1f9e98ef3b3a40869c5a8e494150bb2259687fdd2ecff54c183ea5042a5a7b traceID=82576313678715961966832869975944941755
Candidate included without being backed? candidate_hash=0x12492d99a69e28cb4515f2919a699e3e5eae653e3c448a0b0475463379ff707e traceID=24306066476708271690932888992679042622
Candidate included without being backed? candidate_hash=0x35fea988ed384182eb63c531f369aefdcb66f785c5b138a267ca656ce550b790 traceID=71771365754337463096217014140597284605
Candidate included without being backed? candidate_hash=0xd843667366fd957d20deca440795d5cd1ef750abcd2bbc4a4c638500398f8b41 traceID=28746320892795595878407314198133406663
Candidate included without being backed? candidate_hash=0x7a4e26b707aecb16f51f05ee52a695a1f22cbe659ebc1ac52873c3696c8576b0 traceID=16257159987342183914838932558375122268
Candidate included without being backed? candidate_hash=0x5008af4ce57806b65086ac3e54762846e84d4a8a93a8fdc7364eb712e4c4611c traceID=10638333355169970862449726609694432672
Candidate included without being backed? candidate_hash=0x54390c0bdf5fbf57ad8ae4c371de7c3d1d24650797fa6cadc44e5df8328af893 traceID=11195135689642535870809219370593120978
Candidate included without being backed? candidate_hash=0x7e8baf476ec036a34835f8d8f8e66854e5da073b1eb7670b16bcf184c80951bc traceID=16820801181339164143886425238771744162
Candidate included without being backed? candidate_hash=0x7810994c4e9063c028da34944de6a1224a73773eb931425b5a9cb0adfc67ba5a traceID=15959354549825055892326766754050333929
Candidate included without being backed? candidate_hash=0x418ee8ebcc992b9f6deaed5c8f781fb0c3121021e13c7e74360e086f01879e33 traceID=87141850080897828958323386916730838960
Candidate included without being backed? candidate_hash=0xc73039716d594dbc2e7afc7ab7580e522ca8209dd2a66afbb80913566c5b2c49 traceID=26476676649437953025314003006790356539
Candidate included without being backed? candidate_hash=0x13fa9c1027b710c80f0379438fb57e9d09be99efbfee4eca7fa03cb9213744f5 traceID=26556571470387106437436812637586685597
Candidate included without being backed? candidate_hash=0x1819e0fda1dda96763ab30049eb7bfe8b6857ca8a0a9a10abf6567ac71aa733b traceID=32035842674872744897960527707464253416

5

12D3KooWHRd8ANXyUvkRNJiXsSUzB2pSaE7QvVATdsTNRCjRvdiP

alexggh commented 4 months ago

Candidate included without being backed Some network error occurred when fetching erasure chunk

This two errors are red-herring they are kind of expected on restarts, so they can be ignored.

First time troubleshooting various things after like 2 hours, when I set the --no-private-ip flag, but then the next session it started missing all again. Then I tried various things again after ~2-3 hours it started producing blocks and has been good since then.

Bare in mind that propagating node addresses and IDs in the network is not instantaneous(because of DHT), so that might explain what you noticed, did your node change reported IP/Ports between restarts ? This is what I'm seeing now for it:

authority="12ePg7VRj9AM1euhf2ZR3oxG7NkQGPhHhQGaHnsSv72zeYJH" peer_id=PeerId("12D3KooWHRd8ANXyUvkRNJiXsSUzB2pSaE7QvVATdsTNRCjRvdiP") addresses="Parity Polkadot/v1.13.0-d5160c1d567 (Amforc 3)" version={"/ip4/91.90.166.20/tcp/30333/p2p/12D3KooWHRd8ANXyUvkRNJiXsSUzB2pSaE7QvVATdsTNRCjRvdiP"}

Also, it would help a lot if you could provide the full logs.

tugytur commented 4 months ago

Bare in mind that propagating node addresses and IDs in the network is not instantaneous(because of DHT), so that might explain what you noticed, did your node change reported IP/Ports between restarts ? This is what I'm seeing now for it:

But the first time it joined the active set the validator had a uptime of 7 days. After 4-6 of hours I assigned a new public IP to the host, thought maybe for a reason that IP got banned or so. Port stayed the same.

Logs for the last 24h are attached valdiator3_last_24h.txt

alexggh commented 4 months ago

But the first time it joined the active set the validator had a uptime of 7 days. After 4-6 of hours I assigned a new public IP to the host, thought maybe for a reason that IP got banned or so. Port stayed the same.

Ok, I think I have a theory of what happened here:

  1. You enter active set on session 8976 at 15:36:21 and your validator is in group 7

    15:36:21 Node uses the following topology indices index_in_gossip_topology=Some(ValidatorIndex(40)) index_in_parachain_authorities=Some(Active(ValidatorIndex(40)))

    Because of https://github.com/paritytech/polkadot-sdk/issues/4324, your validator is slow on backing stuff, because it needs to precompile stuff. One thing to note here is that you are in group 7 and we have 44 parachains producing blocks with 59 total groups, because each parachain is rotated on groups every 1 minute, after 7 minute even if it caught up on work it would've passed another 15 minutes(59 groups - 44 paras), until your group would've been assigned to another parachain to get points.

  2. However, your validator got restarted at 15:46:19, now your validator would have to do the PVF precompilation again because it does that at restart and re-discover all validators, so it won't be ready to back things right away, it got restarted again at 15:57:48, 16:40:56, 17:06:15, but despite those restart that won't help, you still got 11680 out of 18420 on session 8976, so it started performing well once restart cycles stopped.

  3. Next session at 8977 at 19:36:22 your validator is in group 52, so the first 8 minute it wouldn't have produced any points because its groups don't get assigned parachains, not sure how fast you decided to again restart the validator but it enters again a big cycle of restarts between 20:13:12 and 21:33:30, I would assume it is you trying to fix it, you changed your public IP as well at 21:30(Changing your IP while in the active set is kind of unsafe, because the old record might still be present in the DHT for another 36h, but I think you got lucky here, because your node seems to be performing well now). You still get 6,900 out of 18000 points there.

  4. Once your validator is not restarted anymore, validator seems to start performing well.

Overall, I think this was a bit unfortunate, things are in progress on fixing https://github.com/paritytech/polkadot-sdk/issues/4324, so that the initial cause that triggered you to restart the validator and try various fixes won't happen anymore, till then I would always recommend waiting around 20 minutes and always try to compare the relative performance with your peers, before concurring something is off with the validator and restart it.

tugytur commented 4 months ago

@alexggh thanks. I'll close the ticket.