search

ChainSafe / lodestar

🌟 TypeScript Implementation of Ethereum Consensus
https://lodestar.chainsafe.io
Apache License 2.0
1.15k stars 283 forks source link

fix: fix uncaught exception on poll validator indices causing sync and attestation duties failure for that epoch #6888

Closed g11tech closed 3 months ago

g11tech commented 3 months ago

fixes the following scenario where validator doesn't even perform attestations and sync duties for the indices it already knows for the epoch where this api errors

un-15 18:49:36.030[]                error: uncaughtException: Can not fetch state validators from beacon node - Internal Server Error: Internal server error
Error: Can not fetch state validators from beacon node - Internal Server Error: Internal server error
    at Function.assert (file:///usr/app/packages/api/src/utils/client/httpClient.ts:44:13)
    at IndicesService.fetchValidatorIndices (file:///usr/app/packages/validator/src/services/indices.ts:130:14)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at IndicesService.pollValidatorIndicesInternal (file:///usr/app/packages/validator/src/services/indices.ts:119:35) - Can not fetch state validators from beacon node - Internal Server Error: Internal server error
Error: Can not fetch state validators from beacon node - Internal Server Error: Internal server error
    at Function.assert (file:///usr/app/packages/api/src/utils/client/httpClient.ts:44:13)
    at IndicesService.fetchValidatorIndices (file:///usr/app/packages/validator/src/services/indices.ts:130:14)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at IndicesService.pollValidatorIndicesInternal (file:///usr/app/packages/validator/src/services/indices.ts:119:35)
Jun-15 18:49:36.035[]                error: Failed to download SyncDuties epoch=58789 - Failed to obtain SyncDuties - Internal Server Error: Internal server error
Error: Failed to obtain SyncDuties - Internal Server Error: Internal server error
    at Function.assert (file:///usr/app/packages/api/src/utils/client/httpClient.ts:44:13)
    at SyncCommitteeDutiesService.pollSyncCommitteesForEpoch (file:///usr/app/packages/validator/src/services/syncCommitteeDuties.ts:240:14)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at SyncCommitteeDutiesService.pollSyncCommittees (file:///usr/app/packages/validator/src/services/syncCommitteeDuties.ts:185:7)
    at async Promise.all (index 0)
    at SyncCommitteeDutiesService.runDutiesTasks (file:///usr/app/packages/validator/src/services/syncCommitteeDuties.ts:147:5)
    at Clock.runAtMostEvery (file:///usr/app/packages/validator/src/util/clock.ts:96:7)
Jun-15 18:49:36.043[]                error: Failed to download attester duties epoch=58789 - Failed to obtain attester duty - Internal Server Error: Internal server error
Error: Failed to obtain attester duty - Internal Server Error: Internal server error
    at Function.assert (file:///usr/app/packages/api/src/utils/client/httpClient.ts:44:13)
    at AttestationDutiesService.pollBeaconAttestersForEpoch (file:///usr/app/packages/validator/src/services/attestationDuties.ts:234:14)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at AttestationDutiesService.pollBeaconAttesters (file:///usr/app/packages/validator/src/s
codecov[bot] commented 3 months ago

Codecov Report

Attention: Patch coverage is 75.00000% with 2 lines in your changes missing coverage. Please review.

Project coverage is 62.75%. Comparing base (f29a6db) to head (069ceb6).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## unstable #6888 +/- ## ============================================ - Coverage 62.75% 62.75% -0.01% ============================================ Files 578 578 Lines 61347 61350 +3 Branches 2114 2114 ============================================ + Hits 38498 38499 +1 - Misses 22811 22813 +2 Partials 38 38 ```
github-actions[bot] commented 3 months ago

Performance Report

✔️ no performance regression detected

Full benchmark results | Benchmark suite | Current: d2688502ffdc81f91de9455b7a33274030aad4e2 | Previous: f29a6db05624fc454e586790e4f977ec0817ed25 | Ratio | |-|-|-|-| | getPubkeys - index2pubkey - req 1000 vs - 250000 vc | 822.44 us/op | 871.43 us/op | 0.94 | | getPubkeys - validatorsArr - req 1000 vs - 250000 vc | 38.229 us/op | 47.264 us/op | 0.81 | | BLS verify - blst-native | 1.0512 ms/op | 1.1055 ms/op | 0.95 | | BLS verifyMultipleSignatures 3 - blst-native | 2.2396 ms/op | 2.3523 ms/op | 0.95 | | BLS verifyMultipleSignatures 8 - blst-native | 4.9576 ms/op | 5.2188 ms/op | 0.95 | | BLS verifyMultipleSignatures 32 - blst-native | 18.230 ms/op | 19.084 ms/op | 0.96 | | BLS verifyMultipleSignatures 64 - blst-native | 35.929 ms/op | 37.290 ms/op | 0.96 | | BLS verifyMultipleSignatures 128 - blst-native | 73.832 ms/op | 74.619 ms/op | 0.99 | | BLS deserializing 10000 signatures | 749.95 ms/op | 776.85 ms/op | 0.97 | | BLS deserializing 100000 signatures | 7.7401 s/op | 8.1414 s/op | 0.95 | | BLS verifyMultipleSignatures - same message - 3 - blst-native | 1.1314 ms/op | 1.2076 ms/op | 0.94 | | BLS verifyMultipleSignatures - same message - 8 - blst-native | 1.2839 ms/op | 1.2794 ms/op | 1.00 | | BLS verifyMultipleSignatures - same message - 32 - blst-native | 2.0174 ms/op | 2.0024 ms/op | 1.01 | | BLS verifyMultipleSignatures - same message - 64 - blst-native | 3.6951 ms/op | 3.0417 ms/op | 1.21 | | BLS verifyMultipleSignatures - same message - 128 - blst-native | 4.9552 ms/op | 6.1370 ms/op | 0.81 | | BLS aggregatePubkeys 32 - blst-native | 22.672 us/op | 24.819 us/op | 0.91 | | BLS aggregatePubkeys 128 - blst-native | 88.166 us/op | 91.399 us/op | 0.96 | | notSeenSlots=1 numMissedVotes=1 numBadVotes=10 | 57.350 ms/op | 70.197 ms/op | 0.82 | | notSeenSlots=1 numMissedVotes=0 numBadVotes=4 | 56.739 ms/op | 58.153 ms/op | 0.98 | | notSeenSlots=2 numMissedVotes=1 numBadVotes=10 | 26.483 ms/op | 33.573 ms/op | 0.79 | | getSlashingsAndExits - default max | 75.399 us/op | 83.895 us/op | 0.90 | | getSlashingsAndExits - 2k | 215.68 us/op | 352.76 us/op | 0.61 | | proposeBlockBody type=full, size=empty | 5.1971 ms/op | 5.5130 ms/op | 0.94 | | isKnown best case - 1 super set check | 475.00 ns/op | 603.00 ns/op | 0.79 | | isKnown normal case - 2 super set checks | 463.00 ns/op | 581.00 ns/op | 0.80 | | isKnown worse case - 16 super set checks | 465.00 ns/op | 535.00 ns/op | 0.87 | | InMemoryCheckpointStateCache - add get delete | 3.8360 us/op | 4.7450 us/op | 0.81 | | validate api signedAggregateAndProof - struct | 2.2550 ms/op | 2.4714 ms/op | 0.91 | | validate gossip signedAggregateAndProof - struct | 2.2563 ms/op | 2.4786 ms/op | 0.91 | | validate gossip attestation - vc 640000 | 1.1207 ms/op | 1.1933 ms/op | 0.94 | | batch validate gossip attestation - vc 640000 - chunk 32 | 132.02 us/op | 149.01 us/op | 0.89 | | batch validate gossip attestation - vc 640000 - chunk 64 | 113.51 us/op | 127.85 us/op | 0.89 | | batch validate gossip attestation - vc 640000 - chunk 128 | 107.07 us/op | 116.21 us/op | 0.92 | | batch validate gossip attestation - vc 640000 - chunk 256 | 103.89 us/op | 112.53 us/op | 0.92 | | pickEth1Vote - no votes | 786.41 us/op | 852.62 us/op | 0.92 | | pickEth1Vote - max votes | 7.8147 ms/op | 8.4516 ms/op | 0.92 | | pickEth1Vote - Eth1Data hashTreeRoot value x2048 | 16.370 ms/op | 17.640 ms/op | 0.93 | | pickEth1Vote - Eth1Data hashTreeRoot tree x2048 | 19.468 ms/op | 21.443 ms/op | 0.91 | | pickEth1Vote - Eth1Data fastSerialize value x2048 | 377.96 us/op | 375.71 us/op | 1.01 | | pickEth1Vote - Eth1Data fastSerialize tree x2048 | 5.1698 ms/op | 5.8576 ms/op | 0.88 | | bytes32 toHexString | 595.00 ns/op | 591.00 ns/op | 1.01 | | bytes32 Buffer.toString(hex) | 427.00 ns/op | 435.00 ns/op | 0.98 | | bytes32 Buffer.toString(hex) from Uint8Array | 545.00 ns/op | 549.00 ns/op | 0.99 | | bytes32 Buffer.toString(hex) + 0x | 422.00 ns/op | 434.00 ns/op | 0.97 | | Object access 1 prop | 0.31200 ns/op | 0.32200 ns/op | 0.97 | | Map access 1 prop | 0.31500 ns/op | 0.32500 ns/op | 0.97 | | Object get x1000 | 4.8520 ns/op | 5.2230 ns/op | 0.93 | | Map get x1000 | 5.4440 ns/op | 5.9810 ns/op | 0.91 | | Object set x1000 | 22.203 ns/op | 26.297 ns/op | 0.84 | | Map set x1000 | 19.006 ns/op | 19.229 ns/op | 0.99 | | Return object 10000 times | 0.27820 ns/op | 0.30110 ns/op | 0.92 | | Throw Error 10000 times | 2.5017 us/op | 2.7681 us/op | 0.90 | | fastMsgIdFn sha256 / 200 bytes | 1.8580 us/op | 2.0190 us/op | 0.92 | | fastMsgIdFn h32 xxhash / 200 bytes | 393.00 ns/op | 434.00 ns/op | 0.91 | | fastMsgIdFn h64 xxhash / 200 bytes | 433.00 ns/op | 456.00 ns/op | 0.95 | | fastMsgIdFn sha256 / 1000 bytes | 5.6180 us/op | 5.9760 us/op | 0.94 | | fastMsgIdFn h32 xxhash / 1000 bytes | 506.00 ns/op | 538.00 ns/op | 0.94 | | fastMsgIdFn h64 xxhash / 1000 bytes | 495.00 ns/op | 533.00 ns/op | 0.93 | | fastMsgIdFn sha256 / 10000 bytes | 47.211 us/op | 50.942 us/op | 0.93 | | fastMsgIdFn h32 xxhash / 10000 bytes | 1.8070 us/op | 1.9230 us/op | 0.94 | | fastMsgIdFn h64 xxhash / 10000 bytes | 1.2690 us/op | 1.3380 us/op | 0.95 | | send data - 1000 256B messages | 9.9996 ms/op | 10.014 ms/op | 1.00 | | send data - 1000 512B messages | 13.511 ms/op | 12.936 ms/op | 1.04 | | send data - 1000 1024B messages | 22.387 ms/op | 21.573 ms/op | 1.04 | | send data - 1000 1200B messages | 23.794 ms/op | 22.236 ms/op | 1.07 | | send data - 1000 2048B messages | 29.184 ms/op | 32.824 ms/op | 0.89 | | send data - 1000 4096B messages | 26.836 ms/op | 25.680 ms/op | 1.04 | | send data - 1000 16384B messages | 66.610 ms/op | 70.502 ms/op | 0.94 | | send data - 1000 65536B messages | 236.75 ms/op | 315.87 ms/op | 0.75 | | enrSubnets - fastDeserialize 64 bits | 1.0940 us/op | 1.4980 us/op | 0.73 | | enrSubnets - ssz BitVector 64 bits | 503.00 ns/op | 654.00 ns/op | 0.77 | | enrSubnets - fastDeserialize 4 bits | 313.00 ns/op | 396.00 ns/op | 0.79 | | enrSubnets - ssz BitVector 4 bits | 495.00 ns/op | 652.00 ns/op | 0.76 | | prioritizePeers score -10:0 att 32-0.1 sync 2-0 | 123.49 us/op | 160.29 us/op | 0.77 | | prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 | 170.98 us/op | 153.63 us/op | 1.11 | | prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 | 235.50 us/op | 303.90 us/op | 0.77 | | prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 | 464.22 us/op | 479.31 us/op | 0.97 | | prioritizePeers score 0:0 att 64-1 sync 4-1 | 611.42 us/op | 595.13 us/op | 1.03 | | array of 16000 items push then shift | 1.1984 us/op | 1.3551 us/op | 0.88 | | LinkedList of 16000 items push then shift | 7.1090 ns/op | 6.6040 ns/op | 1.08 | | array of 16000 items push then pop | 80.419 ns/op | 91.693 ns/op | 0.88 | | LinkedList of 16000 items push then pop | 6.0400 ns/op | 6.4010 ns/op | 0.94 | | array of 24000 items push then shift | 1.7552 us/op | 1.9711 us/op | 0.89 | | LinkedList of 24000 items push then shift | 6.6060 ns/op | 6.5880 ns/op | 1.00 | | array of 24000 items push then pop | 101.91 ns/op | 117.44 ns/op | 0.87 | | LinkedList of 24000 items push then pop | 6.0380 ns/op | 6.4950 ns/op | 0.93 | | intersect bitArray bitLen 8 | 5.1600 ns/op | 5.5300 ns/op | 0.93 | | intersect array and set length 8 | 35.983 ns/op | 39.236 ns/op | 0.92 | | intersect bitArray bitLen 128 | 25.310 ns/op | 26.901 ns/op | 0.94 | | intersect array and set length 128 | 539.77 ns/op | 596.98 ns/op | 0.90 | | bitArray.getTrueBitIndexes() bitLen 128 | 1.9770 us/op | 1.5690 us/op | 1.26 | | bitArray.getTrueBitIndexes() bitLen 248 | 3.2310 us/op | 2.8790 us/op | 1.12 | | bitArray.getTrueBitIndexes() bitLen 512 | 6.7380 us/op | 5.4390 us/op | 1.24 | | Buffer.concat 32 items | 995.00 ns/op | 981.00 ns/op | 1.01 | | Uint8Array.set 32 items | 1.6920 us/op | 1.3310 us/op | 1.27 | | Buffer.copy | 1.8680 us/op | 1.4420 us/op | 1.30 | | Uint8Array.set - with subarray | 2.3470 us/op | 2.0410 us/op | 1.15 | | Uint8Array.set - without subarray | 1.6560 us/op | 1.4690 us/op | 1.13 | | Set add up to 64 items then delete first | 1.8261 us/op | 1.7939 us/op | 1.02 | | OrderedSet add up to 64 items then delete first | 2.8678 us/op | 2.8043 us/op | 1.02 | | Set add up to 64 items then delete last | 2.0855 us/op | 2.0549 us/op | 1.01 | | OrderedSet add up to 64 items then delete last | 3.1650 us/op | 3.0954 us/op | 1.02 | | Set add up to 64 items then delete middle | 2.0842 us/op | 2.0413 us/op | 1.02 | | OrderedSet add up to 64 items then delete middle | 4.5683 us/op | 4.5568 us/op | 1.00 | | Set add up to 128 items then delete first | 4.1215 us/op | 4.0683 us/op | 1.01 | | OrderedSet add up to 128 items then delete first | 6.4518 us/op | 6.4289 us/op | 1.00 | | Set add up to 128 items then delete last | 3.9909 us/op | 3.9191 us/op | 1.02 | | OrderedSet add up to 128 items then delete last | 6.0669 us/op | 5.9879 us/op | 1.01 | | Set add up to 128 items then delete middle | 3.9945 us/op | 3.9134 us/op | 1.02 | | OrderedSet add up to 128 items then delete middle | 11.920 us/op | 12.150 us/op | 0.98 | | Set add up to 256 items then delete first | 8.0858 us/op | 8.0196 us/op | 1.01 | | OrderedSet add up to 256 items then delete first | 12.858 us/op | 12.723 us/op | 1.01 | | Set add up to 256 items then delete last | 7.8574 us/op | 7.7654 us/op | 1.01 | | OrderedSet add up to 256 items then delete last | 12.026 us/op | 11.986 us/op | 1.00 | | Set add up to 256 items then delete middle | 7.8030 us/op | 7.8457 us/op | 0.99 | | OrderedSet add up to 256 items then delete middle | 34.809 us/op | 34.932 us/op | 1.00 | | transfer serialized Status (84 B) | 1.4000 us/op | 1.5360 us/op | 0.91 | | copy serialized Status (84 B) | 1.2630 us/op | 1.3450 us/op | 0.94 | | transfer serialized SignedVoluntaryExit (112 B) | 1.6300 us/op | 1.4270 us/op | 1.14 | | copy serialized SignedVoluntaryExit (112 B) | 1.3330 us/op | 1.3860 us/op | 0.96 | | transfer serialized ProposerSlashing (416 B) | 2.1840 us/op | 1.6950 us/op | 1.29 | | copy serialized ProposerSlashing (416 B) | 1.8430 us/op | 1.6420 us/op | 1.12 | | transfer serialized Attestation (485 B) | 2.2390 us/op | 2.0340 us/op | 1.10 | | copy serialized Attestation (485 B) | 1.9180 us/op | 2.0520 us/op | 0.93 | | transfer serialized AttesterSlashing (33232 B) | 2.3150 us/op | 1.9490 us/op | 1.19 | | copy serialized AttesterSlashing (33232 B) | 5.7280 us/op | 6.5000 us/op | 0.88 | | transfer serialized Small SignedBeaconBlock (128000 B) | 3.2890 us/op | 2.0750 us/op | 1.59 | | copy serialized Small SignedBeaconBlock (128000 B) | 13.152 us/op | 13.533 us/op | 0.97 | | transfer serialized Avg SignedBeaconBlock (200000 B) | 2.7270 us/op | 2.7990 us/op | 0.97 | | copy serialized Avg SignedBeaconBlock (200000 B) | 19.723 us/op | 14.113 us/op | 1.40 | | transfer serialized BlobsSidecar (524380 B) | 3.1830 us/op | 3.0930 us/op | 1.03 | | copy serialized BlobsSidecar (524380 B) | 113.71 us/op | 72.124 us/op | 1.58 | | transfer serialized Big SignedBeaconBlock (1000000 B) | 3.3770 us/op | 3.5200 us/op | 0.96 | | copy serialized Big SignedBeaconBlock (1000000 B) | 208.34 us/op | 236.09 us/op | 0.88 | | pass gossip attestations to forkchoice per slot | 2.4721 ms/op | 2.6152 ms/op | 0.95 | | forkChoice updateHead vc 100000 bc 64 eq 0 | 379.98 us/op | 479.10 us/op | 0.79 | | forkChoice updateHead vc 600000 bc 64 eq 0 | 2.3953 ms/op | 2.4708 ms/op | 0.97 | | forkChoice updateHead vc 1000000 bc 64 eq 0 | 4.4471 ms/op | 4.0792 ms/op | 1.09 | | forkChoice updateHead vc 600000 bc 320 eq 0 | 2.5131 ms/op | 2.6149 ms/op | 0.96 | | forkChoice updateHead vc 600000 bc 1200 eq 0 | 2.6309 ms/op | 2.5952 ms/op | 1.01 | | forkChoice updateHead vc 600000 bc 7200 eq 0 | 2.7785 ms/op | 2.8169 ms/op | 0.99 | | forkChoice updateHead vc 600000 bc 64 eq 1000 | 9.7275 ms/op | 9.6750 ms/op | 1.01 | | forkChoice updateHead vc 600000 bc 64 eq 10000 | 9.5390 ms/op | 9.6372 ms/op | 0.99 | | forkChoice updateHead vc 600000 bc 64 eq 300000 | 11.823 ms/op | 11.780 ms/op | 1.00 | | computeDeltas 500000 validators 300 proto nodes | 2.8481 ms/op | 3.0749 ms/op | 0.93 | | computeDeltas 500000 validators 1200 proto nodes | 2.8726 ms/op | 3.0013 ms/op | 0.96 | | computeDeltas 500000 validators 7200 proto nodes | 2.9197 ms/op | 2.9767 ms/op | 0.98 | | computeDeltas 750000 validators 300 proto nodes | 4.3133 ms/op | 4.5770 ms/op | 0.94 | | computeDeltas 750000 validators 1200 proto nodes | 4.3432 ms/op | 4.5186 ms/op | 0.96 | | computeDeltas 750000 validators 7200 proto nodes | 4.4405 ms/op | 4.5847 ms/op | 0.97 | | computeDeltas 1400000 validators 300 proto nodes | 8.4147 ms/op | 8.8077 ms/op | 0.96 | | computeDeltas 1400000 validators 1200 proto nodes | 8.0079 ms/op | 8.4983 ms/op | 0.94 | | computeDeltas 1400000 validators 7200 proto nodes | 8.0598 ms/op | 8.3770 ms/op | 0.96 | | computeDeltas 2100000 validators 300 proto nodes | 12.329 ms/op | 12.969 ms/op | 0.95 | | computeDeltas 2100000 validators 1200 proto nodes | 12.466 ms/op | 12.962 ms/op | 0.96 | | computeDeltas 2100000 validators 7200 proto nodes | 12.337 ms/op | 12.929 ms/op | 0.95 | | altair processAttestation - 250000 vs - 7PWei normalcase | 1.3645 ms/op | 1.6835 ms/op | 0.81 | | altair processAttestation - 250000 vs - 7PWei worstcase | 2.0536 ms/op | 2.0722 ms/op | 0.99 | | altair processAttestation - setStatus - 1/6 committees join | 62.600 us/op | 76.203 us/op | 0.82 | | altair processAttestation - setStatus - 1/3 committees join | 139.74 us/op | 143.35 us/op | 0.97 | | altair processAttestation - setStatus - 1/2 committees join | 180.93 us/op | 198.80 us/op | 0.91 | | altair processAttestation - setStatus - 2/3 committees join | 234.56 us/op | 266.87 us/op | 0.88 | | altair processAttestation - setStatus - 4/5 committees join | 373.32 us/op | 395.64 us/op | 0.94 | | altair processAttestation - setStatus - 100% committees join | 448.43 us/op | 477.79 us/op | 0.94 | | altair processBlock - 250000 vs - 7PWei normalcase | 3.1631 ms/op | 3.1163 ms/op | 1.02 | | altair processBlock - 250000 vs - 7PWei normalcase hashState | 29.769 ms/op | 24.676 ms/op | 1.21 | | altair processBlock - 250000 vs - 7PWei worstcase | 40.111 ms/op | 45.197 ms/op | 0.89 | | altair processBlock - 250000 vs - 7PWei worstcase hashState | 78.469 ms/op | 74.252 ms/op | 1.06 | | phase0 processBlock - 250000 vs - 7PWei normalcase | 1.5385 ms/op | 2.0201 ms/op | 0.76 | | phase0 processBlock - 250000 vs - 7PWei worstcase | 22.347 ms/op | 25.873 ms/op | 0.86 | | altair processEth1Data - 250000 vs - 7PWei normalcase | 244.66 us/op | 267.45 us/op | 0.91 | | getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15 | 4.2630 us/op | 4.9010 us/op | 0.87 | | getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219 | 18.041 us/op | 19.933 us/op | 0.91 | | getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:42 | 4.0520 us/op | 6.9970 us/op | 0.58 | | getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18 | 5.3750 us/op | 4.7430 us/op | 1.13 | | getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1020 | 81.415 us/op | 75.491 us/op | 1.08 | | getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11777 | 752.24 us/op | 855.26 us/op | 0.88 | | getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384 | 640.95 us/op | 757.11 us/op | 0.85 | | getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384 | 1.0263 ms/op | 1.0300 ms/op | 1.00 | | getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384 | 1.9356 ms/op | 1.9440 ms/op | 1.00 | | getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384 | 1.1148 ms/op | 1.1218 ms/op | 0.99 | | getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384 | 2.9470 ms/op | 2.8338 ms/op | 1.04 | | Tree 40 250000 create | 189.29 ms/op | 177.68 ms/op | 1.07 | | Tree 40 250000 get(125000) | 106.42 ns/op | 106.69 ns/op | 1.00 | | Tree 40 250000 set(125000) | 519.33 ns/op | 544.96 ns/op | 0.95 | | Tree 40 250000 toArray() | 12.729 ms/op | 15.037 ms/op | 0.85 | | Tree 40 250000 iterate all - toArray() + loop | 12.584 ms/op | 15.403 ms/op | 0.82 | | Tree 40 250000 iterate all - get(i) | 40.386 ms/op | 46.461 ms/op | 0.87 | | MutableVector 250000 create | 9.1321 ms/op | 9.1358 ms/op | 1.00 | | MutableVector 250000 get(125000) | 5.7460 ns/op | 5.7910 ns/op | 0.99 | | MutableVector 250000 set(125000) | 166.13 ns/op | 177.29 ns/op | 0.94 | | MutableVector 250000 toArray() | 3.0055 ms/op | 3.7150 ms/op | 0.81 | | MutableVector 250000 iterate all - toArray() + loop | 3.1050 ms/op | 4.1611 ms/op | 0.75 | | MutableVector 250000 iterate all - get(i) | 1.4335 ms/op | 1.3875 ms/op | 1.03 | | Array 250000 create | 2.5032 ms/op | 3.2145 ms/op | 0.78 | | Array 250000 clone - spread | 1.1241 ms/op | 1.3428 ms/op | 0.84 | | Array 250000 get(125000) | 0.58200 ns/op | 0.57100 ns/op | 1.02 | | Array 250000 set(125000) | 0.59100 ns/op | 0.57900 ns/op | 1.02 | | Array 250000 iterate all - loop | 73.659 us/op | 74.202 us/op | 0.99 | | effectiveBalanceIncrements clone Uint8Array 300000 | 18.880 us/op | 24.427 us/op | 0.77 | | effectiveBalanceIncrements clone MutableVector 300000 | 316.00 ns/op | 305.00 ns/op | 1.04 | | effectiveBalanceIncrements rw all Uint8Array 300000 | 159.68 us/op | 160.91 us/op | 0.99 | | effectiveBalanceIncrements rw all MutableVector 300000 | 48.751 ms/op | 64.072 ms/op | 0.76 | | phase0 afterProcessEpoch - 250000 vs - 7PWei | 76.370 ms/op | 76.400 ms/op | 1.00 | | phase0 beforeProcessEpoch - 250000 vs - 7PWei | 37.118 ms/op | 46.521 ms/op | 0.80 | | altair processEpoch - mainnet_e81889 | 335.75 ms/op | 332.34 ms/op | 1.01 | | mainnet_e81889 - altair beforeProcessEpoch | 35.931 ms/op | 59.731 ms/op | 0.60 | | mainnet_e81889 - altair processJustificationAndFinalization | 9.5550 us/op | 10.873 us/op | 0.88 | | mainnet_e81889 - altair processInactivityUpdates | 4.0485 ms/op | 4.4566 ms/op | 0.91 | | mainnet_e81889 - altair processRewardsAndPenalties | 57.609 ms/op | 50.583 ms/op | 1.14 | | mainnet_e81889 - altair processRegistryUpdates | 2.0760 us/op | 2.4160 us/op | 0.86 | | mainnet_e81889 - altair processSlashings | 851.00 ns/op | 754.00 ns/op | 1.13 | | mainnet_e81889 - altair processEth1DataReset | 659.00 ns/op | 882.00 ns/op | 0.75 | | mainnet_e81889 - altair processEffectiveBalanceUpdates | 774.72 us/op | 1.3033 ms/op | 0.59 | | mainnet_e81889 - altair processSlashingsReset | 1.3130 us/op | 2.6550 us/op | 0.49 | | mainnet_e81889 - altair processRandaoMixesReset | 1.4660 us/op | 2.6740 us/op | 0.55 | | mainnet_e81889 - altair processHistoricalRootsUpdate | 399.00 ns/op | 712.00 ns/op | 0.56 | | mainnet_e81889 - altair processParticipationFlagUpdates | 1.2470 us/op | 1.7670 us/op | 0.71 | | mainnet_e81889 - altair processSyncCommitteeUpdates | 619.00 ns/op | 693.00 ns/op | 0.89 | | mainnet_e81889 - altair afterProcessEpoch | 75.162 ms/op | 76.912 ms/op | 0.98 | | capella processEpoch - mainnet_e217614 | 1.1577 s/op | 1.2282 s/op | 0.94 | | mainnet_e217614 - capella beforeProcessEpoch | 240.80 ms/op | 262.60 ms/op | 0.92 | | mainnet_e217614 - capella processJustificationAndFinalization | 11.305 us/op | 14.194 us/op | 0.80 | | mainnet_e217614 - capella processInactivityUpdates | 16.510 ms/op | 16.277 ms/op | 1.01 | | mainnet_e217614 - capella processRewardsAndPenalties | 258.51 ms/op | 254.27 ms/op | 1.02 | | mainnet_e217614 - capella processRegistryUpdates | 14.450 us/op | 13.072 us/op | 1.11 | | mainnet_e217614 - capella processSlashings | 789.00 ns/op | 897.00 ns/op | 0.88 | | mainnet_e217614 - capella processEth1DataReset | 857.00 ns/op | 914.00 ns/op | 0.94 | | mainnet_e217614 - capella processEffectiveBalanceUpdates | 4.8432 ms/op | 17.406 ms/op | 0.28 | | mainnet_e217614 - capella processSlashingsReset | 2.3590 us/op | 3.4790 us/op | 0.68 | | mainnet_e217614 - capella processRandaoMixesReset | 3.0840 us/op | 6.5210 us/op | 0.47 | | mainnet_e217614 - capella processHistoricalRootsUpdate | 791.00 ns/op | 1.3250 us/op | 0.60 | | mainnet_e217614 - capella processParticipationFlagUpdates | 1.4400 us/op | 4.2880 us/op | 0.34 | | mainnet_e217614 - capella afterProcessEpoch | 244.09 ms/op | 284.76 ms/op | 0.86 | | phase0 processEpoch - mainnet_e58758 | 352.89 ms/op | 418.12 ms/op | 0.84 | | mainnet_e58758 - phase0 beforeProcessEpoch | 101.20 ms/op | 134.14 ms/op | 0.75 | | mainnet_e58758 - phase0 processJustificationAndFinalization | 11.650 us/op | 18.194 us/op | 0.64 | | mainnet_e58758 - phase0 processRewardsAndPenalties | 30.924 ms/op | 29.729 ms/op | 1.04 | | mainnet_e58758 - phase0 processRegistryUpdates | 7.7460 us/op | 9.5440 us/op | 0.81 | | mainnet_e58758 - phase0 processSlashings | 782.00 ns/op | 895.00 ns/op | 0.87 | | mainnet_e58758 - phase0 processEth1DataReset | 710.00 ns/op | 776.00 ns/op | 0.91 | | mainnet_e58758 - phase0 processEffectiveBalanceUpdates | 1.1352 ms/op | 782.00 us/op | 1.45 | | mainnet_e58758 - phase0 processSlashingsReset | 2.7950 us/op | 3.5820 us/op | 0.78 | | mainnet_e58758 - phase0 processRandaoMixesReset | 3.2030 us/op | 5.4050 us/op | 0.59 | | mainnet_e58758 - phase0 processHistoricalRootsUpdate | 785.00 ns/op | 776.00 ns/op | 1.01 | | mainnet_e58758 - phase0 processParticipationRecordUpdates | 2.7270 us/op | 3.4780 us/op | 0.78 | | mainnet_e58758 - phase0 afterProcessEpoch | 67.244 ms/op | 69.548 ms/op | 0.97 | | phase0 processEffectiveBalanceUpdates - 250000 normalcase | 1.4090 ms/op | 916.68 us/op | 1.54 | | phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 | 1.1173 ms/op | 1.1813 ms/op | 0.95 | | altair processInactivityUpdates - 250000 normalcase | 15.945 ms/op | 17.527 ms/op | 0.91 | | altair processInactivityUpdates - 250000 worstcase | 18.932 ms/op | 20.532 ms/op | 0.92 | | phase0 processRegistryUpdates - 250000 normalcase | 6.1350 us/op | 7.3390 us/op | 0.84 | | phase0 processRegistryUpdates - 250000 badcase_full_deposits | 315.46 us/op | 306.70 us/op | 1.03 | | phase0 processRegistryUpdates - 250000 worstcase 0.5 | 112.63 ms/op | 96.857 ms/op | 1.16 | | altair processRewardsAndPenalties - 250000 normalcase | 33.087 ms/op | 49.100 ms/op | 0.67 | | altair processRewardsAndPenalties - 250000 worstcase | 38.751 ms/op | 45.819 ms/op | 0.85 | | phase0 getAttestationDeltas - 250000 normalcase | 5.6427 ms/op | 6.0606 ms/op | 0.93 | | phase0 getAttestationDeltas - 250000 worstcase | 6.0919 ms/op | 6.3772 ms/op | 0.96 | | phase0 processSlashings - 250000 worstcase | 89.748 us/op | 65.561 us/op | 1.37 | | altair processSyncCommitteeUpdates - 250000 | 108.00 ms/op | 104.89 ms/op | 1.03 | | BeaconState.hashTreeRoot - No change | 483.00 ns/op | 540.00 ns/op | 0.89 | | BeaconState.hashTreeRoot - 1 full validator | 119.08 us/op | 92.397 us/op | 1.29 | | BeaconState.hashTreeRoot - 32 full validator | 1.1323 ms/op | 754.10 us/op | 1.50 | | BeaconState.hashTreeRoot - 512 full validator | 12.497 ms/op | 12.549 ms/op | 1.00 | | BeaconState.hashTreeRoot - 1 validator.effectiveBalance | 142.51 us/op | 111.58 us/op | 1.28 | | BeaconState.hashTreeRoot - 32 validator.effectiveBalance | 2.1684 ms/op | 1.6160 ms/op | 1.34 | | BeaconState.hashTreeRoot - 512 validator.effectiveBalance | 26.899 ms/op | 24.659 ms/op | 1.09 | | BeaconState.hashTreeRoot - 1 balances | 106.49 us/op | 97.514 us/op | 1.09 | | BeaconState.hashTreeRoot - 32 balances | 961.51 us/op | 988.77 us/op | 0.97 | | BeaconState.hashTreeRoot - 512 balances | 11.331 ms/op | 10.488 ms/op | 1.08 | | BeaconState.hashTreeRoot - 250000 balances | 165.44 ms/op | 168.57 ms/op | 0.98 | | aggregationBits - 2048 els - zipIndexesInBitList | 19.460 us/op | 21.480 us/op | 0.91 | | byteArrayEquals 32 | 48.411 ns/op | 46.327 ns/op | 1.04 | | Buffer.compare 32 | 38.891 ns/op | 35.046 ns/op | 1.11 | | byteArrayEquals 1024 | 1.2721 us/op | 1.2303 us/op | 1.03 | | Buffer.compare 1024 | 47.081 ns/op | 42.588 ns/op | 1.11 | | byteArrayEquals 16384 | 20.168 us/op | 19.244 us/op | 1.05 | | Buffer.compare 16384 | 188.28 ns/op | 219.90 ns/op | 0.86 | | byteArrayEquals 123687377 | 146.47 ms/op | 144.89 ms/op | 1.01 | | Buffer.compare 123687377 | 5.4473 ms/op | 5.2861 ms/op | 1.03 | | byteArrayEquals 32 - diff last byte | 46.388 ns/op | 45.107 ns/op | 1.03 | | Buffer.compare 32 - diff last byte | 37.475 ns/op | 34.979 ns/op | 1.07 | | byteArrayEquals 1024 - diff last byte | 1.2276 us/op | 1.1993 us/op | 1.02 | | Buffer.compare 1024 - diff last byte | 42.224 ns/op | 42.201 ns/op | 1.00 | | byteArrayEquals 16384 - diff last byte | 19.459 us/op | 19.057 us/op | 1.02 | | Buffer.compare 16384 - diff last byte | 217.23 ns/op | 180.80 ns/op | 1.20 | | byteArrayEquals 123687377 - diff last byte | 148.09 ms/op | 144.64 ms/op | 1.02 | | Buffer.compare 123687377 - diff last byte | 5.3177 ms/op | 5.3298 ms/op | 1.00 | | byteArrayEquals 32 - random bytes | 4.6980 ns/op | 4.8430 ns/op | 0.97 | | Buffer.compare 32 - random bytes | 38.556 ns/op | 37.798 ns/op | 1.02 | | byteArrayEquals 1024 - random bytes | 4.7050 ns/op | 4.7600 ns/op | 0.99 | | Buffer.compare 1024 - random bytes | 37.032 ns/op | 37.019 ns/op | 1.00 | | byteArrayEquals 16384 - random bytes | 4.6960 ns/op | 4.7390 ns/op | 0.99 | | Buffer.compare 16384 - random bytes | 36.591 ns/op | 37.411 ns/op | 0.98 | | byteArrayEquals 123687377 - random bytes | 7.5800 ns/op | 7.6800 ns/op | 0.99 | | Buffer.compare 123687377 - random bytes | 45.740 ns/op | 40.070 ns/op | 1.14 | | regular array get 100000 times | 29.467 us/op | 29.751 us/op | 0.99 | | wrappedArray get 100000 times | 29.429 us/op | 29.679 us/op | 0.99 | | arrayWithProxy get 100000 times | 9.1833 ms/op | 9.4650 ms/op | 0.97 | | ssz.Root.equals | 37.815 ns/op | 39.294 ns/op | 0.96 | | byteArrayEquals | 41.179 ns/op | 41.767 ns/op | 0.99 | | Buffer.compare | 8.5800 ns/op | 8.6600 ns/op | 0.99 | | shuffle list - 16384 els | 5.5124 ms/op | 5.5561 ms/op | 0.99 | | shuffle list - 250000 els | 82.030 ms/op | 81.152 ms/op | 1.01 | | processSlot - 1 slots | 12.558 us/op | 13.151 us/op | 0.95 | | processSlot - 32 slots | 2.7663 ms/op | 2.9088 ms/op | 0.95 | | getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei | 42.920 ms/op | 40.973 ms/op | 1.05 | | getCommitteeAssignments - req 1 vs - 250000 vc | 1.6984 ms/op | 1.7998 ms/op | 0.94 | | getCommitteeAssignments - req 100 vs - 250000 vc | 3.5351 ms/op | 3.5334 ms/op | 1.00 | | getCommitteeAssignments - req 1000 vs - 250000 vc | 3.7956 ms/op | 3.7766 ms/op | 1.01 | | findModifiedValidators - 10000 modified validators | 243.41 ms/op | 240.10 ms/op | 1.01 | | findModifiedValidators - 1000 modified validators | 142.53 ms/op | 146.70 ms/op | 0.97 | | findModifiedValidators - 100 modified validators | 134.41 ms/op | 148.46 ms/op | 0.91 | | findModifiedValidators - 10 modified validators | 144.43 ms/op | 131.49 ms/op | 1.10 | | findModifiedValidators - 1 modified validators | 122.78 ms/op | 140.64 ms/op | 0.87 | | findModifiedValidators - no difference | 139.93 ms/op | 130.62 ms/op | 1.07 | | compare ViewDUs | 3.3386 s/op | 2.9264 s/op | 1.14 | | compare each validator Uint8Array | 1.6016 s/op | 1.1548 s/op | 1.39 | | compare ViewDU to Uint8Array | 769.61 ms/op | 805.87 ms/op | 0.96 | | migrate state 1000000 validators, 24 modified, 0 new | 581.23 ms/op | 592.17 ms/op | 0.98 | | migrate state 1000000 validators, 1700 modified, 1000 new | 833.92 ms/op | 812.27 ms/op | 1.03 | | migrate state 1000000 validators, 3400 modified, 2000 new | 1.0264 s/op | 931.17 ms/op | 1.10 | | migrate state 1500000 validators, 24 modified, 0 new | 579.26 ms/op | 628.32 ms/op | 0.92 | | migrate state 1500000 validators, 1700 modified, 1000 new | 824.03 ms/op | 745.02 ms/op | 1.11 | | migrate state 1500000 validators, 3400 modified, 2000 new | 1.0506 s/op | 959.19 ms/op | 1.10 | | RootCache.getBlockRootAtSlot - 250000 vs - 7PWei | 5.9000 ns/op | 5.9900 ns/op | 0.98 | | state getBlockRootAtSlot - 250000 vs - 7PWei | 842.33 ns/op | 430.12 ns/op | 1.96 | | computeProposers - vc 250000 | 6.7401 ms/op | 5.4297 ms/op | 1.24 | | computeEpochShuffling - vc 250000 | 81.870 ms/op | 81.522 ms/op | 1.00 | | getNextSyncCommittee - vc 250000 | 114.56 ms/op | 94.324 ms/op | 1.21 | | computeSigningRoot for AttestationData | 28.421 us/op | 15.361 us/op | 1.85 | | hash AttestationData serialized data then Buffer.toString(base64) | 1.1722 us/op | 1.1879 us/op | 0.99 | | toHexString serialized data | 768.98 ns/op | 746.48 ns/op | 1.03 | | Buffer.toString(base64) | 144.63 ns/op | 127.55 ns/op | 1.13 |

by benchmarkbot/action

twoeths commented 3 months ago 
Failed to obtain attester duty

agree with @nflaig , we need to figure out the root cause of the error by either getting more log at the beacon-node side or reproducing the issue

g11tech commented 3 months ago

right, could be other errors coupled with this one causing sync and attester duties failure, will try to figure it out, most likely the BN api latency would be an issue because of epoch transition (just guessing), but in any case the duties apis should be retried atleast on next slot and not on next epoch.

will send followup PRs

g11tech commented 3 months ago

actually this log using this fixed branch indicates that the attestation and sync participation which would "freeze" earlier till next epoch continues ( see timestamps), so the uncaught exception might have had some impact on the event loop although I am not fully sure of it

Jun-16 10:11:05.425[]                 info: Published attestations slot=1885855, index=50, count=1
Jun-16 10:11:14.402[]                 warn: Failed to get indices for pending pubkeys pending=1000 - getStateValidators failed with status 500: Internal server error
Error: getStateValidators failed with status 500: Internal server error
    at ApiResponse.error (file:///usr/app/packages/api/src/utils/client/response.ts:164:12)
    at ApiResponse.assertOk (file:///usr/app/packages/api/src/utils/client/response.ts:155:18)
    at ApiResponse.value (file:///usr/app/packages/api/src/utils/client/response.ts:104:10)
    at IndicesService.fetchValidatorIndices (file:///usr/app/packages/validator/src/services/indices.ts:132:112)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at IndicesService.pollValidatorIndicesInternal (file:///usr/app/packages/validator/src/services/indices.ts:122:35)
Jun-16 10:11:14.415[]                error: Error on attestation aggregation selection slot=1885856 - submitBeaconCommitteeSelections failed with status 500: Internal server error
Error: submitBeaconCommitteeSelections failed with status 500: Internal server error
    at ApiResponse.error (file:///usr/app/packages/api/src/utils/client/response.ts:164:12)
    at ApiResponse.assertOk (file:///usr/app/packages/api/src/utils/client/response.ts:155:18)
    at ApiResponse.value (file:///usr/app/packages/api/src/utils/client/response.ts:104:10)
    at AttestationService.runDistributedAggregationSelectionTasks (file:///usr/app/packages/validator/src/services/attestation.ts:334:36)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
Jun-16 10:11:14.418[]                error: Failed to download attester duties epoch=58933 - getAttesterDuties failed with status 500: Internal server error
Error: getAttesterDuties failed with status 500: Internal server error
    at ApiResponse.error (file:///usr/app/packages/api/src/utils/client/response.ts:164:12)
    at ApiResponse.assertOk (file:///usr/app/packages/api/src/utils/client/response.ts:155:18)
    at ApiResponse.value (file:///usr/app/packages/api/src/utils/client/response.ts:104:10)
    at AttestationDutiesService.pollBeaconAttestersForEpoch (file:///usr/app/packages/validator/src/services/attestationDuties.ts:236:32)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at AttestationDutiesService.pollBeaconAttesters (file:///usr/app/packages/validator/src/services/attestationDuties.ts:183:7)
    at async Promise.all (index 0)
    at AttestationDutiesService.runDutiesTasks (file:///usr/app/packages/validator/src/services/attestationDuties.ts:144:5)
    at Clock.runAtMostEvery (file:///usr/app/packages/validator/src/util/clock.ts:96:7)
Jun-16 10:11:14.424[]                error: Failed to download attester duties epoch=58934 - getAttesterDuties failed with status 500: Internal server error
Error: getAttesterDuties failed with status 500: Internal server error
    at ApiResponse.error (file:///usr/app/packages/api/src/utils/client/response.ts:164:12)
    at ApiResponse.assertOk (file:///usr/app/packages/api/src/utils/client/response.ts:155:18)
    at ApiResponse.value (file:///usr/app/packages/api/src/utils/client/response.ts:104:10)
    at AttestationDutiesService.pollBeaconAttestersForEpoch (file:///usr/app/packages/validator/src/services/attestationDuties.ts:236:32)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at AttestationDutiesService.pollBeaconAttesters (file:///usr/app/packages/validator/src/services/attestationDuties.ts:183:7)
    at async Promise.all (index 0)
    at AttestationDutiesService.runDutiesTasks (file:///usr/app/packages/validator/src/services/attestationDuties.ts:144:5)
    at Clock.runAtMostEvery (file:///usr/app/packages/validator/src/util/clock.ts:96:7)
Jun-16 10:11:14.428[]                error: Failed to publish validator registrations to builder epoch=58933 - registerValidator failed with status 500: Internal server error
Error: registerValidator failed with status 500: Internal server error
    at ApiResponse.error (file:///usr/app/packages/api/src/utils/client/response.ts:164:12)
    at ApiResponse.assertOk (file:///usr/app/packages/api/src/utils/client/response.ts:155:18)
    at registerValidator (file:///usr/app/packages/validator/src/services/prepareBeaconProposer.ts:105:68)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Clock.runAtMostEvery (file:///usr/app/packages/validator/src/util/clock.ts:96:7)
Jun-16 10:11:14.431[]                error: Failed to download SyncDuties epoch=59189 - getSyncCommitteeDuties failed with status 500: Internal server error
Error: getSyncCommitteeDuties failed with status 500: Internal server error
    at ApiResponse.error (file:///usr/app/packages/api/src/utils/client/response.ts:164:12)
    at ApiResponse.assertOk (file:///usr/app/packages/api/src/utils/client/response.ts:155:18)
    at ApiResponse.value (file:///usr/app/packages/api/src/utils/client/response.ts:104:10)
    at SyncCommitteeDutiesService.pollSyncCommitteesForEpoch (file:///usr/app/packages/validator/src/services/syncCommitteeDuties.ts:238:98)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at SyncCommitteeDutiesService.pollSyncCommittees (file:///usr/app/packages/validator/src/services/syncCommitteeDuties.ts:185:7)
    at async Promise.all (index 0)
    at SyncCommitteeDutiesService.runDutiesTasks (file:///usr/app/packages/validator/src/services/syncCommitteeDuties.ts:147:5)
    at Clock.runAtMostEvery (file:///usr/app/packages/validator/src/util/clock.ts:96:7)
Jun-16 10:11:17.420[]                 info: Published attestations slot=1885856, index=38, count=1
Jun-16 10:11:29.511[]                 info: Published attestations slot=1885857, index=25, count=1
nflaig commented 3 months ago

I would assume getting attester duties is just a pass through request to the beacon node as it does not require any threshold aggregation or cluster participants to be online.

Error: getAttesterDuties failed with status 500: Internal server error

Investigating this by looking at the vc logs does not make much sense imo, as charon does not return the reason why it failed with 500 status code.

nflaig commented 3 months ago

Closed in favor of https://github.com/ChainSafe/lodestar/pull/6891

nflaig commented 3 months ago

actually this log using this fixed branch indicates that the attestation and sync participation which would "freeze" earlier till next epoch continues ( see timestamps), so the uncaught exception might have had some impact on the event loop although I am not fully sure of it

@g11tech After debugging the issue, it's just a wrongly handled promise which has no side effect on anything else, opened a PR to properly handle it and avoid the uncaught exception.