Invalid validators index value while connected to nimbus beacon node

[TobiWo]

NOTE

I don't know whether the following error is an issue with lodestar or nimbus.

Description

The following was tested on Goerli (before the merge): While connecting the lodestar validator to a nimbus beacon node I receive the following error:

Aug-09 21:34:25.031[] error: Failed to download attester duties epoch=113472 Bad Request: Invalid validator's index value(s) - Failed to obtain attester duty
Error: Bad Request: Invalid validator's index value(s) - Failed to obtain attester duty
    at HttpClient.requestWithBody (file:///usr/app/node_modules/@lodestar/api/src/utils/client/httpClient.ts:121:15)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at HttpClient.json (file:///usr/app/node_modules/@lodestar/api/src/utils/client/httpClient.ts:75:12)
    at Object.request [as getAttesterDuties] (file:///usr/app/node_modules/@lodestar/api/src/utils/client/client.ts:75:21)
    at AttestationDutiesService.pollBeaconAttestersForEpoch (file:///usr/app/node_modules/@lodestar/validator/src/services/attestationDuties.ts:203:28)
    at AttestationDutiesService.pollBeaconAttesters (file:///usr/app/node_modules/@lodestar/validator/src/services/attestationDuties.ts:155:7)
    at async Promise.all (index 0)
    at AttestationDutiesService.runDutiesTasks (file:///usr/app/node_modules/@lodestar/validator/src/services/attestationDuties.ts:116:5)
    at Clock.runAtMostEvery (file:///usr/app/node_modules/@lodestar/validator/src/util/clock.ts:96:7)
Aug-09 21:34:25.033[]                error: Failed to download SyncDuties epoch=113727 Bad Request: Invalid validator's index value(s) - Failed to obtain SyncDuties
Error: Bad Request: Invalid validator's index value(s) - Failed to obtain SyncDuties
    at HttpClient.requestWithBody (file:///usr/app/node_modules/@lodestar/api/src/utils/client/httpClient.ts:121:15)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at HttpClient.json (file:///usr/app/node_modules/@lodestar/api/src/utils/client/httpClient.ts:75:12)
    at Object.request [as getSyncCommitteeDuties] (file:///usr/app/node_modules/@lodestar/api/src/utils/client/client.ts:75:21)
    at SyncCommitteeDutiesService.pollSyncCommitteesForEpoch (file:///usr/app/node_modules/@lodestar/validator/src/services/syncCommitteeDuties.ts:234:24)
    at SyncCommitteeDutiesService.pollSyncCommittees (file:///usr/app/node_modules/@lodestar/validator/src/services/syncCommitteeDuties.ts:179:7)
    at async Promise.all (index 0)
    at SyncCommitteeDutiesService.runDutiesTasks (file:///usr/app/node_modules/@lodestar/validator/src/services/syncCommitteeDuties.ts:141:5)
    at Clock.runAtMostEvery (file:///usr/app/node_modules/@lodestar/validator/src/util/clock.ts:96:7)

The validator fails completely and does not produce any attestations or blocks.

lodestar version: 0.41.0 nimbus version: 22.7.0

[dapplion]

If you can, could you put an API sniffer between the beacon and validator? We use this one a lot while testing the merge. It logs all requests and would be great to debug issues like this as we see the exact payload being sent by each party

docker pull parithoshj/json_rpc_snoop:v1.0.0-x86

You can run it with command

./json_rpc_snoop -p {{beacon_port}} http://127.0.0.1:{{snooper_port}}

And

• tell the validator to connect to http://127.0.0.1:{{snooper_port}}
• beacon should serve REST API on beacon_port

[TobiWo]

@dapplion Sorry for the late reply. I tried the tool but currently have issues to connect to nimbus at all with that tool. I asked parithosh already. Let's see if he can help. EDIT: I managed to get the tool running but don't see any rpc requests, only the errors of the vc from my original post.

[dapplion]

@dapplion Sorry for the late reply. I tried the tool but currently have issues to connect to nimbus at all with that tool. I asked parithosh already. Let's see if he can help. EDIT: I managed to get the tool running but don't see any rpc requests, only the errors of the vc from my original post.

Could you dump here your setup? Is the validator client connecting to the beacon node directly or to the snooper?

[TobiWo]

@dapplion Sorry for the delay. I finally got rpc-snoop to work 😃 (seems my first attempt wasn't successful) . This is what I see in the logs of rpc-snoop when I connect the validator to the exposed port of rpc-snoop which forwards everything to the nimbus beacon client:

Aug 28 08:26:39.294 2022 WARNING: request not formatted as JSON-RPC request [EOF while parsing a value at line 1 column 0]:

Edit: Mhh, I think there is still something going on in my setup with rpc-snoop as I just realized that the log I posted is a very general one and has nothing to do with the error I receive from the validator while I connect it directly to the nimbus beacon client. Sorry for the inconvenience.

Edit2: Did you ever use rpc-snoop with a validator/beacon node combination because I would interpret that error as a consequence that the validator connects to the beacon node via REST and not via json-rpc. So the tool cannot work with a vc/bn combination.

[dapplion]

Oh you are right, sorry for pointing to the wrong tool. snooper doesn't seem to support paths. I've been researching but didn't find an alternative snooper that logged arbitrary REST calls.

There's an open PR https://github.com/ChainSafe/lodestar/pull/4493 to ensure programatically that Lodestar follows the OpenAPI spec for the beacon API. An inconsistency it found was that the body of getAttesterDuties and getSyncCommitteeDuties send an array of uint64 instead of an array of strings. If Nimbus parsing is very strict that would explain the error.