beacon metadata missing

[henridf]

I'm running armiarma eth2 and not getting any of the beacon metadata fields. The logs indicate mostly "deadline exceeded" errors, and some others. Before I dig further into this, any obvious gotchas that I may be missing?

(note this is not the verbatim log, it is grepped for "metadata")

time="2022-07-26T16:23:44-07:00" level=debug msg="ReqMetadata Peer: 16Uiu2HAmU23jf7WQxMQUQpWKCJdgVrU4x98iks9xR5Ain8H6G2d3" ERROR="context deadline exceeded" module=LIBP2P-HOST
time="2022-07-26T16:23:45-07:00" level=debug msg="ReqMetadata Peer: 16Uiu2HAmVm7uVwKzR4QYpoEhrN8mVyb1CgekBP597rbmZz7eSkb5" ERROR="context deadline exceeded" module=LIBP2P-HOST
time="2022-07-26T16:23:45-07:00" level=debug msg="ReqMetadata Peer: 16Uiu2HAmB8EATz3JaS3MGEtyduJLug7Cc79wVh51rFJkXGnBGiLF" ERROR="context deadline exceeded" module=LIBP2P-HOST
time="2022-07-26T16:23:45-07:00" level=debug msg="ReqMetadata Peer: 16Uiu2HAm6xfMBzEPBA1si5G8rgSWVZMqqBs9aeBFxDePFYGRhn24" ERROR="context deadline exceeded" module=LIBP2P-HOST
time="2022-07-26T16:23:45-07:00" level=debug msg="ReqMetadata Peer: 16Uiu2HAm59TzKHD4rHZHFcMDMmxFBFH2Nqfo4XaA48P4MoovGmFw" ERROR="context deadline exceeded" module=LIBP2P-HOST
time="2022-07-26T16:23:45-07:00" level=debug msg="ReqMetadata Peer: 16Uiu2HAm1mUQfc1wR6tSHAc25Ya9PP5o5bEi2XAbSERU41DMshuM" ERROR="failed to read chunk 0 result byte: closed stream" module=LIBP2P-HOST
time="2022-07-26T16:23:45-07:00" level=debug msg="ReqMetadata Peer: 16Uiu2HAmE67h2tZatmsZwz8eRPua1gYpkggvs7fgbGsnhnhpQdxF" ERROR="context deadline exceeded" module=LIBP2P-HOST
time="2022-07-26T16:23:45-07:00" level=debug msg="ReqMetadata Peer: 16Uiu2HAmFPihXrfpxzNUksmFYBbQJT1sjZ8mKygtiticW6s1qRSx" ERROR="context deadline exceeded" module=LIBP2P-HOST
time="2022-07-26T16:23:46-07:00" level=debug msg="ReqMetadata Peer: 16Uiu2HAm7QBmaAP13qUXEUdiuytmPaUfCjpA45MtKUiAbvnmWgcE" ERROR="context deadline exceeded" module=LIBP2P-HOST
time="2022-07-26T16:23:46-07:00" level=debug msg="ReqMetadata Peer: 16Uiu2HAmDkzgqaNpsw5Qd8CnDVuNKmJWw6tJmyFAvUUnJBJahaJQ" ERROR="context deadline exceeded" module=LIBP2P-HOST
time="2022-07-26T16:23:46-07:00" level=debug msg="ReqMetadata Peer: 16Uiu2HAm3uqmFmnBoLAUaasBMLD89z7o3FxWn9GnwukZai9isjDw" ERROR="context deadline exceeded" module=LIBP2P-HOST
time="2022-07-26T16:23:46-07:00" level=debug msg="ReqMetadata Peer: 16Uiu2HAmM2RTALFXApXphK6UwCCsmcqNHj2zVnpVHgYtvhtfMTx6" ERROR="context deadline exceeded" module=LIBP2P-HOST
time="2022-07-26T16:23:46-07:00" level=debug msg="ReqMetadata Peer: 16Uiu2HAkyujMxJNySf9MTyKn2eXVE6qHkWRv121ZMccP8nxc5Fos" ERROR="context deadline exceeded" module=LIBP2P-HOST
time="2022-07-26T16:23:47-07:00" level=debug msg="ReqMetadata Peer: 16Uiu2HAm4y3WS7tPwixthVUUiLDwNJZmW33rmv3m3bo3HWTFzEq7" ERROR="protocol not supported" module=LIBP2P-HOST
time="2022-07-26T16:23:48-07:00" level=debug msg="ReqMetadata Peer: 16Uiu2HAm8DdMJpDi21QQYXqigrLicBjGeb8D73DMA19gAMx9D3ek" ERROR="protocol not supported" module=LIBP2P-HOST
time="2022-07-26T16:23:51-07:00" level=debug msg="ReqMetadata Peer: 16Uiu2HAm5d6ySV8k1Yxa9uDvDp7aALvpL22rezoy2em5gzY414QZ" ERROR="context deadline exceeded" module=LIBP2P-HOST
time="2022-07-26T16:23:54-07:00" level=debug msg="ReqMetadata Peer: 16Uiu2HAmJzxHPgv6qaE3EBPq7bkZhAEDgzmwXioRk2Z3krJAUeck" ERROR="protocol not supported" module=LIBP2P-HOST
time="2022-07-26T16:23:55-07:00" level=debug msg="ReqMetadata Peer: 16Uiu2HAm4n8vadewKeCRTuZgK2vNjJAzrSVKgwGS7wkAYRpMVEC8" ERROR="stream reset" module=LIBP2P-HOST

[henridf]

Adding a few comments/questions as I poke around:

• the "context deadline exceeded" errors are for peers where getting the host info failed (why/when does this typically fail?)
• the "protocol not supported" errors are for peers where the host info was successfully obtained.
• this lead me to looking at MetaDataRPCv1 where the protocol is marked as /eth2/beacon_chain/req/metadata/1/ssz. But based on the spec, shouldn't the encoding be ssz_snappy? (I did try with that but no change in results)

[cortze]

Hey @henridf ! Thank you very much for being active in the repo!

About the errors that were listed in the Issue:

• You are right about the context deadline exceeded error. By using the connection notifier of the Libp2p.Host, I'm trying to re-use the libp2p stream to request all the metadata for every connected peer (incoming or outcoming connection). For some reason, there are some not-so-rare cases where the host.Connect() function in peering/peering.go raises the connection notifier while the function itself reports a connection error context deadline exceed (I didn't have much time to take a look at it). In this case, we are trying to use a Libp2p.Stream linked to a no-longer live context.

• About the protocol not supported, initially, RPC requests with plain ssz serialization were supported along the snappy compressed ones by every client, that is why I didn't bother to change them (shame on me 😓).

• Now, after the Altair fork, it seems they have added an extra field in the RPC response-chunk, which we are not supporting (yet) and that causes the failed to read chunk 0 result byte: closed stream" one.

I won't have much time to dig into this over this week, although I might be able to fix this over the late-next one.

Also, just for you to know, I'm planning on reorganizing the crawler over the summer (especially the SQL database), so let me know or feel free to open an issue if there is something you think is missing.

[henridf]

Thanks for the quick response @cortze ! I'm going to take a stab at addressing these issues.

Also, just for you to know, I'm planning on reorganizing the crawler over the summer (especially the SQL database), so let me know or feel free to open an issue if there is something you think is missing.

Interesting and thanks for heads up. Is this a general refactor/cleanup type of reorg or do you have any feature/functional changes in mind?

[cortze]

It will be a general clean up refactor with a specific focus on the SQL database. I really think that the code can be refactored in a more modular way and it starts by cleaning up the DB.

After that, I'm thinking of adding more GossipSub-oriented metrics or submodules, but the overall functionality of the tool should remain intact. We are, in fact, using it as the main data source for our public dashboard

[cortze]

closing this, as it is solved in #52