search

celestiaorg / celestia-node

Celestia Data Availability Nodes
Apache License 2.0
915 stars 913 forks source link

serving request {"endpoint": "/namespaced_data", "err": "getter/shrex: client-nd: verifying response: row verification failed: row 0 doesn't match original root: ... #2066

Open jcstein opened 1 year ago

jcstein commented 1 year ago

Celestia Node version

v0.8.2

OS

Ubuntu

Install tools

No response

Others

No response

Steps to reproduce it

Running https://github.com/rollkit/gm on top of this, occasionally see this error

Expected result

No error

Actual result

2023-04-13T02:44:32.375Z        INFO    header/store    store/store.go:353      new head        {"height": 236317, "hash": "2043EB2C8A48DC2B4E84EF608B5259A03242924AB2FF8379CCE67AE6A600811A"}
2023-04-13T02:44:32.376Z        INFO    das     das/subscriber.go:35    new header received via subscription      {"height": 236317}
2023-04-13T02:44:32.472Z        INFO    das     das/worker.go:80        finished sampling headers{"from": 236317, "to": 236317, "errors": 0, "finished (s)": 0.096353672}
2023-04-13T02:44:34.594Z        ERROR   shrex/nd        shrexnd/client.go:195   client-nd: request status INTERNAL returned
2023-04-13T02:44:34.616Z        ERROR   gateway gateway/util.go:9       serving request {"endpoint": "/namespaced_data", "err": "getter/shrex: client-nd: verifying response: row verification failed: ro
w 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row veri
fication failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying
 response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; c
lient-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\u
fffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\u
fffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\uff
fdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\u
fffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufff
d\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u00
12\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original roo
t: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: response code is not OK: server returned an invalid respon
se or caused an internal error; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\u
fffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,
.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\uff
fdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\uf
ffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \u
fffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't matc
h original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed:
 row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; failed to find peers: routing: not f
ound; context deadline exceeded; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\
ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; context deadline exceeded; getter/ipld: failed to retrieve shares by namespace: retrieving nID 05fafd7808fd77a7 for row 626167616f34616d6235796174616
16161616161616161616161716d79327569636f7a366266673662677672736575356f616b6f64367133727a616a35687a76616f36763670677474727471783276746f37647a62376467667979: context canceled", "errCauses": [{"error": "ge
tter/shrex: client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u
0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\uf
ffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.
ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; context deadline exceeded; getter/ipld: failed to retrieve shares by namespace: retrieving nID 05fafd7808fd77a7 for row 626167616f34616d6235796174616
16161616161616161616161716d79327569636f7a366266673662677672736575356f616b6f64367133727a616a35687a76616f36763670677474727471783276746f37647a62376467667979: context canceled", "errCauses": [{"error": "ge
tter/shrex: client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u
0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\uf
ffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.
\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufff
dB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\uff
fd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \uf
ffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match
 original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed:
row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row ve
rification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifyi
ng response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd;
 client-nd: response code is not OK: server returned an invalid response or caused an internal error; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\u
fffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match orig
inal root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0
 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying response: row verific
ation failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; client-nd: verifying re
sponse: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; clie
nt-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufff
d\ufffd\ufffd; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufff
d\u0011\u00107\ufffd\ufffd\ufffd; failed to find peers: routing: not found; context deadline exceeded; client-nd: verifying response: row verification failed: row 0 doesn't match original root: \ufffd\
ufffd\u0012\ufffd\ufffd\ufffd\ufffdB<Om\u0017\ufffd,.\ufffdzz]\ufffdp\ufffdAj\ufffd\ufffd\u0011\u00107\ufffd\ufffd\ufffd; context deadline exceeded"}, {"error": "getter/ipld: failed to retrieve shares
by namespace: retrieving nID 05fafd7808fd77a7 for row 626167616f34616d623579617461616161616161616161616161716d79327569636f7a366266673662677672736575356f616b6f64367133727a616a35687a76616f367636706774747
27471783276746f37647a62376467667979: context canceled"}]}
2023-04-13T02:44:43.315Z        INFO    net/identify    identify/id.go:396      failed negotiate identify protocol with peer      {"peer": "12D3KooWARVnGZsAUCUZ9UhU94tsvW3caK2WpCnwpqUqkKeVNJG8", "error
": "Application error 0x0 (local)"}
2023-04-13T02:44:43.315Z        WARN    net/identify    identify/id.go:371      failed to identify 12D3KooWARVnGZsAUCUZ9UhU94tsvW3caK2WpCnwpqUqkKeVNJG8: Application error 0x0 (local)

Relevant log output

No response

Notes

No response

Wondertan commented 1 year ago

Oooops, this is very concerning

jcstein commented 1 year ago

Link to issue: https://github.com/celestiaorg/celestia-node/issues/2051

This is also related from a user in discord: image Thread: https://discord.com/channels/638338779505229824/1093868258802814986/1094027383713046568

jcstein commented 1 year ago

And here https://github.com/rollkit/rollkit/issues/861