Open walldiss opened 4 months ago
hey @walldiss so i will put here information that seems relevant to the issue. hope that will help.
so today we had the issue for around 20 min
issue was resolved with a manual restart
here are the logs captured just before the issue and after
2024-04-25T07:51:00.825Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "80A22C1AC58BFB6482A0ECFB8DD045C971A98FC0B13E5779459AFB93C1545A6E"}
2024-04-25T07:51:09.589Z INFO header/store store/store.go:375 new head {"height": 1685431, "hash": "F6B9A16D324B16E492DA9F2B3F8A2D0E32691EF41FB056D6F887D7CAE9A73C6E"}
2024-04-25T07:51:16.904Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "8CA74145AD1A865FB99727428D03EC0166D995725574A860053D56861DA77C37"}
2024-04-25T07:51:21.544Z INFO header/store store/store.go:375 new head {"height": 1685432, "hash": "D284D7162C514311161AFE9DD6CB6E55A7BE6BF901A88A36AB065DA2ABC23D7C"}
2024-04-25T07:51:22.336Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T07:51:27.981Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "C36C693ABD7E239572F93481C9D01111C4CBCB775B4E219AFCEB196476844A02"}
2024-04-25T07:51:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2432177 miss: 61534984 keys-added: 5820577 keys-updated: 13703 keys-evicted: 5820556 cost-added: 60101415840754 cost-evicted: 60101154390680 sets-dropped: 191 sets-rejected: 48316623 gets-dropped: 4414016 gets-kept: 59550016 gets-total: 63967161 hit-ratio: 0.04
2024-04-25T07:51:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 5820366
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 5814781 99.90% 99.90%
[2, 4) 4497 0.08% 99.98%
[4, 8) 1004 0.02% 100.00%
[8, 16) 83 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T07:51:35.936Z INFO canonical-log swarm/swarm_dial.go:559 CANONICAL_PEER_STATUS: peer=12D3KooWEjMK8hk451SvUixvSC3eA4Mj2jqxwhgTvuQGJ71wcV1r addr=/ip6/2607:5300:203:6730::/udp/2121/quic-v1 sample_rate=100 connection_status="established" dir="outbound"
2024-04-25T07:51:41.668Z ERROR shrex/eds shrexeds/server.go:120 server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "DD8A7EA64D5BC8DA35A9607B63886E7375F619BA1BE5B12682281822A71FE0B4", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to acquire reader of mount so we can return the accessor: open /home/pops/.celestia-bridge-mocha-4/blocks/DD8A7EA64D5BC8DA35A9607B63886E7375F619BA1BE5B12682281822A71FE0B4: too many open files"}
2024-04-25T07:51:42.114Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T07:51:55.177Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "B79369019A087FD68B234E81E24B7EA9CB5905572262B5571352ACD342C6EDEB"}
2024-04-25T07:52:09.108Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "158E8D9F71F7EE6E2923782118656C09F132BD9A8C709DAAD1456E241639DE74"}
2024-04-25T07:52:11.544Z INFO core core/listener.go:201 listener: listening stopped
2024-04-25T07:52:11.545Z WARN core core/listener.go:134 listener: subscriber error, resubscribing... {"err": "underlying subscription is stuck"}
2024-04-25T07:52:18.977Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "268B0C7C095F81FC23F0CFD47601628C60CD472F1540883E1DEF8712BD5EFE42"}
2024-04-25T07:52:23.875Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "D406C8E570566EBAE40558B66559529F08D08EC343412D547A9CBB5FB4183693"}
2024-04-25T07:52:28.815Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "8C6A1B85FC532538E7300E6F0371E1007633C38C7F8E6C0C588698B3DD9A4376"}
2024-04-25T07:52:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2446119 miss: 61823594 keys-added: 5846729 keys-updated: 13773 keys-evicted: 5846716 cost-added: 60383210996091 cost-evicted: 60382948333249 sets-dropped: 191 sets-rejected: 48579011 gets-dropped: 4414016 gets-kept: 59852672 gets-total: 64269713 hit-ratio: 0.04
2024-04-25T07:52:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 5846526
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 5840888 99.90% 99.90%
[2, 4) 4539 0.08% 99.98%
[4, 8) 1013 0.02% 100.00%
[8, 16) 85 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T07:52:49.450Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "25ECCF519AB6A8A6967F34B90933B6A7E101E014BA87BAC824D322FBD08BC7CA"}
2024-04-25T07:52:52.111Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T07:53:01.185Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "BC75CA8425AFD344C0E2AE42EA9DF96D9AED2ACE809EE4C8F7D9F010E3BED11E"}
2024-04-25T07:53:16.177Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "1D9B5337C3A6A2BD0CE7E484D0324567E62D50BB8A5F28DD1860131727184AAB"}
2024-04-25T07:53:17.593Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T07:53:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2456866 miss: 62052959 keys-added: 5868417 keys-updated: 13824 keys-evicted: 5868394 cost-added: 60622560334831 cost-evicted: 60622294449708 sets-dropped: 191 sets-rejected: 48786637 gets-dropped: 4414016 gets-kept: 60092864 gets-total: 64509825 hit-ratio: 0.04
2024-04-25T07:53:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 5868204
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 5862515 99.90% 99.90%
[2, 4) 4584 0.08% 99.98%
[4, 8) 1019 0.02% 100.00%
[8, 16) 85 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T07:53:42.662Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T07:53:44.113Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "C93A7F6AF89660C45430B2789E26422962A1913899275DFFEE433E4E24DA7F19"}
2024-04-25T07:53:58.154Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "257997AAA44BBC5A996FC94E106738910EAEFE5D32AB3CB3240121238039E342"}
2024-04-25T07:54:22.114Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T07:54:32.000Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "1DFDC94B427F670BF162D61F5C4A3CB3F3B634221986EDB6C09F8B6E98B3DEFC"}
2024-04-25T07:54:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2468315 miss: 62299488 keys-added: 5892560 keys-updated: 13873 keys-evicted: 5892545 cost-added: 60863255870186 cost-evicted: 60862998787177 sets-dropped: 191 sets-rejected: 49008974 gets-dropped: 4414080 gets-kept: 60350784 gets-total: 64767803 hit-ratio: 0.04
2024-04-25T07:54:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 5892355
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 5886608 99.90% 99.90%
[2, 4) 4626 0.08% 99.98%
[4, 8) 1033 0.02% 100.00%
[8, 16) 87 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T07:54:37.059Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T07:54:46.385Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "AB57DD8617CE7FC583A661AD48425933061428350CF52F93952F8CDA1E94E5F5"}
2024-04-25T07:54:54.872Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "4C8052FE35825456B4DFCF5D725A678B8D72617C81ABA15CDD753DD77789DDF9"}
2024-04-25T07:55:04.001Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "897194E2B7739FB971830B2A12BECA1B8439593DB51B14F29A774181F02ECD2B"}
2024-04-25T07:55:28.386Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "C76938FABE7F1E7CD35D60A25CCA7763692A3C70B26486C63D22DDE50E98CEA1"}
2024-04-25T07:55:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2476202 miss: 62476577 keys-added: 5910384 keys-updated: 13896 keys-evicted: 5910366 cost-added: 61065462695834 cost-evicted: 61065204139792 sets-dropped: 191 sets-rejected: 49168213 gets-dropped: 4414144 gets-kept: 60535616 gets-total: 64952780 hit-ratio: 0.04
2024-04-25T07:55:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 5910175
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 5904324 99.90% 99.90%
[2, 4) 4714 0.08% 99.98%
[4, 8) 1047 0.02% 100.00%
[8, 16) 89 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T07:55:36.367Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "B41752E35B81BDE6BEDD0105AEC2323B5A2D28A0CF3A618299641DBD9BED54FC"}
2024-04-25T07:55:41.221Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "2917D2836C625BBEC179A8B9732FF2D5026C89DCFCB4C1E9DCDD86361E3B33E2"}
2024-04-25T07:55:42.112Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T07:55:42.662Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T07:55:45.811Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "39F656BAE5D7CB42F43FDC4E5F188BDDE8A38C2AF3659FBAF773EA953F96E1C8"}
2024-04-25T07:55:52.050Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "81F884A07C5D404B676D221036FABB7E231EDC235D886B0F4BEC4C4C72D8A4EB"}
2024-04-25T07:55:58.376Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "8EB8023C472AD5E5C0C2D718C32C548CC18B7B47131A41D0CE13397B1F31EF08"}
2024-04-25T07:56:02.809Z ERROR shrex/eds shrexeds/server.go:120 server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "06E27242189AF9FAC12F75E6F412084CDE93F902D10BD5F576E9F3788FB38439", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to recover index for shard 06E27242189AF9FAC12F75E6F412084CDE93F902D10BD5F576E9F3788FB38439: open /home/pops/.celestia-bridge-mocha-4/index/06E27242189AF9FAC12F75E6F412084CDE93F902D10BD5F576E9F3788FB38439.full.idx: too many open files"}
2024-04-25T07:56:04.946Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "1359FEF60DF42A8E9BBEE34323C00D1EF42C2B15584AA442D2692B281269647E"}
2024-04-25T07:56:17.114Z ERROR shrex/eds shrexeds/server.go:120 server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "4576AC0D9BAC9B60071E8E8ED971067EDFF5A804DD640FC1F29EAD4B9971FC70", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to acquire reader of mount so we can return the accessor: open /home/pops/.celestia-bridge-mocha-4/blocks/4576AC0D9BAC9B60071E8E8ED971067EDFF5A804DD640FC1F29EAD4B9971FC70: too many open files"}
2024-04-25T07:56:19.601Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "9E54C93D31AE2CBF40852EDEE90B181FEB374DAFD6BEF27054A748CFD4AEE189"}
2024-04-25T07:56:32.614Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "39F86185E4246BEF6BF1F1568C77D9BBEE935D70F4E697541E063B6A8F2E584F"}
2024-04-25T07:56:35.058Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2486870 miss: 62702951 keys-added: 5934641 keys-updated: 13952 keys-evicted: 5934610 cost-added: 61315199697615 cost-evicted: 61314935808776 sets-dropped: 191 sets-rejected: 49370276 gets-dropped: 4414208 gets-kept: 60772672 gets-total: 65189821 hit-ratio: 0.04
2024-04-25T07:56:35.058Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 5934420
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 5928529 99.90% 99.90%
[2, 4) 4751 0.08% 99.98%
[4, 8) 1049 0.02% 100.00%
[8, 16) 90 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T07:56:43.369Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "144078B15D706E5D42CBEA2EA2A0BAC4CB9D1ABF54D692495AEAE6B3BE6CAFB9"}
2024-04-25T07:56:56.418Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "761E626C6DADC536989C4F0E8BE63ED84F447565323A08FB62C088B793F11174"}
2024-04-25T07:57:09.014Z ERROR shrex/eds shrexeds/server.go:120 server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "6FCA2825072472A6C125CD871D5AC10A727B245509D4A2381FE52E2F3784BB10", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to recover index for shard 6FCA2825072472A6C125CD871D5AC10A727B245509D4A2381FE52E2F3784BB10: open /home/pops/.celestia-bridge-mocha-4/index/6FCA2825072472A6C125CD871D5AC10A727B245509D4A2381FE52E2F3784BB10.full.idx: too many open files"}
2024-04-25T07:57:22.116Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T07:57:31.559Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "7AAF0551B755CAEF7BE63981CECF6E052508400D7DAB647F3F700C69AFEC48F2"}
2024-04-25T07:57:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2496805 miss: 62909162 keys-added: 5955219 keys-updated: 13983 keys-evicted: 5955198 cost-added: 61505584066530 cost-evicted: 61505320080300 sets-dropped: 191 sets-rejected: 49555878 gets-dropped: 4414272 gets-kept: 60988800 gets-total: 65405967 hit-ratio: 0.04
2024-04-25T07:57:35.058Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 5955008
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 5949006 99.90% 99.90%
[2, 4) 4836 0.08% 99.98%
[4, 8) 1072 0.02% 100.00%
[8, 16) 93 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T07:57:41.946Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "89C48C59D7F981818727D038A9475DA4CA26F6C4671CC4D23EFF4A98C83B518E"}
2024-04-25T07:57:42.662Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T07:58:12.181Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "8C15835E0E3406B5184B112241745B76B85166010E01CC983C1D476988F7AF46"}
2024-04-25T07:58:16.513Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "69E784EC92386B991240EA8C6FAB8978E2A4D771EB74134903E608643A8B0B78"}
2024-04-25T07:58:16.768Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T07:58:27.983Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "867F0F7CF371E8D4F262796751581F4F139A003E1FF413AD0A20ABC122A1E935"}
2024-04-25T07:58:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2508064 miss: 63143410 keys-added: 5980931 keys-updated: 14041 keys-evicted: 5980912 cost-added: 61776633321108 cost-evicted: 61776372161892 sets-dropped: 191 sets-rejected: 49764356 gets-dropped: 4414400 gets-kept: 61234176 gets-total: 65651474 hit-ratio: 0.04
2024-04-25T07:58:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 5980722
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 5974689 99.90% 99.90%
[2, 4) 4866 0.08% 99.98%
[4, 8) 1073 0.02% 100.00%
[8, 16) 93 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T07:58:40.547Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "C86830074C2E67E22D9F2A3C2C88B2F30064874F339A1D76E4036F09C6BE84D4"}
2024-04-25T07:58:42.114Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T07:58:48.817Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "152B4FF3FCC5B8184AE6AD8F7A36CA5D1A72BD13BEED68CAAE98DED0457E3DE9"}
2024-04-25T07:59:13.033Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "1FE8AB4F4E3CF679A096ACBEC0F95003E40318E3F86097B2B61ABE4DEA031A2F"}
2024-04-25T07:59:21.503Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "1E71F32AB19F7B0BC42852DC8FD621C007AC7A7A441D8FFE9A25F86CA8836A97"}
2024-04-25T07:59:27.322Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "B61E769A505647830555B0393C0F3997862A0627642001CFC04A5C1DEC140F21"}
2024-04-25T07:59:35.058Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2517378 miss: 63339663 keys-added: 6002317 keys-updated: 14085 keys-evicted: 6002304 cost-added: 61993113001270 cost-evicted: 61992845144346 sets-dropped: 191 sets-rejected: 49939179 gets-dropped: 4414464 gets-kept: 61439552 gets-total: 65857041 hit-ratio: 0.04
2024-04-25T07:59:35.058Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 6002114
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 5995993 99.90% 99.90%
[2, 4) 4934 0.08% 99.98%
[4, 8) 1091 0.02% 100.00%
[8, 16) 95 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T07:59:42.663Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T07:59:47.250Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "8669CFB0C8FED83F5811CB7B7E7A833CCFF43FE525598F0598B677AB51692C1F"}
2024-04-25T07:59:58.667Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "1A40C1C5F70C43B799396D7CA8B2F45E77152C5F045DD4730C1DAF73EB255D5C"}
2024-04-25T08:00:04.435Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "E2475D5C780A4D870B6F21F9A3FE9DC36D8AA3217F0955737301A510FB244C10"}
2024-04-25T08:00:09.504Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "E346C56C569B3543C9823051CC9F72D817770F0CB2965E36042788D8D8508B38"}
2024-04-25T08:00:15.436Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "7B80F9156F03F6C2921CEB6048BF8857AF4C333B310B977F074A615BFCC61DE5"}
2024-04-25T08:00:22.114Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:00:23.813Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "54D2BCA4072515039B19E60E2DE14F1768A656EDE819A144324E6B0C28758622"}
2024-04-25T08:00:25.780Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "499F39C762BC2CD9F72223E354470CA61DA942BA4CD84A05FE77F929E297841E"}
2024-04-25T08:00:25.881Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEhsc7GmWo9VcexP4SnRymojTaWqQUM7RxVF111bZxVp1", "hash": "499F39C762BC2CD9F72223E354470CA61DA942BA4CD84A05FE77F929E297841E"}
2024-04-25T08:00:26.106Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWNEDvy2uYwBrtic5BzfDioKHDdSoTC72jwzHewNoVv1B3", "hash": "499F39C762BC2CD9F72223E354470CA61DA942BA4CD84A05FE77F929E297841E"}
2024-04-25T08:00:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2529947 miss: 63602091 keys-added: 6029169 keys-updated: 14141 keys-evicted: 6029156 cost-added: 62265398471922 cost-evicted: 62265131402218 sets-dropped: 191 sets-rejected: 50174699 gets-dropped: 4414464 gets-kept: 61714624 gets-total: 66132038 hit-ratio: 0.04
2024-04-25T08:00:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 6028966
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 6022828 99.90% 99.90%
[2, 4) 4949 0.08% 99.98%
[4, 8) 1093 0.02% 100.00%
[8, 16) 95 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T08:00:38.421Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWNEDvy2uYwBrtic5BzfDioKHDdSoTC72jwzHewNoVv1B3", "hash": "4EE5E09736A4F410D671B817F3BFB19296798F2468838585EE1F944919A6A27A"}
2024-04-25T08:00:39.792Z ERROR shrex/eds shrexeds/server.go:120 server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "34EF2288D12F5970BA428390A0B0AE8F864F5E2C0C45F6C231BC1DEFE44FF864", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to acquire reader of mount so we can return the accessor: open /home/pops/.celestia-bridge-mocha-4/blocks/34EF2288D12F5970BA428390A0B0AE8F864F5E2C0C45F6C231BC1DEFE44FF864: too many open files"}
2024-04-25T08:00:49.823Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "6078DA78BC0ED4B04B73DA0AC8DB546889C30E52446FDDC6FA6B99F5A3FCFB4B"}
2024-04-25T08:00:49.966Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEhsc7GmWo9VcexP4SnRymojTaWqQUM7RxVF111bZxVp1", "hash": "195F34CF7E89AD076F1B7B42B862A2523D305A36E1B98737F8A68DA8011F9EF3"}
2024-04-25T08:00:53.703Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "6F2F6C8F9F8BA70DF4D10014DEACEA1598F011013918320DBB48B40A4787B8DF"}
2024-04-25T08:00:57.296Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "5A57563C32F522FD389F1A5EDA13BC1EFB16E3A963BA013F85AEAD7303F1EF6C"}
2024-04-25T08:01:03.737Z ERROR shrex/eds shrexeds/server.go:120 server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "C4DF337245B41821F21CF8EA8CEF923F7A790C0987B50E156B9F88958F06E791", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to acquire reader of mount so we can return the accessor: open /home/pops/.celestia-bridge-mocha-4/blocks/C4DF337245B41821F21CF8EA8CEF923F7A790C0987B50E156B9F88958F06E791: too many open files"}
2024-04-25T08:01:06.758Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "80EC66847272995B42EC1B3226DE0922C15137002758715A45EE377330B9C97E"}
2024-04-25T08:01:10.575Z ERROR shrex/eds shrexeds/server.go:120 server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "D38ECBD79DECD642F5D109571773DC16F77B35E19B9D6BBF7497756033C4F664", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to acquire reader of mount so we can return the accessor: open /home/pops/.celestia-bridge-mocha-4/blocks/D38ECBD79DECD642F5D109571773DC16F77B35E19B9D6BBF7497756033C4F664: too many open files"}
2024-04-25T08:01:13.099Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "6F245B89A9BB52FB0B19E899C9DC7D26DB40AE8C3F23AE3437586C695E6759F6"}
2024-04-25T08:01:17.353Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "26BEAFA950AC9CAC56A844491AE269F7AD2A8450B8847F2D875B7BFB02846C35"}
2024-04-25T08:01:24.559Z ERROR shrex/eds shrexeds/server.go:120 server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "28F7B44497104B64BBFC9C17CD6D1FC2862215867D37BDC3DC8E172585CB44AC", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to acquire reader of mount so we can return the accessor: open /home/pops/.celestia-bridge-mocha-4/blocks/28F7B44497104B64BBFC9C17CD6D1FC2862215867D37BDC3DC8E172585CB44AC: too many open files"}
2024-04-25T08:01:25.152Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEhsc7GmWo9VcexP4SnRymojTaWqQUM7RxVF111bZxVp1", "hash": "01864E5FA7FB0DA4A0AEDA1EC256EE8EFB41CE91C6C0F39180CA5220EA420F54"}
2024-04-25T08:01:30.081Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "2803D7651F963BFCDECF1DB043AEAC4F56194726DD316F5E448AA221652491B6"}
2024-04-25T08:01:32.115Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:01:35.058Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2540507 miss: 63817284 keys-added: 6050291 keys-updated: 14178 keys-evicted: 6050268 cost-added: 62484802899070 cost-evicted: 62484545874197 sets-dropped: 191 sets-rejected: 50368733 gets-dropped: 4414464 gets-kept: 61940480 gets-total: 66357791 hit-ratio: 0.04
2024-04-25T08:01:35.058Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 6050078
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 6043853 99.90% 99.90%
[2, 4) 5018 0.08% 99.98%
[4, 8) 1109 0.02% 100.00%
[8, 16) 97 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T08:01:42.235Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "F403F032A1EA7D44DD223806F18F1AB74202971B9F5A72D956635AA0570A487D"}
2024-04-25T08:01:42.662Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:01:45.058Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:01:45.398Z INFO canonical-log swarm/swarm_listen.go:136 CANONICAL_PEER_STATUS: peer=12D3KooWJ5dfgoAXpG2hsvXhtxTCp5BsxEWkzGWhbmns8zzf9Z2a addr=/ip4/54.191.40.227/udp/2121/quic-v1 sample_rate=100 connection_status="established" dir="inbound"
2024-04-25T08:01:52.571Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:02:02.691Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "2356198DF2B17027119A58F5098D5300AA47A6E902CB75926A80D07885BD9189"}
2024-04-25T08:02:05.260Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:02:08.822Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "7926C35BAACFCB20E6340AEFE16B9B699775A575AE76B2D206217C8A8C8A3068"}
2024-04-25T08:02:17.592Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:02:22.306Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "8B72747F91C2E47ED205CCEBFB49D14AA917F6F4B6272B6662104771D9D8E0AC"}
2024-04-25T08:02:28.325Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "1D17FFADCE03F7DB145ABE5CD4574BF7AB835279883B8FCA228CCF0BDEFDB663"}
2024-04-25T08:02:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2549995 miss: 64026217 keys-added: 6071197 keys-updated: 14219 keys-evicted: 6071175 cost-added: 62716680272679 cost-evicted: 62716440085157 sets-dropped: 191 sets-rejected: 50556719 gets-dropped: 4414464 gets-kept: 62158848 gets-total: 66576212 hit-ratio: 0.04
2024-04-25T08:02:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 6070985
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 6064710 99.90% 99.90%
[2, 4) 5064 0.08% 99.98%
[4, 8) 1113 0.02% 100.00%
[8, 16) 97 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T08:02:42.113Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:02:52.663Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:03:02.498Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "7FF1B0946DC8038981C5AC2E0089166D2F65CED4841E5AE753F55E6833C3AF79"}
2024-04-25T08:03:07.804Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "8025EC3F6A82D7BE5CADD1BFAADC713CA187B0712C9E1D29E7690700BDC0D8D9"}
2024-04-25T08:03:26.378Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "25375639C919C84FE9FF794F0763AFA1F94BA60BA6494C4EE5DE70EFA9644632"}
2024-04-25T08:03:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2557267 miss: 64187707 keys-added: 6086830 keys-updated: 14251 keys-evicted: 6086809 cost-added: 62890035729757 cost-evicted: 62889788131265 sets-dropped: 191 sets-rejected: 50702544 gets-dropped: 4414464 gets-kept: 62327616 gets-total: 66744974 hit-ratio: 0.04
2024-04-25T08:03:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 6086619
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 6080248 99.90% 99.90%
[2, 4) 5155 0.08% 99.98%
[4, 8) 1118 0.02% 100.00%
[8, 16) 97 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T08:04:10.705Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "CDC851B01241D59C13C1741105196A3F3C5D7A46941EC5F00B8BEC534ACE142A"}
2024-04-25T08:04:18.656Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "E3FBD06EB3D7A0AF5FE9166F8AAA8EAE7ED556105C0F84C18B4400DEA2534FCC"}
2024-04-25T08:04:18.819Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "78B586631F989A43DFD17808D2D2053256F86220D7057D7E98C5A274CDB73FEA"}
2024-04-25T08:04:22.114Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:04:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2565874 miss: 64377447 keys-added: 6104628 keys-updated: 14283 keys-evicted: 6104608 cost-added: 63074480913736 cost-evicted: 63074230159181 sets-dropped: 191 sets-rejected: 50874454 gets-dropped: 4414464 gets-kept: 62525888 gets-total: 66943321 hit-ratio: 0.04
2024-04-25T08:04:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 6104418
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 6097941 99.89% 99.89%
[2, 4) 5229 0.09% 99.98%
[4, 8) 1141 0.02% 100.00%
[8, 16) 106 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T08:04:42.662Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:04:52.383Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:04:54.566Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "B730DFD143FCCAAA729476FF88D5A185D0A1431042FD76F7C8CFC65DCB056345"}
2024-04-25T08:04:57.400Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:05:32.275Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "AFC302CAF4BE1F510F86F007C871082000E2F475BD5A5A82020307A859D8BD2C"}
2024-04-25T08:05:34.107Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:05:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2573486 miss: 64555066 keys-added: 6121690 keys-updated: 14304 keys-evicted: 6121664 cost-added: 63257915484247 cost-evicted: 63257658419777 sets-dropped: 191 sets-rejected: 51034987 gets-dropped: 4414464 gets-kept: 62711104 gets-total: 67128552 hit-ratio: 0.04
2024-04-25T08:05:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 6121473
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 6114915 99.89% 99.89%
[2, 4) 5299 0.09% 99.98%
[4, 8) 1151 0.02% 100.00%
[8, 16) 107 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T08:05:37.094Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "198675CD416AC9E8683078F684F31A9B7B03B8880449737A8F37D0340F0DE721"}
2024-04-25T08:05:41.729Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "208DE8C41F9BCAEEE03CFCB4DE52C08481F7F5F5DE213FC37EC226FEA294E28A"}
2024-04-25T08:05:42.113Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:05:51.581Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "E72FA5C69D5606399A45779A74E94F5CED7C760970D49BAD41EBE6D1955E4BA9"}
2024-04-25T08:05:56.242Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "1CEF49FC429D15905A56536D7A61E4F5E7BA0CE8064420A799BAD41496725393"}
2024-04-25T08:06:03.848Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "18EDB3A05FD7D380DE9B89B062DA96774AB3F79BF7450849DED0A639752DE049"}
2024-04-25T08:06:17.594Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:06:24.726Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "23CA4779BB5D99934E780BD1D392EBDB9DA2D2C193CCD9B52B7C85052E291C42"}
2024-04-25T08:06:29.646Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "E4DB13552DD58FFEE8D79A2B143CAE146F5AE24300F5C42AA1E4BEDD0A582EB3"}
2024-04-25T08:06:34.670Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "E0ECDA66217420CAACA75832CF88DF398BB103C06B48183876D8665A2E6AF0CC"}
2024-04-25T08:06:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2581764 miss: 64734414 keys-added: 6139809 keys-updated: 14340 keys-evicted: 6139778 cost-added: 63461873008696 cost-evicted: 63461611787744 sets-dropped: 191 sets-rejected: 51196183 gets-dropped: 4414592 gets-kept: 62898688 gets-total: 67316178 hit-ratio: 0.04
2024-04-25T08:06:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 6139588
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 6132951 99.89% 99.89%
[2, 4) 5359 0.09% 99.98%
[4, 8) 1167 0.02% 100.00%
[8, 16) 110 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T08:06:40.052Z ERROR shrex/eds shrexeds/server.go:120 server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "ECE59C544DB0DBF97E07476A9DCE748E071811D17D22063677FF8DD3A4480DA5", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to acquire reader of mount so we can return the accessor: open /home/pops/.celestia-bridge-mocha-4/blocks/ECE59C544DB0DBF97E07476A9DCE748E071811D17D22063677FF8DD3A4480DA5: no such file or directory"}
2024-04-25T08:06:42.662Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:06:42.676Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "CC1A4A39102972A262B9838DE5ED522617C428F74B384221F7F0D14D2DBCE24A"}
2024-04-25T08:06:47.709Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "F2408B3EA85FB4180E61BD39F7A9B6AE1B54AAB5E932B383304E823B4349C321"}
2024-04-25T08:06:52.111Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:06:52.387Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "18CED3824302CF79512BA865F88D085243F8339325727ED5CB0CCB27F24A1683"}
2024-04-25T08:06:55.634Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "C45FDA45BD6D1E3224085B0818B21E664CAF83CA8D8F880228B8652E51A675C9"}
2024-04-25T08:06:58.821Z ERROR shrex/eds shrexeds/server.go:120 server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "ABDAD379E096FF647535C92E223A87BE52523B1DE7BB9A5F5A15E84BDF20F0A0", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to acquire reader of mount so we can return the accessor: open /home/pops/.celestia-bridge-mocha-4/blocks/ABDAD379E096FF647535C92E223A87BE52523B1DE7BB9A5F5A15E84BDF20F0A0: too many open files"}
2024-04-25T08:07:01.315Z ERROR shrex/eds shrexeds/server.go:120 server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "EBFA469A59788F4DEB110DABE40E5261E6D968AE198C5A494C9270B6D78EC901", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to acquire reader of mount so we can return the accessor: open /home/pops/.celestia-bridge-mocha-4/blocks/EBFA469A59788F4DEB110DABE40E5261E6D968AE198C5A494C9270B6D78EC901: too many open files"}
2024-04-25T08:07:03.688Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "E9F2D6E4B41AED0F80B5C9BFE45A40D24AB7AAA2276F5CD1EB98E07FFEFC2F25"}
2024-04-25T08:07:09.514Z ERROR shrex/eds shrexeds/server.go:120 server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "3CD5329E3B82673F7393039FA4D0346CAF6B27667088DE453A541B49956A1597", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to recover index for shard 3CD5329E3B82673F7393039FA4D0346CAF6B27667088DE453A541B49956A1597: open /home/pops/.celestia-bridge-mocha-4/index/3CD5329E3B82673F7393039FA4D0346CAF6B27667088DE453A541B49956A1597.full.idx: too many open files"}
2024-04-25T08:07:12.829Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "E006FF1CF3DC10F9D60503A21E5942A59B7C4E1FD5844DCDCFB5E410A6EBAD82"}
2024-04-25T08:07:16.293Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "824573E032C9C68DFA0F7C27BDDD786368D41622C890F35CE40C56966AD478CD"}
2024-04-25T08:07:19.644Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "87A693BFC32DEB6E74398D673274E50F3B086D8AE1B5D9E1EAB2C45897C3AA27"}
2024-04-25T08:07:25.998Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "27CC927CEC0D24E51AD6B51F125D4660C73FB6F690D142C4D4E352E07230D6D0"}
2024-04-25T08:07:35.058Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2590472 miss: 64927948 keys-added: 6157559 keys-updated: 14374 keys-evicted: 6157535 cost-added: 63642968724664 cost-evicted: 63642704130515 sets-dropped: 191 sets-rejected: 51371933 gets-dropped: 4414848 gets-kept: 63100544 gets-total: 67518420 hit-ratio: 0.04
2024-04-25T08:07:35.058Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 6157345
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 6150587 99.89% 99.89%
[2, 4) 5454 0.09% 99.98%
[4, 8) 1190 0.02% 100.00%
[8, 16) 113 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T08:07:43.771Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "7C3A1ABDF12367045C17B0677574B4D20F3BAE593E734A6B65295B028D54F11C"}
2024-04-25T08:08:14.021Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "272979BAD3159480207DAC715156ED860FFF9132524BEA96EB6A29E90C26960A"}
2024-04-25T08:08:22.113Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:08:24.134Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "4B8B7BBC06C1DB078AB391EF19C68078F2B910E5292F21D01647958612D2F70B"}
2024-04-25T08:08:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2600072 miss: 65141854 keys-added: 6180385 keys-updated: 14410 keys-evicted: 6180362 cost-added: 63902059741804 cost-evicted: 63901800223991 sets-dropped: 191 sets-rejected: 51562947 gets-dropped: 4414912 gets-kept: 63324160 gets-total: 67741927 hit-ratio: 0.04
2024-04-25T08:08:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 6180172
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 6173381 99.89% 99.89%
[2, 4) 5486 0.09% 99.98%
[4, 8) 1191 0.02% 100.00%
[8, 16) 113 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T08:08:36.374Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWNEDvy2uYwBrtic5BzfDioKHDdSoTC72jwzHewNoVv1B3", "hash": "A630D138C083271F0878CEC1D3A00836CF70C932C8059DFCF1C88499089D120E"}
2024-04-25T08:08:42.589Z INFO rpc rpc/server.go:108 server stopped
2024-04-25T08:08:42.589Z INFO header/p2p p2p/server.go:86 server: stopping server
2024-04-25T08:08:42.662Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:08:48.708Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "79FAF1AE0EB44254CD311375C562E15642EC5D3862CAB52AB32BCB969A216A9B"}
2024-04-25T08:09:17.591Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:09:25.651Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "7DE740A56174AF919DC9BAB979A58DD6C056F1281AB94814A054495EC3AB9F18"}
2024-04-25T08:09:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2601525 miss: 65172940 keys-added: 6183944 keys-updated: 14415 keys-evicted: 6183927 cost-added: 63943646189944 cost-evicted: 63943381978736 sets-dropped: 191 sets-rejected: 51590499 gets-dropped: 4414912 gets-kept: 63356544 gets-total: 67774465 hit-ratio: 0.04
2024-04-25T08:09:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 6183737
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 6176945 99.89% 99.89%
[2, 4) 5487 0.09% 99.98%
[4, 8) 1191 0.02% 100.00%
[8, 16) 113 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T08:09:37.401Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "83FEECF5B531B3716DDA0219D2712566A22D5C6D4B5D9A8AF2AD1F9507756E40"}
2024-04-25T08:09:42.113Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:09:44.300Z INFO core core/listener.go:203 listener: listening stopped
2024-04-25T08:09:44.302Z WARN shrex-sub shrexsub/pubsub.go:83 unregistering topic {"err": "no validator for topic mocha-4/eds-sub/v0.1.0"}
2024-04-25T08:09:44.303Z INFO badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:546 Lifetime L0 stalled for: 0s
2024-04-25T08:09:44.367Z INFO badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:625
Level 0 [ ]: NumTables: 01. Size: 13 MiB of 0 B. Score: 1.00->1.06 StaleData: 0 B Target FileSize: 64 MiB
Level 1 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 2 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 3 [B]: NumTables: 07. Size: 24 MiB of 25 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 4.0 MiB
Level 4 [ ]: NumTables: 46. Size: 253 MiB of 254 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 8.0 MiB
Level 5 [ ]: NumTables: 282. Size: 2.5 GiB of 2.5 GiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 16 MiB
Level 6 [ ]: NumTables: 1269. Size: 25 GiB of 25 GiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 32 MiB
Level 7 [ ]: NumTables: 5539. Size: 248 GiB of 248 GiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 64 MiB
Level Done
2024-04-25T08:09:45.910Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.910Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.910Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.910Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.910Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.910Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.910Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (local)
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (local)
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (local)
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (local)
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.912Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.912Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.912Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.912Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.912Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.912Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.912Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.914Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.914Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.914Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.914Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.914Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.914Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.914Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.914Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.914Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.914Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.914Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.916Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.916Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.916Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.916Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.916Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.916Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.916Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.916Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.916Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.917Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.917Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.917Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.917Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.917Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.917Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.917Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.917Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.917Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.919Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.919Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.919Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.919Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.919Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.919Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.919Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.919Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.919Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.920Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.920Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.920Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.920Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.920Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.920Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.920Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.920Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.920Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.921Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.921Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.921Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.921Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.921Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.921Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.921Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.921Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.922Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.922Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.923Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.923Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.926Z INFO badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:546 Lifetime L0 stalled for: 0s
2024-04-25T08:09:46.085Z INFO badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:625
Level 0 [ ]: NumTables: 00. Size: 0 B of 0 B. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 16 MiB
Level 1 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 2 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 3 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 4 [B]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 5 [ ]: NumTables: 16. Size: 55 MiB of 45 MiB. Score: 1.22->1.22 StaleData: 4.4 KiB Target FileSize: 4.0 MiB
Level 6 [ ]: NumTables: 90. Size: 454 MiB of 454 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 8.0 MiB
Level Done
2024-04-25T08:09:46.355Z INFO node nodebuilder/module.go:26 Accessing keyring...
2024-04-25T08:09:46.382Z INFO module/state state/keyring.go:40 constructed keyring signer {"backend": "test", "path": "/home/pops/.celestia-bridge-mocha-4/keys", "key name": "my_celes_key", "chain-id": "mocha-4"}
2024-04-25T08:09:46.421Z INFO badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/levels.go:171 All 106 tables opened in 24ms
2024-04-25T08:09:46.422Z INFO badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/discard.go:66 Discard stats nextEmptySlot: 48
2024-04-25T08:09:46.422Z INFO badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:368 Set nextTxnTs to 1799707427
2024-04-25T08:09:49.631Z INFO badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/levels.go:118 6619 tables out of 7144 opened in 3s
2024-04-25T08:09:49.866Z INFO badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/levels.go:171 All 7144 tables opened in 3.235s
2024-04-25T08:09:49.871Z INFO badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/discard.go:66 Discard stats nextEmptySlot: 0
2024-04-25T08:09:49.874Z INFO badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:368 Set nextTxnTs to 1071359
2024-04-25T08:09:49.878Z INFO badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/value.go:592 Deleting empty file: /home/pops/.celestia-bridge-mocha-4/inverted_index//000371.vlog
2024-04-25T08:09:56.507Z INFO basichost basic/natmgr.go:112 DiscoverNAT error:no NAT found
2024-04-25T08:09:59.874Z INFO share/discovery discovery/discovery.go:312 discovered wanted peers {"amount": 5}
2024-04-25T08:10:00.940Z INFO header/p2p p2p/subscriber.go:81 joining topic {"topic ID": "/mocha-4/header-sub/v0.0.1"}
2024-04-25T08:10:01.005Z INFO header/store store/store.go:193 loaded head {"height": 1685432, "hash": "D284D7162C514311161AFE9DD6CB6E55A7BE6BF901A88A36AB065DA2ABC23D7C"}
2024-04-25T08:10:01.005Z INFO fraudserv fraudserv/helpers.go:29 joining topic {"id": "/mocha-4/fraud-sub/badencodingv0.1/v0.0.1"}
2024-04-25T08:10:01.005Z INFO fraudserv fraudserv/service.go:114 starting fraud proof service {"protocol ID": "/mocha-4/fraud/v0.0.1"}
2024-04-25T08:10:01.446Z INFO header/sync sync/sync_head.go:140 new network head {"height": 1685527, "hash": "6EBE549FEBDD37258935D9106906E14A898ED695A48F463F404C439E8AC20369"}
2024-04-25T08:10:01.446Z INFO header/p2p p2p/server.go:77 server: listening for inbound header requests {"protocol ID": "/mocha-4/header-ex/v0.0.3"}
2024-04-25T08:10:01.447Z INFO header/sync sync/sync.go:216 syncing headers {"from": 1685433, "to": 1685527}
2024-04-25T08:10:01.447Z INFO rpc rpc/server.go:90 server started {"listening on": "0.0.0.0:26658"}
2024-04-25T08:10:01.447Z INFO node nodebuilder/node.go:113
/_____/ /_____/ /_____/ /_____/ /_____/
Started celestia DA node
node version: v0.13.4
node type: bridge
network: mocha-4
/_____/ /_____/ /_____/ /_____/ /_____/
The p2p host is listening on:
* /ip4/127.0.0.1/udp/2121/quic-v1/webtransport/certhash/uEiAktN9CODcWdqRyeVuGwDnt1Le97oOPipKvqbs0s-uTgQ/certhash/uEiC9AFyLT6belLezugelvCHt2GAQKg8fEs_fP5fVTvbzzg/p2p/12D3KooWA92eW13sbg57xdXhrjfJTJf8t3frDy1bdvfCJLSmcD9F
* /ip4/162.55.237.230/tcp/2121/p2p/12D3KooWA92eW13sbg57xdXhrjfJTJf8t3frDy1bdvfCJLSmcD9F
* /ip4/162.55.237.230/udp/2121/quic-v1/p2p/12D3KooWA92eW13sbg57xdXhrjfJTJf8t3frDy1bdvfCJLSmcD9F
* /ip4/162.55.237.230/udp/2121/quic-v1/webtransport/certhash/uEiAktN9CODcWdqRyeVuGwDnt1Le97oOPipKvqbs0s-uTgQ/certhash/uEiC9AFyLT6belLezugelvCHt2GAQKg8fEs_fP5fVTvbzzg/p2p/12D3KooWA92eW13sbg57xdXhrjfJTJf8t3frDy1bdvfCJLSmcD9F
* /ip6/::1/tcp/2121/p2p/12D3KooWA92eW13sbg57xdXhrjfJTJf8t3frDy1bdvfCJLSmcD9F
* /ip6/::1/udp/2121/quic-v1/p2p/12D3KooWA92eW13sbg57xdXhrjfJTJf8t3frDy1bdvfCJLSmcD9F
* /ip6/::1/udp/2121/quic-v1/webtransport/certhash/uEiAktN9CODcWdqRyeVuGwDnt1Le97oOPipKvqbs0s-uTgQ/certhash/uEiC9AFyLT6belLezugelvCHt2GAQKg8fEs_fP5fVTvbzzg/p2p/12D3KooWA92eW13sbg57xdXhrjfJTJf8t3frDy1bdvfCJLSmcD9F
* /ip6/2a01:4f8:271:3f6e::2/tcp/2121/p2p/12D3KooWA92eW13sbg57xdXhrjfJTJf8t3frDy1bdvfCJLSmcD9F
* /ip6/2a01:4f8:271:3f6e::2/udp/2121/quic-v1/p2p/12D3KooWA92eW13sbg57xdXhrjfJTJf8t3frDy1bdvfCJLSmcD9F
* /ip6/2a01:4f8:271:3f6e::2/udp/2121/quic-v1/webtransport/certhash/uEiAktN9CODcWdqRyeVuGwDnt1Le97oOPipKvqbs0s-uTgQ/certhash/uEiC9AFyLT6belLezugelvCHt2GAQKg8fEs_fP5fVTvbzzg/p2p/12D3KooWA92eW13sbg57xdXhrjfJTJf8t3frDy1bdvfCJLSmcD9F
2024-04-25T08:10:02.192Z INFO canonical-log tls/transport.go:96 CANONICAL_PEER_STATUS: peer= addr=/ip4/34.145.72.169/tcp/19491 sample_rate=100 handshake_failure="tls" err="read tcp4 162.55.237.230:2121->34.145.72.169:19491: read: connection reset by peer"
2024-04-25T08:10:05.050Z INFO header/store store/store.go:375 new head {"height": 1685526, "hash": "AD7F3EA202A2E973DD44C48C0E33FB9ECCAB46AFEFADE6C371C831F3F7425500"}
2024-04-25T08:10:05.050Z INFO header/store store/store.go:375 new head {"height": 1685527, "hash": "6EBE549FEBDD37258935D9106906E14A898ED695A48F463F404C439E8AC20369"}
2024-04-25T08:10:05.051Z INFO header/sync sync/sync.go:234 finished syncing headers {"from": 1685433, "to": 1685527, "elapsed time": 3.604018927}
2024-04-25T08:10:06.555Z INFO canonical-log swarm/swarm_listen.go:136 CANONICAL_PEER_STATUS: peer=12D3KooWKHePZ7CdFE9xgF7tHQxuPsj25DPFgWRuW3S7DK5gTtKk addr=/ip4/34.83.65.52/tcp/50168 sample_rate=100 connection_status="established" dir="inbound"
2024-04-25T08:10:09.447Z INFO header/store store/store.go:375 new head {"height": 1685528, "hash": "5B4E863EA638B9C750999C683F339B7A155AF66829140B589CA5E1A011560F22"}
2024-04-25T08:10:21.194Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "55020748E2B2C67A2C50F0DCD8AA72C8DA60541A422D63F1DA1439FC190C52BA"}
2024-04-25T08:10:21.475Z INFO header/store store/store.go:375 new head {"height": 1685529, "hash": "F1CBC8BAF54EF2CF2918D93B1ABCF6AACFF0F020E3DC381228A675E850AC750C"}
2024-04-25T08:10:23.317Z INFO canonical-log swarm/swarm_listen.go:136 CANONICAL_PEER_STATUS: peer=12D3KooWSi9rYk3JF9iH9ZrvUJhu5YwiKynmjsomnrtUVomHL3kB addr=/ip4/34.145.72.169/tcp/18202 sample_rate=100 connection_status="established" dir="inbound"
2024-04-25T08:10:32.888Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "CA9F8B031E4490592ADAB06A484DF8462BC7A153B11D9439525C7E91ED233FE3"}
2024-04-25T08:10:33.073Z INFO header/store store/store.go:375 new head {"height": 1685530, "hash": "BD977AFDF2E8E3243193ACA1360EE6B1094E4661E2A6B81DCCCEFA4F22943D09"}
2024-04-25T08:10:44.959Z INFO header/store store/store.go:375 new head {"height": 1685531, "hash": "99E9A7601796525AAB4FDCD061FA23EBFC8D799EFE101383F8595E93A9D4BD91"}
2024-04-25T08:10:46.572Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 6170 miss: 927875 keys-added: 12014 keys-updated: 30 keys-evicted: 11999 cost-added: 117314086439 cost-evicted: 117061167309 sets-dropped: 0 sets-rejected: 123797 gets-dropped: 306624 gets-kept: 626496 gets-total: 934045 hit-ratio: 0.01
2024-04-25T08:10:46.572Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 24
Count: 11994
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 11913 99.32% 99.32%
[2, 4) 33 0.28% 99.60%
[4, 8) 24 0.20% 99.80%
[8, 16) 11 0.09% 99.89%
[16, 32) 13 0.11% 100.00%
--
2024-04-25T08:10:51.185Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: timeout: no recent network activity
2024-04-25T08:10:52.112Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:10:56.505Z INFO header/store store/store.go:375 new head {"height": 1685532, "hash": "49A3AF2CAD3DB293A4A7A9949C674E2C0ADFB760628B49E25BC5FCC64E83D70A"}
2024-04-25T08:10:57.590Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:11:02.663Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:11:08.184Z INFO header/store store/store.go:375 new head {"height": 1685533, "hash": "0682F70785C46981C78C6699B00766B134C6D45351175141D619D6DB30A3C6FD"}
2024-04-25T08:11:19.903Z INFO header/store store/store.go:375 new head {"height": 1685534, "hash": "FA98F3F2AB4EF47D2090A86F7130EFC5E124E7B7CD84BD032BA73881300E6F9C"}
Based on the logs, it is clear that resubscribing does not help resolve the issue, which I did not observe in other bridge nodes I checked earlier today. @malise800 Thank you for providing the logs. Could you confirm if you are running multiple bridge nodes against the same RPC endpoint, and experiencing simultaneous lag across these nodes? Additionally, how often does the stall event occur?
hey @walldiss we only run one, but we utilize our public rpc for it which is in the docs. But this hasnt been a problem at all since these latest versions (dont have exact timing but we didnt see this problem like a month ago or 2 months max). Also ive got another report with same problem and a private rpc used just for 1 bridge node.
As for often, its not that often, i think i know of about 4-5 incidents so far between mainnet and testnet
related/dup #3264
Implementation ideas
Initially reported by the P-OPS team, this issue was noted when bridge nodes occasionally fell behind consensus nodes. The issue is consistently marked by the following log line:
Upon further investigation across other bridge nodes, similar occurrences were noted a few times each week. These issues are generally resolved automatically by the node's internal logic, which triggers a resubscription in response to such events. Notably, these events affect multiple nodes simultaneously, which suggests potential connectivity issues at the time of the incident or a problem with the core node RPC becoming stuck.