malise800
commented
6 months ago hey @walldiss so i will put here information that seems relevant to the issue. hope that will help.
so today we had the issue for around 20 min
issue was resolved with a manual restart
here are the logs captured just before the issue and after
2024-04-25T07:51:00.825Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "80A22C1AC58BFB6482A0ECFB8DD045C971A98FC0B13E5779459AFB93C1545A6E"}
2024-04-25T07:51:09.589Z INFO header/store store/store.go:375 new head {"height": 1685431, "hash": "F6B9A16D324B16E492DA9F2B3F8A2D0E32691EF41FB056D6F887D7CAE9A73C6E"}
2024-04-25T07:51:16.904Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "8CA74145AD1A865FB99727428D03EC0166D995725574A860053D56861DA77C37"}
2024-04-25T07:51:21.544Z INFO header/store store/store.go:375 new head {"height": 1685432, "hash": "D284D7162C514311161AFE9DD6CB6E55A7BE6BF901A88A36AB065DA2ABC23D7C"}
2024-04-25T07:51:22.336Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T07:51:27.981Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "C36C693ABD7E239572F93481C9D01111C4CBCB775B4E219AFCEB196476844A02"}
2024-04-25T07:51:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2432177 miss: 61534984 keys-added: 5820577 keys-updated: 13703 keys-evicted: 5820556 cost-added: 60101415840754 cost-evicted: 60101154390680 sets-dropped: 191 sets-rejected: 48316623 gets-dropped: 4414016 gets-kept: 59550016 gets-total: 63967161 hit-ratio: 0.04
2024-04-25T07:51:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 5820366
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 5814781 99.90% 99.90%
[2, 4) 4497 0.08% 99.98%
[4, 8) 1004 0.02% 100.00%
[8, 16) 83 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T07:51:35.936Z INFO canonical-log swarm/swarm_dial.go:559 CANONICAL_PEER_STATUS: peer=12D3KooWEjMK8hk451SvUixvSC3eA4Mj2jqxwhgTvuQGJ71wcV1r addr=/ip6/2607:5300:203:6730::/udp/2121/quic-v1 sample_rate=100 connection_status="established" dir="outbound"
2024-04-25T07:51:41.668Z ERROR shrex/eds shrexeds/server.go:120 server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "DD8A7EA64D5BC8DA35A9607B63886E7375F619BA1BE5B12682281822A71FE0B4", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to acquire reader of mount so we can return the accessor: open /home/pops/.celestia-bridge-mocha-4/blocks/DD8A7EA64D5BC8DA35A9607B63886E7375F619BA1BE5B12682281822A71FE0B4: too many open files"}
2024-04-25T07:51:42.114Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T07:51:55.177Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "B79369019A087FD68B234E81E24B7EA9CB5905572262B5571352ACD342C6EDEB"}
2024-04-25T07:52:09.108Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "158E8D9F71F7EE6E2923782118656C09F132BD9A8C709DAAD1456E241639DE74"}
2024-04-25T07:52:11.544Z INFO core core/listener.go:201 listener: listening stopped
2024-04-25T07:52:11.545Z WARN core core/listener.go:134 listener: subscriber error, resubscribing... {"err": "underlying subscription is stuck"}
2024-04-25T07:52:18.977Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "268B0C7C095F81FC23F0CFD47601628C60CD472F1540883E1DEF8712BD5EFE42"}
2024-04-25T07:52:23.875Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "D406C8E570566EBAE40558B66559529F08D08EC343412D547A9CBB5FB4183693"}
2024-04-25T07:52:28.815Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "8C6A1B85FC532538E7300E6F0371E1007633C38C7F8E6C0C588698B3DD9A4376"}
2024-04-25T07:52:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2446119 miss: 61823594 keys-added: 5846729 keys-updated: 13773 keys-evicted: 5846716 cost-added: 60383210996091 cost-evicted: 60382948333249 sets-dropped: 191 sets-rejected: 48579011 gets-dropped: 4414016 gets-kept: 59852672 gets-total: 64269713 hit-ratio: 0.04
2024-04-25T07:52:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 5846526
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 5840888 99.90% 99.90%
[2, 4) 4539 0.08% 99.98%
[4, 8) 1013 0.02% 100.00%
[8, 16) 85 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T07:52:49.450Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "25ECCF519AB6A8A6967F34B90933B6A7E101E014BA87BAC824D322FBD08BC7CA"}
2024-04-25T07:52:52.111Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T07:53:01.185Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "BC75CA8425AFD344C0E2AE42EA9DF96D9AED2ACE809EE4C8F7D9F010E3BED11E"}
2024-04-25T07:53:16.177Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "1D9B5337C3A6A2BD0CE7E484D0324567E62D50BB8A5F28DD1860131727184AAB"}
2024-04-25T07:53:17.593Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T07:53:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2456866 miss: 62052959 keys-added: 5868417 keys-updated: 13824 keys-evicted: 5868394 cost-added: 60622560334831 cost-evicted: 60622294449708 sets-dropped: 191 sets-rejected: 48786637 gets-dropped: 4414016 gets-kept: 60092864 gets-total: 64509825 hit-ratio: 0.04
2024-04-25T07:53:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 5868204
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 5862515 99.90% 99.90%
[2, 4) 4584 0.08% 99.98%
[4, 8) 1019 0.02% 100.00%
[8, 16) 85 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T07:53:42.662Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T07:53:44.113Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "C93A7F6AF89660C45430B2789E26422962A1913899275DFFEE433E4E24DA7F19"}
2024-04-25T07:53:58.154Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "257997AAA44BBC5A996FC94E106738910EAEFE5D32AB3CB3240121238039E342"}
2024-04-25T07:54:22.114Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T07:54:32.000Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "1DFDC94B427F670BF162D61F5C4A3CB3F3B634221986EDB6C09F8B6E98B3DEFC"}
2024-04-25T07:54:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2468315 miss: 62299488 keys-added: 5892560 keys-updated: 13873 keys-evicted: 5892545 cost-added: 60863255870186 cost-evicted: 60862998787177 sets-dropped: 191 sets-rejected: 49008974 gets-dropped: 4414080 gets-kept: 60350784 gets-total: 64767803 hit-ratio: 0.04
2024-04-25T07:54:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 5892355
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 5886608 99.90% 99.90%
[2, 4) 4626 0.08% 99.98%
[4, 8) 1033 0.02% 100.00%
[8, 16) 87 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T07:54:37.059Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T07:54:46.385Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "AB57DD8617CE7FC583A661AD48425933061428350CF52F93952F8CDA1E94E5F5"}
2024-04-25T07:54:54.872Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "4C8052FE35825456B4DFCF5D725A678B8D72617C81ABA15CDD753DD77789DDF9"}
2024-04-25T07:55:04.001Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "897194E2B7739FB971830B2A12BECA1B8439593DB51B14F29A774181F02ECD2B"}
2024-04-25T07:55:28.386Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "C76938FABE7F1E7CD35D60A25CCA7763692A3C70B26486C63D22DDE50E98CEA1"}
2024-04-25T07:55:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2476202 miss: 62476577 keys-added: 5910384 keys-updated: 13896 keys-evicted: 5910366 cost-added: 61065462695834 cost-evicted: 61065204139792 sets-dropped: 191 sets-rejected: 49168213 gets-dropped: 4414144 gets-kept: 60535616 gets-total: 64952780 hit-ratio: 0.04
2024-04-25T07:55:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 5910175
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 5904324 99.90% 99.90%
[2, 4) 4714 0.08% 99.98%
[4, 8) 1047 0.02% 100.00%
[8, 16) 89 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T07:55:36.367Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "B41752E35B81BDE6BEDD0105AEC2323B5A2D28A0CF3A618299641DBD9BED54FC"}
2024-04-25T07:55:41.221Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "2917D2836C625BBEC179A8B9732FF2D5026C89DCFCB4C1E9DCDD86361E3B33E2"}
2024-04-25T07:55:42.112Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T07:55:42.662Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T07:55:45.811Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "39F656BAE5D7CB42F43FDC4E5F188BDDE8A38C2AF3659FBAF773EA953F96E1C8"}
2024-04-25T07:55:52.050Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "81F884A07C5D404B676D221036FABB7E231EDC235D886B0F4BEC4C4C72D8A4EB"}
2024-04-25T07:55:58.376Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "8EB8023C472AD5E5C0C2D718C32C548CC18B7B47131A41D0CE13397B1F31EF08"}
2024-04-25T07:56:02.809Z ERROR shrex/eds shrexeds/server.go:120 server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "06E27242189AF9FAC12F75E6F412084CDE93F902D10BD5F576E9F3788FB38439", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to recover index for shard 06E27242189AF9FAC12F75E6F412084CDE93F902D10BD5F576E9F3788FB38439: open /home/pops/.celestia-bridge-mocha-4/index/06E27242189AF9FAC12F75E6F412084CDE93F902D10BD5F576E9F3788FB38439.full.idx: too many open files"}
2024-04-25T07:56:04.946Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "1359FEF60DF42A8E9BBEE34323C00D1EF42C2B15584AA442D2692B281269647E"}
2024-04-25T07:56:17.114Z ERROR shrex/eds shrexeds/server.go:120 server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "4576AC0D9BAC9B60071E8E8ED971067EDFF5A804DD640FC1F29EAD4B9971FC70", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to acquire reader of mount so we can return the accessor: open /home/pops/.celestia-bridge-mocha-4/blocks/4576AC0D9BAC9B60071E8E8ED971067EDFF5A804DD640FC1F29EAD4B9971FC70: too many open files"}
2024-04-25T07:56:19.601Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "9E54C93D31AE2CBF40852EDEE90B181FEB374DAFD6BEF27054A748CFD4AEE189"}
2024-04-25T07:56:32.614Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "39F86185E4246BEF6BF1F1568C77D9BBEE935D70F4E697541E063B6A8F2E584F"}
2024-04-25T07:56:35.058Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2486870 miss: 62702951 keys-added: 5934641 keys-updated: 13952 keys-evicted: 5934610 cost-added: 61315199697615 cost-evicted: 61314935808776 sets-dropped: 191 sets-rejected: 49370276 gets-dropped: 4414208 gets-kept: 60772672 gets-total: 65189821 hit-ratio: 0.04
2024-04-25T07:56:35.058Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 5934420
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 5928529 99.90% 99.90%
[2, 4) 4751 0.08% 99.98%
[4, 8) 1049 0.02% 100.00%
[8, 16) 90 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T07:56:43.369Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "144078B15D706E5D42CBEA2EA2A0BAC4CB9D1ABF54D692495AEAE6B3BE6CAFB9"}
2024-04-25T07:56:56.418Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "761E626C6DADC536989C4F0E8BE63ED84F447565323A08FB62C088B793F11174"}
2024-04-25T07:57:09.014Z ERROR shrex/eds shrexeds/server.go:120 server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "6FCA2825072472A6C125CD871D5AC10A727B245509D4A2381FE52E2F3784BB10", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to recover index for shard 6FCA2825072472A6C125CD871D5AC10A727B245509D4A2381FE52E2F3784BB10: open /home/pops/.celestia-bridge-mocha-4/index/6FCA2825072472A6C125CD871D5AC10A727B245509D4A2381FE52E2F3784BB10.full.idx: too many open files"}
2024-04-25T07:57:22.116Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T07:57:31.559Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "7AAF0551B755CAEF7BE63981CECF6E052508400D7DAB647F3F700C69AFEC48F2"}
2024-04-25T07:57:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2496805 miss: 62909162 keys-added: 5955219 keys-updated: 13983 keys-evicted: 5955198 cost-added: 61505584066530 cost-evicted: 61505320080300 sets-dropped: 191 sets-rejected: 49555878 gets-dropped: 4414272 gets-kept: 60988800 gets-total: 65405967 hit-ratio: 0.04
2024-04-25T07:57:35.058Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 5955008
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 5949006 99.90% 99.90%
[2, 4) 4836 0.08% 99.98%
[4, 8) 1072 0.02% 100.00%
[8, 16) 93 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T07:57:41.946Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "89C48C59D7F981818727D038A9475DA4CA26F6C4671CC4D23EFF4A98C83B518E"}
2024-04-25T07:57:42.662Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T07:58:12.181Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "8C15835E0E3406B5184B112241745B76B85166010E01CC983C1D476988F7AF46"}
2024-04-25T07:58:16.513Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "69E784EC92386B991240EA8C6FAB8978E2A4D771EB74134903E608643A8B0B78"}
2024-04-25T07:58:16.768Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T07:58:27.983Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "867F0F7CF371E8D4F262796751581F4F139A003E1FF413AD0A20ABC122A1E935"}
2024-04-25T07:58:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2508064 miss: 63143410 keys-added: 5980931 keys-updated: 14041 keys-evicted: 5980912 cost-added: 61776633321108 cost-evicted: 61776372161892 sets-dropped: 191 sets-rejected: 49764356 gets-dropped: 4414400 gets-kept: 61234176 gets-total: 65651474 hit-ratio: 0.04
2024-04-25T07:58:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 5980722
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 5974689 99.90% 99.90%
[2, 4) 4866 0.08% 99.98%
[4, 8) 1073 0.02% 100.00%
[8, 16) 93 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T07:58:40.547Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "C86830074C2E67E22D9F2A3C2C88B2F30064874F339A1D76E4036F09C6BE84D4"}
2024-04-25T07:58:42.114Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T07:58:48.817Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "152B4FF3FCC5B8184AE6AD8F7A36CA5D1A72BD13BEED68CAAE98DED0457E3DE9"}
2024-04-25T07:59:13.033Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "1FE8AB4F4E3CF679A096ACBEC0F95003E40318E3F86097B2B61ABE4DEA031A2F"}
2024-04-25T07:59:21.503Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "1E71F32AB19F7B0BC42852DC8FD621C007AC7A7A441D8FFE9A25F86CA8836A97"}
2024-04-25T07:59:27.322Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "B61E769A505647830555B0393C0F3997862A0627642001CFC04A5C1DEC140F21"}
2024-04-25T07:59:35.058Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2517378 miss: 63339663 keys-added: 6002317 keys-updated: 14085 keys-evicted: 6002304 cost-added: 61993113001270 cost-evicted: 61992845144346 sets-dropped: 191 sets-rejected: 49939179 gets-dropped: 4414464 gets-kept: 61439552 gets-total: 65857041 hit-ratio: 0.04
2024-04-25T07:59:35.058Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 6002114
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 5995993 99.90% 99.90%
[2, 4) 4934 0.08% 99.98%
[4, 8) 1091 0.02% 100.00%
[8, 16) 95 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T07:59:42.663Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T07:59:47.250Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "8669CFB0C8FED83F5811CB7B7E7A833CCFF43FE525598F0598B677AB51692C1F"}
2024-04-25T07:59:58.667Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "1A40C1C5F70C43B799396D7CA8B2F45E77152C5F045DD4730C1DAF73EB255D5C"}
2024-04-25T08:00:04.435Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "E2475D5C780A4D870B6F21F9A3FE9DC36D8AA3217F0955737301A510FB244C10"}
2024-04-25T08:00:09.504Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "E346C56C569B3543C9823051CC9F72D817770F0CB2965E36042788D8D8508B38"}
2024-04-25T08:00:15.436Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "7B80F9156F03F6C2921CEB6048BF8857AF4C333B310B977F074A615BFCC61DE5"}
2024-04-25T08:00:22.114Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:00:23.813Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "54D2BCA4072515039B19E60E2DE14F1768A656EDE819A144324E6B0C28758622"}
2024-04-25T08:00:25.780Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "499F39C762BC2CD9F72223E354470CA61DA942BA4CD84A05FE77F929E297841E"}
2024-04-25T08:00:25.881Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEhsc7GmWo9VcexP4SnRymojTaWqQUM7RxVF111bZxVp1", "hash": "499F39C762BC2CD9F72223E354470CA61DA942BA4CD84A05FE77F929E297841E"}
2024-04-25T08:00:26.106Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWNEDvy2uYwBrtic5BzfDioKHDdSoTC72jwzHewNoVv1B3", "hash": "499F39C762BC2CD9F72223E354470CA61DA942BA4CD84A05FE77F929E297841E"}
2024-04-25T08:00:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2529947 miss: 63602091 keys-added: 6029169 keys-updated: 14141 keys-evicted: 6029156 cost-added: 62265398471922 cost-evicted: 62265131402218 sets-dropped: 191 sets-rejected: 50174699 gets-dropped: 4414464 gets-kept: 61714624 gets-total: 66132038 hit-ratio: 0.04
2024-04-25T08:00:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 6028966
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 6022828 99.90% 99.90%
[2, 4) 4949 0.08% 99.98%
[4, 8) 1093 0.02% 100.00%
[8, 16) 95 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T08:00:38.421Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWNEDvy2uYwBrtic5BzfDioKHDdSoTC72jwzHewNoVv1B3", "hash": "4EE5E09736A4F410D671B817F3BFB19296798F2468838585EE1F944919A6A27A"}
2024-04-25T08:00:39.792Z ERROR shrex/eds shrexeds/server.go:120 server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "34EF2288D12F5970BA428390A0B0AE8F864F5E2C0C45F6C231BC1DEFE44FF864", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to acquire reader of mount so we can return the accessor: open /home/pops/.celestia-bridge-mocha-4/blocks/34EF2288D12F5970BA428390A0B0AE8F864F5E2C0C45F6C231BC1DEFE44FF864: too many open files"}
2024-04-25T08:00:49.823Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "6078DA78BC0ED4B04B73DA0AC8DB546889C30E52446FDDC6FA6B99F5A3FCFB4B"}
2024-04-25T08:00:49.966Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEhsc7GmWo9VcexP4SnRymojTaWqQUM7RxVF111bZxVp1", "hash": "195F34CF7E89AD076F1B7B42B862A2523D305A36E1B98737F8A68DA8011F9EF3"}
2024-04-25T08:00:53.703Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "6F2F6C8F9F8BA70DF4D10014DEACEA1598F011013918320DBB48B40A4787B8DF"}
2024-04-25T08:00:57.296Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "5A57563C32F522FD389F1A5EDA13BC1EFB16E3A963BA013F85AEAD7303F1EF6C"}
2024-04-25T08:01:03.737Z ERROR shrex/eds shrexeds/server.go:120 server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "C4DF337245B41821F21CF8EA8CEF923F7A790C0987B50E156B9F88958F06E791", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to acquire reader of mount so we can return the accessor: open /home/pops/.celestia-bridge-mocha-4/blocks/C4DF337245B41821F21CF8EA8CEF923F7A790C0987B50E156B9F88958F06E791: too many open files"}
2024-04-25T08:01:06.758Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "80EC66847272995B42EC1B3226DE0922C15137002758715A45EE377330B9C97E"}
2024-04-25T08:01:10.575Z ERROR shrex/eds shrexeds/server.go:120 server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "D38ECBD79DECD642F5D109571773DC16F77B35E19B9D6BBF7497756033C4F664", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to acquire reader of mount so we can return the accessor: open /home/pops/.celestia-bridge-mocha-4/blocks/D38ECBD79DECD642F5D109571773DC16F77B35E19B9D6BBF7497756033C4F664: too many open files"}
2024-04-25T08:01:13.099Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "6F245B89A9BB52FB0B19E899C9DC7D26DB40AE8C3F23AE3437586C695E6759F6"}
2024-04-25T08:01:17.353Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "26BEAFA950AC9CAC56A844491AE269F7AD2A8450B8847F2D875B7BFB02846C35"}
2024-04-25T08:01:24.559Z ERROR shrex/eds shrexeds/server.go:120 server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "28F7B44497104B64BBFC9C17CD6D1FC2862215867D37BDC3DC8E172585CB44AC", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to acquire reader of mount so we can return the accessor: open /home/pops/.celestia-bridge-mocha-4/blocks/28F7B44497104B64BBFC9C17CD6D1FC2862215867D37BDC3DC8E172585CB44AC: too many open files"}
2024-04-25T08:01:25.152Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEhsc7GmWo9VcexP4SnRymojTaWqQUM7RxVF111bZxVp1", "hash": "01864E5FA7FB0DA4A0AEDA1EC256EE8EFB41CE91C6C0F39180CA5220EA420F54"}
2024-04-25T08:01:30.081Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "2803D7651F963BFCDECF1DB043AEAC4F56194726DD316F5E448AA221652491B6"}
2024-04-25T08:01:32.115Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:01:35.058Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2540507 miss: 63817284 keys-added: 6050291 keys-updated: 14178 keys-evicted: 6050268 cost-added: 62484802899070 cost-evicted: 62484545874197 sets-dropped: 191 sets-rejected: 50368733 gets-dropped: 4414464 gets-kept: 61940480 gets-total: 66357791 hit-ratio: 0.04
2024-04-25T08:01:35.058Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 6050078
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 6043853 99.90% 99.90%
[2, 4) 5018 0.08% 99.98%
[4, 8) 1109 0.02% 100.00%
[8, 16) 97 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T08:01:42.235Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "F403F032A1EA7D44DD223806F18F1AB74202971B9F5A72D956635AA0570A487D"}
2024-04-25T08:01:42.662Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:01:45.058Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:01:45.398Z INFO canonical-log swarm/swarm_listen.go:136 CANONICAL_PEER_STATUS: peer=12D3KooWJ5dfgoAXpG2hsvXhtxTCp5BsxEWkzGWhbmns8zzf9Z2a addr=/ip4/54.191.40.227/udp/2121/quic-v1 sample_rate=100 connection_status="established" dir="inbound"
2024-04-25T08:01:52.571Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:02:02.691Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "2356198DF2B17027119A58F5098D5300AA47A6E902CB75926A80D07885BD9189"}
2024-04-25T08:02:05.260Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:02:08.822Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "7926C35BAACFCB20E6340AEFE16B9B699775A575AE76B2D206217C8A8C8A3068"}
2024-04-25T08:02:17.592Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:02:22.306Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "8B72747F91C2E47ED205CCEBFB49D14AA917F6F4B6272B6662104771D9D8E0AC"}
2024-04-25T08:02:28.325Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "1D17FFADCE03F7DB145ABE5CD4574BF7AB835279883B8FCA228CCF0BDEFDB663"}
2024-04-25T08:02:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2549995 miss: 64026217 keys-added: 6071197 keys-updated: 14219 keys-evicted: 6071175 cost-added: 62716680272679 cost-evicted: 62716440085157 sets-dropped: 191 sets-rejected: 50556719 gets-dropped: 4414464 gets-kept: 62158848 gets-total: 66576212 hit-ratio: 0.04
2024-04-25T08:02:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 6070985
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 6064710 99.90% 99.90%
[2, 4) 5064 0.08% 99.98%
[4, 8) 1113 0.02% 100.00%
[8, 16) 97 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T08:02:42.113Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:02:52.663Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:03:02.498Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "7FF1B0946DC8038981C5AC2E0089166D2F65CED4841E5AE753F55E6833C3AF79"}
2024-04-25T08:03:07.804Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "8025EC3F6A82D7BE5CADD1BFAADC713CA187B0712C9E1D29E7690700BDC0D8D9"}
2024-04-25T08:03:26.378Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "25375639C919C84FE9FF794F0763AFA1F94BA60BA6494C4EE5DE70EFA9644632"}
2024-04-25T08:03:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2557267 miss: 64187707 keys-added: 6086830 keys-updated: 14251 keys-evicted: 6086809 cost-added: 62890035729757 cost-evicted: 62889788131265 sets-dropped: 191 sets-rejected: 50702544 gets-dropped: 4414464 gets-kept: 62327616 gets-total: 66744974 hit-ratio: 0.04
2024-04-25T08:03:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 6086619
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 6080248 99.90% 99.90%
[2, 4) 5155 0.08% 99.98%
[4, 8) 1118 0.02% 100.00%
[8, 16) 97 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T08:04:10.705Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "CDC851B01241D59C13C1741105196A3F3C5D7A46941EC5F00B8BEC534ACE142A"}
2024-04-25T08:04:18.656Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "E3FBD06EB3D7A0AF5FE9166F8AAA8EAE7ED556105C0F84C18B4400DEA2534FCC"}
2024-04-25T08:04:18.819Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "78B586631F989A43DFD17808D2D2053256F86220D7057D7E98C5A274CDB73FEA"}
2024-04-25T08:04:22.114Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:04:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2565874 miss: 64377447 keys-added: 6104628 keys-updated: 14283 keys-evicted: 6104608 cost-added: 63074480913736 cost-evicted: 63074230159181 sets-dropped: 191 sets-rejected: 50874454 gets-dropped: 4414464 gets-kept: 62525888 gets-total: 66943321 hit-ratio: 0.04
2024-04-25T08:04:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 6104418
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 6097941 99.89% 99.89%
[2, 4) 5229 0.09% 99.98%
[4, 8) 1141 0.02% 100.00%
[8, 16) 106 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T08:04:42.662Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:04:52.383Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:04:54.566Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "B730DFD143FCCAAA729476FF88D5A185D0A1431042FD76F7C8CFC65DCB056345"}
2024-04-25T08:04:57.400Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:05:32.275Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "AFC302CAF4BE1F510F86F007C871082000E2F475BD5A5A82020307A859D8BD2C"}
2024-04-25T08:05:34.107Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:05:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2573486 miss: 64555066 keys-added: 6121690 keys-updated: 14304 keys-evicted: 6121664 cost-added: 63257915484247 cost-evicted: 63257658419777 sets-dropped: 191 sets-rejected: 51034987 gets-dropped: 4414464 gets-kept: 62711104 gets-total: 67128552 hit-ratio: 0.04
2024-04-25T08:05:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 6121473
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 6114915 99.89% 99.89%
[2, 4) 5299 0.09% 99.98%
[4, 8) 1151 0.02% 100.00%
[8, 16) 107 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T08:05:37.094Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "198675CD416AC9E8683078F684F31A9B7B03B8880449737A8F37D0340F0DE721"}
2024-04-25T08:05:41.729Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "208DE8C41F9BCAEEE03CFCB4DE52C08481F7F5F5DE213FC37EC226FEA294E28A"}
2024-04-25T08:05:42.113Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:05:51.581Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "E72FA5C69D5606399A45779A74E94F5CED7C760970D49BAD41EBE6D1955E4BA9"}
2024-04-25T08:05:56.242Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "1CEF49FC429D15905A56536D7A61E4F5E7BA0CE8064420A799BAD41496725393"}
2024-04-25T08:06:03.848Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "18EDB3A05FD7D380DE9B89B062DA96774AB3F79BF7450849DED0A639752DE049"}
2024-04-25T08:06:17.594Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:06:24.726Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "23CA4779BB5D99934E780BD1D392EBDB9DA2D2C193CCD9B52B7C85052E291C42"}
2024-04-25T08:06:29.646Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "E4DB13552DD58FFEE8D79A2B143CAE146F5AE24300F5C42AA1E4BEDD0A582EB3"}
2024-04-25T08:06:34.670Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "E0ECDA66217420CAACA75832CF88DF398BB103C06B48183876D8665A2E6AF0CC"}
2024-04-25T08:06:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2581764 miss: 64734414 keys-added: 6139809 keys-updated: 14340 keys-evicted: 6139778 cost-added: 63461873008696 cost-evicted: 63461611787744 sets-dropped: 191 sets-rejected: 51196183 gets-dropped: 4414592 gets-kept: 62898688 gets-total: 67316178 hit-ratio: 0.04
2024-04-25T08:06:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 6139588
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 6132951 99.89% 99.89%
[2, 4) 5359 0.09% 99.98%
[4, 8) 1167 0.02% 100.00%
[8, 16) 110 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T08:06:40.052Z ERROR shrex/eds shrexeds/server.go:120 server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "ECE59C544DB0DBF97E07476A9DCE748E071811D17D22063677FF8DD3A4480DA5", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to acquire reader of mount so we can return the accessor: open /home/pops/.celestia-bridge-mocha-4/blocks/ECE59C544DB0DBF97E07476A9DCE748E071811D17D22063677FF8DD3A4480DA5: no such file or directory"}
2024-04-25T08:06:42.662Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:06:42.676Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "CC1A4A39102972A262B9838DE5ED522617C428F74B384221F7F0D14D2DBCE24A"}
2024-04-25T08:06:47.709Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "F2408B3EA85FB4180E61BD39F7A9B6AE1B54AAB5E932B383304E823B4349C321"}
2024-04-25T08:06:52.111Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:06:52.387Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "18CED3824302CF79512BA865F88D085243F8339325727ED5CB0CCB27F24A1683"}
2024-04-25T08:06:55.634Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "C45FDA45BD6D1E3224085B0818B21E664CAF83CA8D8F880228B8652E51A675C9"}
2024-04-25T08:06:58.821Z ERROR shrex/eds shrexeds/server.go:120 server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "ABDAD379E096FF647535C92E223A87BE52523B1DE7BB9A5F5A15E84BDF20F0A0", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to acquire reader of mount so we can return the accessor: open /home/pops/.celestia-bridge-mocha-4/blocks/ABDAD379E096FF647535C92E223A87BE52523B1DE7BB9A5F5A15E84BDF20F0A0: too many open files"}
2024-04-25T08:07:01.315Z ERROR shrex/eds shrexeds/server.go:120 server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "EBFA469A59788F4DEB110DABE40E5261E6D968AE198C5A494C9270B6D78EC901", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to acquire reader of mount so we can return the accessor: open /home/pops/.celestia-bridge-mocha-4/blocks/EBFA469A59788F4DEB110DABE40E5261E6D968AE198C5A494C9270B6D78EC901: too many open files"}
2024-04-25T08:07:03.688Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "E9F2D6E4B41AED0F80B5C9BFE45A40D24AB7AAA2276F5CD1EB98E07FFEFC2F25"}
2024-04-25T08:07:09.514Z ERROR shrex/eds shrexeds/server.go:120 server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "3CD5329E3B82673F7393039FA4D0346CAF6B27667088DE453A541B49956A1597", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to recover index for shard 3CD5329E3B82673F7393039FA4D0346CAF6B27667088DE453A541B49956A1597: open /home/pops/.celestia-bridge-mocha-4/index/3CD5329E3B82673F7393039FA4D0346CAF6B27667088DE453A541B49956A1597.full.idx: too many open files"}
2024-04-25T08:07:12.829Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "E006FF1CF3DC10F9D60503A21E5942A59B7C4E1FD5844DCDCFB5E410A6EBAD82"}
2024-04-25T08:07:16.293Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "824573E032C9C68DFA0F7C27BDDD786368D41622C890F35CE40C56966AD478CD"}
2024-04-25T08:07:19.644Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "87A693BFC32DEB6E74398D673274E50F3B086D8AE1B5D9E1EAB2C45897C3AA27"}
2024-04-25T08:07:25.998Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "27CC927CEC0D24E51AD6B51F125D4660C73FB6F690D142C4D4E352E07230D6D0"}
2024-04-25T08:07:35.058Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2590472 miss: 64927948 keys-added: 6157559 keys-updated: 14374 keys-evicted: 6157535 cost-added: 63642968724664 cost-evicted: 63642704130515 sets-dropped: 191 sets-rejected: 51371933 gets-dropped: 4414848 gets-kept: 63100544 gets-total: 67518420 hit-ratio: 0.04
2024-04-25T08:07:35.058Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 6157345
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 6150587 99.89% 99.89%
[2, 4) 5454 0.09% 99.98%
[4, 8) 1190 0.02% 100.00%
[8, 16) 113 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T08:07:43.771Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "7C3A1ABDF12367045C17B0677574B4D20F3BAE593E734A6B65295B028D54F11C"}
2024-04-25T08:08:14.021Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "272979BAD3159480207DAC715156ED860FFF9132524BEA96EB6A29E90C26960A"}
2024-04-25T08:08:22.113Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:08:24.134Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "4B8B7BBC06C1DB078AB391EF19C68078F2B910E5292F21D01647958612D2F70B"}
2024-04-25T08:08:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2600072 miss: 65141854 keys-added: 6180385 keys-updated: 14410 keys-evicted: 6180362 cost-added: 63902059741804 cost-evicted: 63901800223991 sets-dropped: 191 sets-rejected: 51562947 gets-dropped: 4414912 gets-kept: 63324160 gets-total: 67741927 hit-ratio: 0.04
2024-04-25T08:08:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 6180172
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 6173381 99.89% 99.89%
[2, 4) 5486 0.09% 99.98%
[4, 8) 1191 0.02% 100.00%
[8, 16) 113 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T08:08:36.374Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWNEDvy2uYwBrtic5BzfDioKHDdSoTC72jwzHewNoVv1B3", "hash": "A630D138C083271F0878CEC1D3A00836CF70C932C8059DFCF1C88499089D120E"}
2024-04-25T08:08:42.589Z INFO rpc rpc/server.go:108 server stopped
2024-04-25T08:08:42.589Z INFO header/p2p p2p/server.go:86 server: stopping server
2024-04-25T08:08:42.662Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:08:48.708Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "79FAF1AE0EB44254CD311375C562E15642EC5D3862CAB52AB32BCB969A216A9B"}
2024-04-25T08:09:17.591Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:09:25.651Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "7DE740A56174AF919DC9BAB979A58DD6C056F1281AB94814A054495EC3AB9F18"}
2024-04-25T08:09:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 2601525 miss: 65172940 keys-added: 6183944 keys-updated: 14415 keys-evicted: 6183927 cost-added: 63943646189944 cost-evicted: 63943381978736 sets-dropped: 191 sets-rejected: 51590499 gets-dropped: 4414912 gets-kept: 63356544 gets-total: 67774465 hit-ratio: 0.04
2024-04-25T08:09:35.057Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 20
Count: 6183737
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 6176945 99.89% 99.89%
[2, 4) 5487 0.09% 99.98%
[4, 8) 1191 0.02% 100.00%
[8, 16) 113 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
--
2024-04-25T08:09:37.401Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "83FEECF5B531B3716DDA0219D2712566A22D5C6D4B5D9A8AF2AD1F9507756E40"}
2024-04-25T08:09:42.113Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:09:44.300Z INFO core core/listener.go:203 listener: listening stopped
2024-04-25T08:09:44.302Z WARN shrex-sub shrexsub/pubsub.go:83 unregistering topic {"err": "no validator for topic mocha-4/eds-sub/v0.1.0"}
2024-04-25T08:09:44.303Z INFO badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:546 Lifetime L0 stalled for: 0s
2024-04-25T08:09:44.367Z INFO badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:625
Level 0 [ ]: NumTables: 01. Size: 13 MiB of 0 B. Score: 1.00->1.06 StaleData: 0 B Target FileSize: 64 MiB
Level 1 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 2 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 3 [B]: NumTables: 07. Size: 24 MiB of 25 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 4.0 MiB
Level 4 [ ]: NumTables: 46. Size: 253 MiB of 254 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 8.0 MiB
Level 5 [ ]: NumTables: 282. Size: 2.5 GiB of 2.5 GiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 16 MiB
Level 6 [ ]: NumTables: 1269. Size: 25 GiB of 25 GiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 32 MiB
Level 7 [ ]: NumTables: 5539. Size: 248 GiB of 248 GiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 64 MiB
Level Done
2024-04-25T08:09:45.910Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.910Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.910Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.910Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.910Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.910Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.910Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (local)
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (local)
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (local)
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (local)
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.912Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.912Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.912Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.912Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.912Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.912Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.912Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.914Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.914Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.914Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.914Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.914Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.914Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.914Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.914Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.914Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.914Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.914Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.916Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.916Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.916Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.916Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.916Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.916Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.916Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.916Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.916Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.917Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.917Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.917Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.917Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.917Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.917Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.917Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.917Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.917Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.919Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.919Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.919Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.919Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.919Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.919Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.919Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.919Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.919Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.920Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.920Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.920Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.920Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.920Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.920Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.920Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.920Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.920Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.921Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.921Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.921Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.921Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.921Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.921Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.921Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.921Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.922Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.922Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.923Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.923Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.926Z INFO badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:546 Lifetime L0 stalled for: 0s
2024-04-25T08:09:46.085Z INFO badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:625
Level 0 [ ]: NumTables: 00. Size: 0 B of 0 B. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 16 MiB
Level 1 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 2 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 3 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 4 [B]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 5 [ ]: NumTables: 16. Size: 55 MiB of 45 MiB. Score: 1.22->1.22 StaleData: 4.4 KiB Target FileSize: 4.0 MiB
Level 6 [ ]: NumTables: 90. Size: 454 MiB of 454 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 8.0 MiB
Level Done
2024-04-25T08:09:46.355Z INFO node nodebuilder/module.go:26 Accessing keyring...
2024-04-25T08:09:46.382Z INFO module/state state/keyring.go:40 constructed keyring signer {"backend": "test", "path": "/home/pops/.celestia-bridge-mocha-4/keys", "key name": "my_celes_key", "chain-id": "mocha-4"}
2024-04-25T08:09:46.421Z INFO badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/levels.go:171 All 106 tables opened in 24ms
2024-04-25T08:09:46.422Z INFO badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/discard.go:66 Discard stats nextEmptySlot: 48
2024-04-25T08:09:46.422Z INFO badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:368 Set nextTxnTs to 1799707427
2024-04-25T08:09:49.631Z INFO badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/levels.go:118 6619 tables out of 7144 opened in 3s
2024-04-25T08:09:49.866Z INFO badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/levels.go:171 All 7144 tables opened in 3.235s
2024-04-25T08:09:49.871Z INFO badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/discard.go:66 Discard stats nextEmptySlot: 0
2024-04-25T08:09:49.874Z INFO badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:368 Set nextTxnTs to 1071359
2024-04-25T08:09:49.878Z INFO badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/value.go:592 Deleting empty file: /home/pops/.celestia-bridge-mocha-4/inverted_index//000371.vlog
2024-04-25T08:09:56.507Z INFO basichost basic/natmgr.go:112 DiscoverNAT error:no NAT found
2024-04-25T08:09:59.874Z INFO share/discovery discovery/discovery.go:312 discovered wanted peers {"amount": 5}
2024-04-25T08:10:00.940Z INFO header/p2p p2p/subscriber.go:81 joining topic {"topic ID": "/mocha-4/header-sub/v0.0.1"}
2024-04-25T08:10:01.005Z INFO header/store store/store.go:193 loaded head {"height": 1685432, "hash": "D284D7162C514311161AFE9DD6CB6E55A7BE6BF901A88A36AB065DA2ABC23D7C"}
2024-04-25T08:10:01.005Z INFO fraudserv fraudserv/helpers.go:29 joining topic {"id": "/mocha-4/fraud-sub/badencodingv0.1/v0.0.1"}
2024-04-25T08:10:01.005Z INFO fraudserv fraudserv/service.go:114 starting fraud proof service {"protocol ID": "/mocha-4/fraud/v0.0.1"}
2024-04-25T08:10:01.446Z INFO header/sync sync/sync_head.go:140 new network head {"height": 1685527, "hash": "6EBE549FEBDD37258935D9106906E14A898ED695A48F463F404C439E8AC20369"}
2024-04-25T08:10:01.446Z INFO header/p2p p2p/server.go:77 server: listening for inbound header requests {"protocol ID": "/mocha-4/header-ex/v0.0.3"}
2024-04-25T08:10:01.447Z INFO header/sync sync/sync.go:216 syncing headers {"from": 1685433, "to": 1685527}
2024-04-25T08:10:01.447Z INFO rpc rpc/server.go:90 server started {"listening on": "0.0.0.0:26658"}
2024-04-25T08:10:01.447Z INFO node nodebuilder/node.go:113
/_____/ /_____/ /_____/ /_____/ /_____/
Started celestia DA node
node version: v0.13.4
node type: bridge
network: mocha-4
/_____/ /_____/ /_____/ /_____/ /_____/
The p2p host is listening on:
* /ip4/127.0.0.1/udp/2121/quic-v1/webtransport/certhash/uEiAktN9CODcWdqRyeVuGwDnt1Le97oOPipKvqbs0s-uTgQ/certhash/uEiC9AFyLT6belLezugelvCHt2GAQKg8fEs_fP5fVTvbzzg/p2p/12D3KooWA92eW13sbg57xdXhrjfJTJf8t3frDy1bdvfCJLSmcD9F
* /ip4/162.55.237.230/tcp/2121/p2p/12D3KooWA92eW13sbg57xdXhrjfJTJf8t3frDy1bdvfCJLSmcD9F
* /ip4/162.55.237.230/udp/2121/quic-v1/p2p/12D3KooWA92eW13sbg57xdXhrjfJTJf8t3frDy1bdvfCJLSmcD9F
* /ip4/162.55.237.230/udp/2121/quic-v1/webtransport/certhash/uEiAktN9CODcWdqRyeVuGwDnt1Le97oOPipKvqbs0s-uTgQ/certhash/uEiC9AFyLT6belLezugelvCHt2GAQKg8fEs_fP5fVTvbzzg/p2p/12D3KooWA92eW13sbg57xdXhrjfJTJf8t3frDy1bdvfCJLSmcD9F
* /ip6/::1/tcp/2121/p2p/12D3KooWA92eW13sbg57xdXhrjfJTJf8t3frDy1bdvfCJLSmcD9F
* /ip6/::1/udp/2121/quic-v1/p2p/12D3KooWA92eW13sbg57xdXhrjfJTJf8t3frDy1bdvfCJLSmcD9F
* /ip6/::1/udp/2121/quic-v1/webtransport/certhash/uEiAktN9CODcWdqRyeVuGwDnt1Le97oOPipKvqbs0s-uTgQ/certhash/uEiC9AFyLT6belLezugelvCHt2GAQKg8fEs_fP5fVTvbzzg/p2p/12D3KooWA92eW13sbg57xdXhrjfJTJf8t3frDy1bdvfCJLSmcD9F
* /ip6/2a01:4f8:271:3f6e::2/tcp/2121/p2p/12D3KooWA92eW13sbg57xdXhrjfJTJf8t3frDy1bdvfCJLSmcD9F
* /ip6/2a01:4f8:271:3f6e::2/udp/2121/quic-v1/p2p/12D3KooWA92eW13sbg57xdXhrjfJTJf8t3frDy1bdvfCJLSmcD9F
* /ip6/2a01:4f8:271:3f6e::2/udp/2121/quic-v1/webtransport/certhash/uEiAktN9CODcWdqRyeVuGwDnt1Le97oOPipKvqbs0s-uTgQ/certhash/uEiC9AFyLT6belLezugelvCHt2GAQKg8fEs_fP5fVTvbzzg/p2p/12D3KooWA92eW13sbg57xdXhrjfJTJf8t3frDy1bdvfCJLSmcD9F
2024-04-25T08:10:02.192Z INFO canonical-log tls/transport.go:96 CANONICAL_PEER_STATUS: peer= addr=/ip4/34.145.72.169/tcp/19491 sample_rate=100 handshake_failure="tls" err="read tcp4 162.55.237.230:2121->34.145.72.169:19491: read: connection reset by peer"
2024-04-25T08:10:05.050Z INFO header/store store/store.go:375 new head {"height": 1685526, "hash": "AD7F3EA202A2E973DD44C48C0E33FB9ECCAB46AFEFADE6C371C831F3F7425500"}
2024-04-25T08:10:05.050Z INFO header/store store/store.go:375 new head {"height": 1685527, "hash": "6EBE549FEBDD37258935D9106906E14A898ED695A48F463F404C439E8AC20369"}
2024-04-25T08:10:05.051Z INFO header/sync sync/sync.go:234 finished syncing headers {"from": 1685433, "to": 1685527, "elapsed time": 3.604018927}
2024-04-25T08:10:06.555Z INFO canonical-log swarm/swarm_listen.go:136 CANONICAL_PEER_STATUS: peer=12D3KooWKHePZ7CdFE9xgF7tHQxuPsj25DPFgWRuW3S7DK5gTtKk addr=/ip4/34.83.65.52/tcp/50168 sample_rate=100 connection_status="established" dir="inbound"
2024-04-25T08:10:09.447Z INFO header/store store/store.go:375 new head {"height": 1685528, "hash": "5B4E863EA638B9C750999C683F339B7A155AF66829140B589CA5E1A011560F22"}
2024-04-25T08:10:21.194Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "55020748E2B2C67A2C50F0DCD8AA72C8DA60541A422D63F1DA1439FC190C52BA"}
2024-04-25T08:10:21.475Z INFO header/store store/store.go:375 new head {"height": 1685529, "hash": "F1CBC8BAF54EF2CF2918D93B1ABCF6AACFF0F020E3DC381228A675E850AC750C"}
2024-04-25T08:10:23.317Z INFO canonical-log swarm/swarm_listen.go:136 CANONICAL_PEER_STATUS: peer=12D3KooWSi9rYk3JF9iH9ZrvUJhu5YwiKynmjsomnrtUVomHL3kB addr=/ip4/34.145.72.169/tcp/18202 sample_rate=100 connection_status="established" dir="inbound"
2024-04-25T08:10:32.888Z WARN shrex/eds shrexeds/server.go:116 server: request hash not found {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "CA9F8B031E4490592ADAB06A484DF8462BC7A153B11D9439525C7E91ED233FE3"}
2024-04-25T08:10:33.073Z INFO header/store store/store.go:375 new head {"height": 1685530, "hash": "BD977AFDF2E8E3243193ACA1360EE6B1094E4661E2A6B81DCCCEFA4F22943D09"}
2024-04-25T08:10:44.959Z INFO header/store store/store.go:375 new head {"height": 1685531, "hash": "99E9A7601796525AAB4FDCD061FA23EBFC8D799EFE101383F8595E93A9D4BD91"}
2024-04-25T08:10:46.572Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460 Block cache might be too small. Metrics: hit: 6170 miss: 927875 keys-added: 12014 keys-updated: 30 keys-evicted: 11999 cost-added: 117314086439 cost-evicted: 117061167309 sets-dropped: 0 sets-rejected: 123797 gets-dropped: 306624 gets-kept: 626496 gets-total: 934045 hit-ratio: 0.01
2024-04-25T08:10:46.572Z WARN badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461 Cache life expectancy (in seconds):
-- Histogram:
Min value: 0
Max value: 24
Count: 11994
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 11913 99.32% 99.32%
[2, 4) 33 0.28% 99.60%
[4, 8) 24 0.20% 99.80%
[8, 16) 11 0.09% 99.89%
[16, 32) 13 0.11% 100.00%
--
2024-04-25T08:10:51.185Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: timeout: no recent network activity
2024-04-25T08:10:52.112Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:10:56.505Z INFO header/store store/store.go:375 new head {"height": 1685532, "hash": "49A3AF2CAD3DB293A4A7A9949C674E2C0ADFB760628B49E25BC5FCC64E83D70A"}
2024-04-25T08:10:57.590Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:11:02.663Z INFO bitswap-server server/server.go:547 Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:11:08.184Z INFO header/store store/store.go:375 new head {"height": 1685533, "hash": "0682F70785C46981C78C6699B00766B134C6D45351175141D619D6DB30A3C6FD"}
2024-04-25T08:11:19.903Z INFO header/store store/store.go:375 new head {"height": 1685534, "hash": "FA98F3F2AB4EF47D2090A86F7130EFC5E124E7B7CD84BD032BA73881300E6F9C"}
walldiss
mindstyle85
renaynay
Implementation ideas
Initially reported by the P-OPS team, this issue was noted when bridge nodes occasionally fell behind consensus nodes. The issue is consistently marked by the following log line:
Upon further investigation across other bridge nodes, similar occurrences were noted a few times each week. These issues are generally resolved automatically by the node's internal logic, which triggers a resubscription in response to such events. Notably, these events affect multiple nodes simultaneously, which suggests potential connectivity issues at the time of the incident or a problem with the core node RPC becoming stuck.