search

waku-org / nwaku

Waku node and protocol.
Other
201 stars 53 forks source link

maintenance: failing tests #1357

Closed s1fr0 closed 1 year ago

s1fr0 commented 1 year ago

Problem

In the development of nwaku, we occasionally observe some CI/Jenkins test failures that are usually not triggered when tests are re-executed.

This issue aims to collect observed test failures in order to:

Everyone is encouraged to edit this issue and add new tests that fail during checks. Once the causes of a test failure are identified and addressed, please remove the failing test from this post and comment to this issue by reporting it description, the list of identified causes and link the PR that fixed it.

Any feedback is welcome!

Test report template

Action type: CI/Jenkins OS: linux/macOS Test name/version: (v1/v2) nwaku commit/tree: Log excerpt: Causes: (not identified/speculated/identified) TimesSeen: (times the test was seen) Temporary fix/workaround: (if any)

Failing tests

Action type: CI OS: macOS Test name/version: "Messages are bridged between Waku v1 and Waku v2" (v2) nwaku commit/tree: https://github.com/status-im/nwaku/tree/7bc657b545c1b6eb54d5b70f8a926fb208aad939 Log excerpt: TimeSeen: 4

2022-11-08T12:27:30.0160480Z     /Users/runner/work/nwaku/nwaku/tests/v2/test_waku_bridge.nim(152, 49): Check failed: v1Node.protocolState(Waku).queue.items.len == 1
2022-11-08T12:27:30.0161010Z     v1Node.protocolState(Waku).queue.items.len was 0
2022-11-08T12:27:30.0161840Z /Users/runner/work/nwaku/nwaku/vendor/nimbus-build-system/vendor/Nim/lib/system/fatal.nim(53) sysFatal
2022-11-08T12:27:30.0162530Z /Users/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncfutures2.nim(391) futureContinue
2022-11-08T12:27:30.0162820Z 
2022-11-08T12:27:30.0163040Z     Unhandled exception: index out of bounds, the container is empty [IndexDefect]
2022-11-08T12:27:30.0163440Z   [FAILED] Messages are bridged between Waku v1 and Waku v2

Causes: not identified

Action type: CI OS: macOS Test name/version: "event subscription" (v2) nwaku commit/tree: https://github.com/status-im/nwaku/tree/63137f3e2a9457584276823918d55065f7d8e3bf Log excerpt: TimeSeen: 1

DBG 2022-11-09 20:38:28.017+00:00 Address of the deployed membership contract:  tid=179666 file=test_waku_rln_relay_onchain.nim:73 contractAddress=0xd12eac7850902252129eb346dc3d5e0d279e02a4
DBG 2022-11-09 20:38:28.019+00:00 Account balance after the contract deployment:  tid=179666 file=test_waku_rln_relay_onchain.nim:76 balance=9999997207396538075888
DBG 2022-11-09 20:38:28.019+00:00 disconnected from                          tid=179666 file=test_waku_rln_relay_onchain.nim:79 ethClientAddress=ws://127.0.0.1:8540
DBG 2022-11-09 20:38:28.020+00:00 web3 connected to                          tid=179666 file=test_waku_rln_relay_onchain.nim:184 EthClient=ws://127.0.0.1:8540
DBG 2022-11-09 20:38:28.027+00:00 contract deployer account address          tid=179666 file=test_waku_rln_relay_onchain.nim:189 defaultAccount=0xcbe069e7c30e77b3e17f1929236bcd87e3b89f3b
DBG 2022-11-09 20:38:28.772+00:00 membership commitment key                  tid=179666 file=test_waku_rln_relay_onchain.nim:206 pk=14109699623625531147653880372921537588611513249833251812463573252554215275504
Error from subscription: key not found: data
DBG 2022-11-09 20:38:28.804+00:00 a member is registered                     tid=179666 file=test_waku_rln_relay_onchain.nim:226 tx=0x3fcdc383108516230c6867d0708f1f5a947427475e4947a1c183b287fc3a6298
2022-11-09T20:57:19.5714790Z Error: The operation was canceled.

Causes: not identified

Action type: CI OS: macOS Test name/version: "peer subscription should not be dropped if connection recovers before timeout elapses" (v2) nwaku commit/tree: https://github.com/status-im/nwaku/tree/71c5bfda0e4bfbedd967ed23e2cb53899de10c11 Log excerpt: TimeSeen: 4

2022-11-10T16:44:18.4130470Z     /Users/runner/work/nwaku/nwaku/tests/v2/test_waku_filter.nim(236, 18): Check failed: await pushHandlerFuture.withTimeout(1.seconds)
2022-11-10T16:44:18.4146180Z stack trace: (most recent call last)
2022-11-10T16:44:18.4146930Z /Users/runner/work/nwaku/nwaku/vendor/nimbus-build-system/vendor/Nim/lib/system/nimscript.nim(429, 18)
2022-11-10T16:44:18.4148050Z /Users/runner/work/nwaku/nwaku/waku.nimble(76, 8) test2Task
2022-11-10T16:44:18.4148690Z /Users/runner/work/nwaku/nwaku/waku.nimble(42, 8) test
2022-11-10T16:44:18.4149300Z /Users/runner/work/nwaku/nwaku/vendor/nimbus-build-system/vendor/Nim/lib/system/nimscript.nim(273, 7) exec
2022-11-10T16:44:18.4149980Z /Users/runner/work/nwaku/nwaku/vendor/nimbus-build-system/vendor/Nim/lib/system/nimscript.nim(273, 7) Error: unhandled exception: FAILED: build/all_tests_v2 [OSError]

Causes: not identified

Action type: CI OS: macOS Test name/version: "peer subscription should be dropped if connection fails for second time after the timeout has elapsed" (v2) nwaku commit/tree: https://github.com/waku-org/nwaku/tree/b38bf152fc50e762a3db098a49eb05f253be76a7 Log excerpt: TimeSeen: 1

2022-12-14T09:46:29.9735490Z     /Users/runner/work/nwaku/nwaku/tests/v2/test_waku_filter.nim(172, 18): Check failed: await pushHandlerFuture.withTimeout(1.seconds)
2022-12-14T09:46:29.9835300Z stack trace: (most recent call last)
2022-12-14T09:46:29.9850710Z /Users/runner/work/nwaku/nwaku/vendor/nimbus-build-system/vendor/Nim/lib/system/nimscript.nim(429, 18)
2022-12-14T09:46:29.9948520Z /Users/runner/work/nwaku/nwaku/waku.nimble(76, 8) test2Task
2022-12-14T09:46:29.9990240Z /Users/runner/work/nwaku/nwaku/waku.nimble(42, 8) test
2022-12-14T09:46:29.9995840Z /Users/runner/work/nwaku/nwaku/vendor/nimbus-build-system/vendor/Nim/lib/system/nimscript.nim(273, 7) exec
2022-12-14T09:46:30.0098480Z /Users/runner/work/nwaku/nwaku/vendor/nimbus-build-system/vendor/Nim/lib/system/nimscript.nim(273, 7) Error: unhandled exception: FAILED: build/all_tests_v2 [OSError]

Causes: not identified (root cause seems to be the same as previous failing test)

Action type: CI OS: linux Test name/version: "Admin API: connect to ad-hoc peers" (v2) nwaku commit/tree: https://github.com/waku-org/nwaku/tree/b38bf152fc50e762a3db098a49eb05f253be76a7 Log excerpt:

2022-12-14T09:14:26.3263997Z /home/runner/work/nwaku/nwaku/vendor/nim-json-rpc/json_rpc/client.nim(467) main
2022-12-14T09:14:26.3264712Z /home/runner/work/nwaku/nwaku/vendor/nim-json-rpc/json_rpc/client.nim(457) NimMain
2022-12-14T09:14:26.3265115Z /home/runner/work/nwaku/nwaku/vendor/nim-json-rpc/json_rpc/client.nim(448) PreMain
2022-12-14T09:14:26.3265524Z /home/runner/work/nwaku/nwaku/vendor/nim-json-rpc/json_rpc/client.nim(116) PreMainInner
2022-12-14T09:14:26.3265946Z /home/runner/work/nwaku/nwaku/vendor/nim-unittest2/unittest2.nim(848) suitePayload
2022-12-14T09:14:26.3266387Z /home/runner/work/nwaku/nwaku/vendor/nim-testutils/testutils/unittests.nim(16) runTestX60gensym680
2022-12-14T09:14:26.3266818Z /home/runner/work/nwaku/nwaku/vendor/nim-testutils/testutils/unittests.nim(15) scenario
2022-12-14T09:14:26.3267262Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncfutures2.nim(372) futureContinue
2022-12-14T09:14:26.3267597Z /home/runner/work/nwaku/nwaku/tests/v2/test_jsonrpc_waku.nim(375) scenario
2022-12-14T09:14:26.3267934Z /home/runner/work/nwaku/nwaku/waku/v2/node/jsonrpc/admin_api.nim(33) constructMultiaddrStr
2022-12-14T09:14:26.3268445Z /home/runner/work/nwaku/nwaku/vendor/nimbus-build-system/vendor/Nim/lib/system/chcks.nim(23) raiseIndexError2
2022-12-14T09:14:26.3268965Z /home/runner/work/nwaku/nwaku/vendor/nimbus-build-system/vendor/Nim/lib/system/fatal.nim(53) sysFatal
2022-12-14T09:14:26.3269262Z [[reraised from:
2022-12-14T09:14:26.3269902Z /home/runner/work/nwaku/nwaku/vendor/nim-json-rpc/json_rpc/client.nim(467) main
2022-12-14T09:14:26.3270350Z /home/runner/work/nwaku/nwaku/vendor/nim-json-rpc/json_rpc/client.nim(457) NimMain
2022-12-14T09:14:26.3270779Z /home/runner/work/nwaku/nwaku/vendor/nim-json-rpc/json_rpc/client.nim(448) PreMain
2022-12-14T09:14:26.3271235Z /home/runner/work/nwaku/nwaku/vendor/nim-json-rpc/json_rpc/client.nim(116) PreMainInner
2022-12-14T09:14:26.3271689Z /home/runner/work/nwaku/nwaku/vendor/nim-unittest2/unittest2.nim(848) suitePayload
2022-12-14T09:14:26.3272178Z /home/runner/work/nwaku/nwaku/vendor/nim-testutils/testutils/unittests.nim(16) runTestX60gensym680
2022-12-14T09:14:26.3272646Z /home/runner/work/nwaku/nwaku/vendor/nim-testutils/testutils/unittests.nim(15) scenario
2022-12-14T09:14:26.3273124Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncfutures2.nim(391) futureContinue
2022-12-14T09:14:26.3273398Z ]]
2022-12-14T09:14:26.3273503Z 
2022-12-14T09:14:26.3273687Z     Unhandled exception: index out of bounds, the container is empty [IndexDefect]
2022-12-14T09:14:26.3274065Z   [FAILED] Admin API: connect to ad-hoc peers

Causes: not identified TimesSeen: 1

Action type: CI OS: linux Test name/version: "mounting waku rln-relay: check correct registration of peers without rln-relay credentials in dynamic/on-chain mode" (v2) - happening in different tests nwaku commit/tree: https://github.com/waku-org/nwaku/tree/226b44c86d7a0abe7a4394f2a52efba77f089c90 Log excerpt:

2022-12-14T11:47:56.8062804Z /home/runner/work/nwaku/nwaku/waku/v2/protocol/waku_relay.nim(560) main
2022-12-14T11:47:56.8063383Z /home/runner/work/nwaku/nwaku/waku/v2/protocol/waku_relay.nim(550) NimMain
2022-12-14T11:47:56.8064116Z /home/runner/work/nwaku/nwaku/waku/v2/protocol/waku_relay.nim(541) PreMain
2022-12-14T11:47:56.8064752Z /home/runner/work/nwaku/nwaku/vendor/nim-unittest2/unittest2.nim(848) suitePayload
2022-12-14T11:47:56.8065269Z /home/runner/work/nwaku/nwaku/vendor/nim-testutils/testutils/unittests.nim(16) runTestX60gensym1280
2022-12-14T11:47:56.8065795Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncloop.nim(1203) waitFor
2022-12-14T11:47:56.8066254Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncloop.nim(295) poll
2022-12-14T11:47:56.8066737Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncfutures2.nim(369) futureContinue
2022-12-14T11:47:56.8067121Z /home/runner/work/nwaku/nwaku/tests/v2/test_waku_rln_relay_onchain.nim(572) scenario
2022-12-14T11:47:56.8067451Z /home/runner/work/nwaku/nwaku/waku/v2/node/waku_node.nim(954) start
2022-12-14T11:47:56.8067897Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncfutures2.nim(369) futureContinue
2022-12-14T11:47:56.8068241Z /home/runner/work/nwaku/nwaku/waku/v2/node/waku_node.nim(984) start
2022-12-14T11:47:56.8068649Z /home/runner/work/nwaku/nwaku/vendor/nim-libp2p/libp2p/switch.nim(296) start
2022-12-14T11:47:56.8069109Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncfutures2.nim(369) futureContinue
2022-12-14T11:47:56.8069548Z /home/runner/work/nwaku/nwaku/vendor/nim-libp2p/libp2p/switch.nim(311) start
2022-12-14T11:47:56.8070231Z /home/runner/work/nwaku/nwaku/vendor/nim-libp2p/libp2p/transports/tcptransport.nim(138) start
2022-12-14T11:47:56.8070717Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncfutures2.nim(372) futureContinue
2022-12-14T11:47:56.8071204Z /home/runner/work/nwaku/nwaku/vendor/nim-libp2p/libp2p/transports/tcptransport.nim(157) start
2022-12-14T11:47:56.8071672Z /home/runner/work/nwaku/nwaku/vendor/nim-libp2p/libp2p/wire.nim(145) createStreamServer
2022-12-14T11:47:56.8071959Z [[reraised from:
2022-12-14T11:47:56.8072226Z /home/runner/work/nwaku/nwaku/waku/v2/protocol/waku_relay.nim(560) main
2022-12-14T11:47:56.8072565Z /home/runner/work/nwaku/nwaku/waku/v2/protocol/waku_relay.nim(550) NimMain
2022-12-14T11:47:56.8072895Z /home/runner/work/nwaku/nwaku/waku/v2/protocol/waku_relay.nim(541) PreMain
2022-12-14T11:47:56.8073324Z /home/runner/work/nwaku/nwaku/vendor/nim-unittest2/unittest2.nim(848) suitePayload
2022-12-14T11:47:56.8073805Z /home/runner/work/nwaku/nwaku/vendor/nim-testutils/testutils/unittests.nim(16) runTestX60gensym1280
2022-12-14T11:47:56.8074280Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncloop.nim(1203) waitFor
2022-12-14T11:47:56.8074717Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncloop.nim(295) poll
2022-12-14T11:47:56.8075177Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncfutures2.nim(369) futureContinue
2022-12-14T11:47:56.8075544Z /home/runner/work/nwaku/nwaku/tests/v2/test_waku_rln_relay_onchain.nim(572) scenario
2022-12-14T11:47:56.8075874Z /home/runner/work/nwaku/nwaku/waku/v2/node/waku_node.nim(954) start
2022-12-14T11:47:56.8076308Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncfutures2.nim(369) futureContinue
2022-12-14T11:47:56.8076656Z /home/runner/work/nwaku/nwaku/waku/v2/node/waku_node.nim(984) start
2022-12-14T11:47:56.8077060Z /home/runner/work/nwaku/nwaku/vendor/nim-libp2p/libp2p/switch.nim(296) start
2022-12-14T11:47:56.8077510Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncfutures2.nim(372) futureContinue
2022-12-14T11:47:56.8077944Z /home/runner/work/nwaku/nwaku/vendor/nim-libp2p/libp2p/switch.nim(318) start
2022-12-14T11:47:56.8078267Z ]]
2022-12-14T11:47:56.8078440Z [[reraised from:
2022-12-14T11:47:56.8078709Z /home/runner/work/nwaku/nwaku/waku/v2/protocol/waku_relay.nim(560) main
2022-12-14T11:47:56.8079043Z /home/runner/work/nwaku/nwaku/waku/v2/protocol/waku_relay.nim(550) NimMain
2022-12-14T11:47:56.8079379Z /home/runner/work/nwaku/nwaku/waku/v2/protocol/waku_relay.nim(541) PreMain
2022-12-14T11:47:56.8079789Z /home/runner/work/nwaku/nwaku/vendor/nim-unittest2/unittest2.nim(848) suitePayload
2022-12-14T11:47:56.8080270Z /home/runner/work/nwaku/nwaku/vendor/nim-testutils/testutils/unittests.nim(16) runTestX60gensym1280
2022-12-14T11:47:56.8080733Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncloop.nim(1203) waitFor
2022-12-14T11:47:56.8081209Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncloop.nim(295) poll
2022-12-14T11:47:56.8081676Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncfutures2.nim(369) futureContinue
2022-12-14T11:47:56.8082057Z /home/runner/work/nwaku/nwaku/tests/v2/test_waku_rln_relay_onchain.nim(572) scenario
2022-12-14T11:47:56.8082383Z /home/runner/work/nwaku/nwaku/waku/v2/node/waku_node.nim(954) start
2022-12-14T11:47:56.8082819Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncfutures2.nim(372) futureContinue
2022-12-14T11:47:56.8083267Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncmacro2.nim(288) start
2022-12-14T11:47:56.8083521Z ]]
2022-12-14T11:47:56.8083692Z [[reraised from:
2022-12-14T11:47:56.8083961Z /home/runner/work/nwaku/nwaku/waku/v2/protocol/waku_relay.nim(560) main
2022-12-14T11:47:56.8084289Z /home/runner/work/nwaku/nwaku/waku/v2/protocol/waku_relay.nim(550) NimMain
2022-12-14T11:47:56.8084619Z /home/runner/work/nwaku/nwaku/waku/v2/protocol/waku_relay.nim(541) PreMain
2022-12-14T11:47:56.8085032Z /home/runner/work/nwaku/nwaku/vendor/nim-unittest2/unittest2.nim(848) suitePayload
2022-12-14T11:47:56.8085557Z /home/runner/work/nwaku/nwaku/vendor/nim-testutils/testutils/unittests.nim(16) runTestX60gensym1280
2022-12-14T11:47:56.8086034Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncloop.nim(1203) waitFor
2022-12-14T11:47:56.8086469Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncloop.nim(295) poll
2022-12-14T11:47:56.8086920Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncfutures2.nim(372) futureContinue
2022-12-14T11:47:56.8087384Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncmacro2.nim(288) scenario
2022-12-14T11:47:56.8087645Z ]]
2022-12-14T11:47:56.8087812Z [[reraised from:
2022-12-14T11:47:56.8088075Z /home/runner/work/nwaku/nwaku/waku/v2/protocol/waku_relay.nim(560) main
2022-12-14T11:47:56.8088410Z /home/runner/work/nwaku/nwaku/waku/v2/protocol/waku_relay.nim(550) NimMain
2022-12-14T11:47:56.8088923Z /home/runner/work/nwaku/nwaku/waku/v2/protocol/waku_relay.nim(541) PreMain
2022-12-14T11:47:56.8089363Z /home/runner/work/nwaku/nwaku/vendor/nim-unittest2/unittest2.nim(848) suitePayload
2022-12-14T11:47:56.8089848Z /home/runner/work/nwaku/nwaku/vendor/nim-testutils/testutils/unittests.nim(16) runTestX60gensym1280
2022-12-14T11:47:56.8090135Z ]]
2022-12-14T11:47:56.8090237Z 
2022-12-14T11:47:56.8090385Z     Unhandled exception: (98) Address already in use [LPError]
2022-12-14T11:47:56.8090898Z   [FAILED] mounting waku rln-relay: check correct registration of peers without rln-relay credentials in dynamic/on-chain mode

Causes: not identified TimesSeen: 3 Temporary workaround: it seems that re-running each failing job one at a time somehow mitigates the issue (root cause might be unrelated).

Action type: CI OS: macOS Test name/version: "prunePeerStore() correctly removes peers to match max quota" v2 (experimental) nwaku commit/tree: https://github.com/waku-org/nwaku/tree/ae0a1171e605e05f03f8efeec3480bba34b62cd7/ Log excerpt:

ERR 2023-02-07 16:33:02.296+00:00 Max number of connections can't be greater than PeerManager capacity tid=67711 file=peer_manager.nim:191 capacity=5 maxConnections=10
  [OK] peer manager cant have more max connections than peerstore size
DBG 2023-02-07 16:33:02.296+00:00 no peer storage found                      tid=67711 file=peer_manager.nim:220
DBG 2023-02-07 16:33:02.297+00:00 Adding peer to manager                     topics="waku node peer_manager" tid=67711 file=peer_manager.nim:244 peerId=16U*Momc9y addresses=@[/ip4/0.0.0.0/tcp/0] proto=
DBG 2023-02-07 16:33:02.297+00:00 Adding peer to manager                     topics="waku node peer_manager" tid=67711 file=peer_manager.nim:244 peerId=16U*8H7Cd9 addresses=@[/ip4/0.0.0.0/tcp/0] proto=
DBG 2023-02-07 16:33:02.297+00:00 Adding peer to manager                     topics="waku node peer_manager" tid=67711 file=peer_manager.nim:244 peerId=16U*qdLnQt addresses=@[/ip4/0.0.0.0/tcp/0] proto=
DBG 2023-02-07 16:33:02.297+00:00 Adding peer to manager                     topics="waku node peer_manager" tid=67711 file=peer_manager.nim:244 peerId=16U*PoAaCn addresses=@[/ip4/0.0.0.0/tcp/0] proto=
DBG 2023-02-07 16:33:02.303+00:00 Adding peer to manager                     topics="waku node peer_manager" tid=67711 file=peer_manager.nim:244 peerId=16U*bX7QpZ addresses=@[/ip4/0.0.0.0/tcp/0] proto=
DBG 2023-02-07 16:33:02.303+00:00 Adding peer to manager                     topics="waku node peer_manager" tid=67711 file=peer_manager.nim:244 peerId=16U*77NSy8 addresses=@[/ip4/0.0.0.0/tcp/0] proto=
DBG 2023-02-07 16:33:02.303+00:00 Adding peer to manager                     topics="waku node peer_manager" tid=67711 file=peer_manager.nim:244 peerId=16U*sBMi5X addresses=@[/ip4/0.0.0.0/tcp/0] proto=
DBG 2023-02-07 16:33:02.303+00:00 Adding peer to manager                     topics="waku node peer_manager" tid=67711 file=peer_manager.nim:244 peerId=16U*vbsUne addresses=@[/ip4/0.0.0.0/tcp/0] proto=
DBG 2023-02-07 16:33:02.303+00:00 Adding peer to manager                     topics="waku node peer_manager" tid=67711 file=peer_manager.nim:244 peerId=16U*r5UTn9 addresses=@[/ip4/0.0.0.0/tcp/0] proto=
DBG 2023-02-07 16:33:02.303+00:00 Adding peer to manager                     topics="waku node peer_manager" tid=67711 file=peer_manager.nim:244 peerId=16U*pQGrsR addresses=@[/ip4/0.0.0.0/tcp/0] proto=
DBG 2023-02-07 16:33:02.303+00:00 Adding peer to manager                     topics="waku node peer_manager" tid=67711 file=peer_manager.nim:244 peerId=16U*twkXMB addresses=@[/ip4/0.0.0.0/tcp/0] proto=
DBG 2023-02-07 16:33:02.303+00:00 Adding peer to manager                     topics="waku node peer_manager" tid=67711 file=peer_manager.nim:244 peerId=16U*TyeLQZ addresses=@[/ip4/0.0.0.0/tcp/0] proto=
DBG 2023-02-07 16:33:02.304+00:00 Adding peer to manager                     topics="waku node peer_manager" tid=67711 file=peer_manager.nim:244 peerId=16U*HzqwTU addresses=@[/ip4/0.0.0.0/tcp/0] proto=
DBG 2023-02-07 16:33:02.304+00:00 Adding peer to manager                     topics="waku node peer_manager" tid=67711 file=peer_manager.nim:244 peerId=16U*WrBXn2 addresses=@[/ip4/0.0.0.0/tcp/0] proto=
DBG 2023-02-07 16:33:02.304+00:00 Adding peer to manager                     topics="waku node peer_manager" tid=67711 file=peer_manager.nim:244 peerId=16U*x6KKq6 addresses=@[/ip4/0.0.0.0/tcp/0] proto=
DBG 2023-02-07 16:33:02.304+00:00 Peer store capacity exceeded               topics="waku node peer_manager" tid=67711 file=peer_manager.nim:403 numPeers=15 capacity=10
DBG 2023-02-07 16:33:02.304+00:00 Finished pruning peer store                topics="waku node peer_manager" tid=67711 file=peer_manager.nim:424 beforeNumPeers=15 afterNumPeers=10 capacity=10 pruned=5
    /Users/runner/work/nwaku/nwaku/tests/v2/test_peer_manager.nim(555, 6): Check failed: not pm.peerStore.peers.anyIt(it.peerId == peers[1].peerId)
    pm.peerStore.peers.anyIt(it.peerId == peers[1].peerId) was true
  [FAILED] prunePeerStore() correctly removes peers to match max quota

Causes: not identified TimesSeen: 1 Temporary fix/workaround: none

jm-clius commented 1 year ago

@s1fr0 any update on this? Have we come to a conclusion whether this is in any way related to timing only?

LNSD commented 1 year ago

Have we come to a conclusion whether this is in any way related to timing only?

This is an umbrella issue. @jm-clius is referring to the investigation held in PR #1368.

jm-clius commented 1 year ago

I will split this into separate issues and address the Waku Bridge failures.

jm-clius commented 1 year ago

Revisiting this. Some impressions from looking at recent errors:

jm-clius commented 1 year ago

Filter test failures are all due to race conditions between client subscribing and service node handling a message that matches that subscription. I will address the timing, but this is a fundamental flaw in the protocol being addressed in: https://github.com/vacp2p/rfc/pull/562

jm-clius commented 1 year ago

Draft for improving filter timeouts: https://github.com/waku-org/nwaku/pull/1529

jm-clius commented 1 year ago

Afaict most of the existing CI issues have been addressed and https://github.com/waku-org/nwaku/pull/1496 will address the remaining timeouts on macos experimental builds.

rymnc commented 1 year ago

1496 has been merged, closing.