ipfs / kubo

An IPFS implementation in Go
https://docs.ipfs.tech/how-to/command-line-quick-start/
Other
16.18k stars 3.01k forks source link

Flaky test: ci/gh-experiment: go test: pubsub_msg_seen_cache_test #9554

Open lidel opened 1 year ago

lidel commented 1 year ago

Creating issue so others don't spend time investigating.

TestMessageSeenCacheTTL sometimes fails on GitHub version of our CI (log below).

@galargh I suggest we wait with investigation until https://github.com/ipfs/kubo/pull/9543 lands, as these tests will be refactored by the mentioned PR anyway.

https://github.com/ipfs/kubo/actions/runs/3926432440/jobs/6712202879#step:5:11887:

=== FAIL: test/integration TestMessageSeenCacheTTL (1.54s)

Computing default go-libp2p Resource Manager limits based on:
    - 'Swarm.ResourceMgr.MaxMemory': "1.8 GB"
    - 'Swarm.ResourceMgr.MaxFileDescriptors': 32768

Applying any user-supplied overrides on top.
Run 'ipfs swarm limit all' to see the resulting limits.

Computing default go-libp2p Resource Manager limits based on:
    - 'Swarm.ResourceMgr.MaxMemory': "1.8 GB"
    - 'Swarm.ResourceMgr.MaxFileDescriptors': 32768

Applying any user-supplied overrides on top.
Run 'ipfs swarm limit all' to see the resulting limits.

Computing default go-libp2p Resource Manager limits based on:
    - 'Swarm.ResourceMgr.MaxMemory': "1.8 GB"
    - 'Swarm.ResourceMgr.MaxFileDescriptors': 32768

Applying any user-supplied overrides on top.
Run 'ipfs swarm limit all' to see the resulting limits.

    pubsub_msg_seen_cache_test.go:136: bootstrap peer=QmeDXb79V25BrSP4ENDvqZMd7qBr7SXuKSNwxxQawjdbwK, consumer peer=QmaxMa42XFxgG8Rn6TPCvsSKfZ4dbxxwnstt4AK8xaqUqS, producer peer=QmZWxchjiixjTNgh5TA8XZFrYfQPSaDkiRRH8AFCkmy41e
    pubsub_msg_seen_cache_test.go:114: sending [msg_1] with duplicate message ID at [Jan 16 02:10:31.625]
    pubsub_msg_seen_cache_test.go:191: did not receive [msg_1] by [Jan 16 02:10:32.626]
    pubsub_msg_seen_cache_test.go:192: context deadline exceeded
lidel commented 1 year ago

https://github.com/ipfs/kubo/pull/9543 by @smrz2001 made this happen less often, but it is still flaky, example:

https://github.com/ipfs/kubo/actions/runs/4070303244/jobs/7011002380#step:5:12546 https://github.com/ipfs/kubo/actions/runs/4070013905/jobs/7010388391#step:5:12546

=== FAIL: test/integration TestMessageSeenCacheTTL (1.92s)

Computing default go-libp2p Resource Manager limits based on:
    - 'Swarm.ResourceMgr.MaxMemory': "3.6 GB"
    - 'Swarm.ResourceMgr.MaxFileDescriptors': 32768

Applying any user-supplied overrides on top.
Run 'ipfs swarm limit all' to see the resulting limits.

Computing default go-libp2p Resource Manager limits based on:
    - 'Swarm.ResourceMgr.MaxMemory': "3.6 GB"
    - 'Swarm.ResourceMgr.MaxFileDescriptors': 32768

Applying any user-supplied overrides on top.
Run 'ipfs swarm limit all' to see the resulting limits.

Computing default go-libp2p Resource Manager limits based on:
    - 'Swarm.ResourceMgr.MaxMemory': "3.6 GB"
    - 'Swarm.ResourceMgr.MaxFileDescriptors': 32768

Applying any user-supplied overrides on top.
Run 'ipfs swarm limit all' to see the resulting limits.

    pubsub_msg_seen_cache_test.go:146: bootstrap peer=Qmdw59arMi2qaCqf3eoYD1N2qt7ppo1tirVRGqA8QyqcKK, consumer peer=QmfFyYgtsqBCmhojuVDGfWEd7hf8HZfcajp5hPVFeZVUut, producer peer=QmXuEfsHGqHtQbTpih1t6zdoHbtgSwvShyrkjs53nWXNK1
    pubsub_msg_seen_cache_test.go:124: sending [msg_1] with message ID [MsgID1] at T0.000000s
    pubsub_msg_seen_cache_test.go:201: expected but did not receive [msg_1] at T1.000629s
    pubsub_msg_seen_cache_test.go:202: context deadline exceeded

DONE 1975 tests, 585 skipped, 1 failure in 0.093s
smrz2001 commented 1 year ago

https://github.com/ipfs/kubo/pull/9543 by @smrz2001 made this happen less often, but it is still flaky, example:

Sorry you're having to deal with this ☹️ I'll spend some time trying to make this test more reliable.