search

vacp2p / nim-libp2p

libp2p implementation in Nim
https://vacp2p.github.io/nim-libp2p/docs/
MIT License
242 stars 52 forks source link

Improve memory efficiency of seen cache #1073

Closed arnetheduck closed 2 months ago

arnetheduck commented 3 months ago

The seen cache currently is a significant memory usage hotspot due to its inefficient implementation: for every entry, two copies of the message id + timing data + seq overhead causes it to use much more memory than it has to.

In addition, each check involves several layers of allocations as the computed message id gets salted.

This PR improves on the situation by:

codecov-commenter commented 3 months ago

Codecov Report

Attention: Patch coverage is 88.70968% with 7 lines in your changes are missing coverage. Please review.

:exclamation: No coverage uploaded for pull request base (master@2b53196). Click here to learn what that means.

Additional details and impacted files [![Impacted file tree graph](https://app.codecov.io/gh/vacp2p/nim-libp2p/pull/1073/graphs/tree.svg?width=650&height=150&src=pr&token=M88zHaQffJ&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vacp2p)](https://app.codecov.io/gh/vacp2p/nim-libp2p/pull/1073?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vacp2p) ```diff @@ Coverage Diff @@ ## master #1073 +/- ## ========================================= Coverage ? 84.53% ========================================= Files ? 91 Lines ? 15517 Branches ? 0 ========================================= Hits ? 13118 Misses ? 2399 Partials ? 0 ``` | [Files](https://app.codecov.io/gh/vacp2p/nim-libp2p/pull/1073?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vacp2p) | Coverage Δ | | |---|---|---| | [libp2p/protocols/pubsub/floodsub.nim](https://app.codecov.io/gh/vacp2p/nim-libp2p/pull/1073?src=pr&el=tree&filepath=libp2p%2Fprotocols%2Fpubsub%2Ffloodsub.nim&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vacp2p#diff-bGlicDJwL3Byb3RvY29scy9wdWJzdWIvZmxvb2RzdWIubmlt) | `88.18% <100.00%> (ø)` | | | [libp2p/protocols/pubsub/gossipsub.nim](https://app.codecov.io/gh/vacp2p/nim-libp2p/pull/1073?src=pr&el=tree&filepath=libp2p%2Fprotocols%2Fpubsub%2Fgossipsub.nim&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vacp2p#diff-bGlicDJwL3Byb3RvY29scy9wdWJzdWIvZ29zc2lwc3ViLm5pbQ==) | `86.46% <100.00%> (ø)` | | | [libp2p/protocols/pubsub/gossipsub/behavior.nim](https://app.codecov.io/gh/vacp2p/nim-libp2p/pull/1073?src=pr&el=tree&filepath=libp2p%2Fprotocols%2Fpubsub%2Fgossipsub%2Fbehavior.nim&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vacp2p#diff-bGlicDJwL3Byb3RvY29scy9wdWJzdWIvZ29zc2lwc3ViL2JlaGF2aW9yLm5pbQ==) | `88.83% <100.00%> (ø)` | | | [libp2p/protocols/pubsub/rpc/messages.nim](https://app.codecov.io/gh/vacp2p/nim-libp2p/pull/1073?src=pr&el=tree&filepath=libp2p%2Fprotocols%2Fpubsub%2Frpc%2Fmessages.nim&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vacp2p#diff-bGlicDJwL3Byb3RvY29scy9wdWJzdWIvcnBjL21lc3NhZ2VzLm5pbQ==) | `52.85% <ø> (ø)` | | | [libp2p/protocols/pubsub/timedcache.nim](https://app.codecov.io/gh/vacp2p/nim-libp2p/pull/1073?src=pr&el=tree&filepath=libp2p%2Fprotocols%2Fpubsub%2Ftimedcache.nim&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vacp2p#diff-bGlicDJwL3Byb3RvY29scy9wdWJzdWIvdGltZWRjYWNoZS5uaW0=) | `82.89% <78.78%> (ø)` | |
arnetheduck commented 3 months ago

On holesky, this PR reduces memory usage of the seen cache by ~100mb