In our latest release of prysm, we updated it to use version v0.9.2 of go-libp2p-pubsub. However, immediately after a few hours
we received multiple reports of memory leaks with prysm and prysm nodes constantly crashing with an OOM. After fetching a few profiles, this is what we found:
The majority of heap usage was being consumed by the first seen cache. After digging into it and the PR which introduced it
https://github.com/libp2p/go-libp2p-pubsub/pull/523 , it was found out that the background routine was not clearing our expired entries from the cache. This is because it used NewTimer instead of NewTicker when performing the sweep. So this would only fire an event once into the channel and fire no more events after.
This PR introduces the fix along with fixing the test so that this particular case is correctly checked for.
In our latest release of prysm, we updated it to use version v0.9.2 of
go-libp2p-pubsub. However, immediately after a few hours we received multiple reports of memory leaks with prysm and prysm nodes constantly crashing with an OOM. After fetching a few profiles, this is what we found:The majority of heap usage was being consumed by the first seen cache. After digging into it and the PR which introduced it https://github.com/libp2p/go-libp2p-pubsub/pull/523 , it was found out that the background routine was not clearing our expired entries from the cache. This is because it used
NewTimerinstead ofNewTickerwhen performing the sweep. So this would only fire an event once into the channel and fire no more events after.This PR introduces the fix along with fixing the test so that this particular case is correctly checked for.