We experienced issues when deploying https://github.com/getsentry/arroyo/pull/295 to production. It was discovered that very old commit log entries are still present. This is likely because the commit log topic has cleanup.policy=compact set, which causes messages to never expire if they have a unique key.
We experienced issues when deploying https://github.com/getsentry/arroyo/pull/295 to production. It was discovered that very old commit log entries are still present. This is likely because the commit log topic has cleanup.policy=compact set, which causes messages to never expire if they have a unique key.
The process of fixing this is underway: https://github.com/getsentry/snuba/pull/4941 and https://github.com/getsentry/ops/pull/8361
However we have to keep support for the legacy commit log format around for a while until this work is complete.