Aiven-Open / tiered-storage-for-apache-kafka

RemoteStorageManager for Apache Kafka® Tiered Storage
Apache License 2.0
85 stars 18 forks source link

Can the remote_log_metadata topic enable the remote.storage.enable configuration? #557

Open funky-eyes opened 1 month ago

funky-eyes commented 1 month ago

What can we help you with?

Because I saw in Kafka documentation that it's desired for the persistence duration of this topic to be greater than the TTL of any other topic, I think this topic might occupy a significant amount of local disk space. Therefore, I would like to store it remotely, for example, in S3.

Where would you expect to find this information?

ivanyu commented 1 month ago

I think, this won't work. There's an explicit prohibition in the code, which guards against the chicken-egg problem: how to bootstrap the remote metadata if they are on the remote storage themselves?

funky-eyes commented 1 month ago

I think, this won't work. There's an explicit prohibition in the code, which guards against the chicken-egg problem: how to bootstrap the remote metadata if they are on the remote storage themselves?

This means that the local disk space must reserve the size of remote_log_metadata. For example, if the TTL of my other topics is six months, remote_log_metadata must be retained for more than six months.

funky-eyes commented 1 month ago

@ivanyu Can __consumer_offsets topic be tiered?