search

Aiven-Open / tiered-storage-for-apache-kafka

RemoteStorageManager for Apache Kafka® Tiered Storage
Apache License 2.0
91 stars 19 forks source link

Allow fetch caching to complete even if input stream closes #404

Closed jeqo closed 1 month ago

jeqo commented 1 year ago

We have experience issues when fetch.max.wait.ms is too small for tiered data to be served. The default config value is 500ms and it may not be enough for S3 to return a result and make it available to the broker. The problem gets worse when consumer retries and the data has to be fetched again from S3.

A potential solution, would be to allow the caching mechanism to continue even when the input stream gets closed, so the next request will find it on the cache, and will have better changes to succeed.

jeqo commented 1 month ago

Already implemented https://github.com/Aiven-Open/tiered-storage-for-apache-kafka/pull/429