Remote fetch operations failed with interrupted exception

Aiven-Open / tiered-storage-for-apache-kafka

RemoteStorageManager for Apache Kafka® Tiered Storage

Apache License 2.0

83 stars 17 forks source link

Remote fetch operations failed with interrupted exception #483

Open jeqo opened 5 months ago

jeqo commented 5 months ago

Currently, fetch.max.wait.ms is reused on remote fetches to cancel operations if taking longer than timeout. Given the default value is 500ms, it may end up being too small for the whole remote operation to complete.

Ideally this would be default by a different configuration. Some alternatives are being discussed here: https://issues.apache.org/jira/browse/KAFKA-15776

Once a better approach for this issue is implemented upstream, update the plugin to support it properly.

jeqo commented 1 month ago

https://cwiki.apache.org/confluence/display/KAFKA/KIP-1018%3A+Introduce+max+remote+fetch+timeout+config+for+DelayedRemoteFetch+requests is accepted and targeting 3.8 -- once that's released, we can close this issue.