search

Aiven-Open / tiered-storage-for-apache-kafka

RemoteStorageManager for Apache Kafka® Tiered Storage
Apache License 2.0
95 stars 20 forks source link

Use direct I/O with disk cache #457

Open ivanyu opened 11 months ago

ivanyu commented 11 months ago

The disk cache (obviously) reads and writes files. Currently, the files are open normally so they get into the OS page cache. Kafka heavily depends on the page cache for its consumer performance. A big read from the remote storage via the disk cache may wipe the hot local data from the page cache, thus affecting negatively non-remote consumer performance.

We should consider opening disk cache files in the direct I/O mode to not touch the page cache and make this behavior configurable.

funky-eyes commented 8 months ago

when is this feature expected to be available back?