Aiven-Open / tiered-storage-for-apache-kafka

RemoteStorageManager for Apache Kafka® Tiered Storage
Apache License 2.0
95 stars 20 forks source link

feat: bypass chunking when no transformation is requested #597

Closed jeqo closed 1 month ago

jeqo commented 1 month ago

Pass directly the file input stream to the storage layer instead of calculating chunks on the fly. Optimizes for the default configuration (no encryption/compression), and prepares for further passing the path to the storage layer if uploading directly from file is preferred.

It contains a couple of meaningful refactoring to clean up RSM logic and simplify chunking. See commits for further details.