In our current system, an index cache miss triggers the loading of the entire index file from s3, regardless of the actual amount of data required. This behavior causes significant inefficiencies in scenarios where only a small portion of the index file is needed for the query.
In some cases, only a small fraction of the index data is accessed. Despite this, the system downloads the entire 200 MiB index file. The download time dominates the query's total execution time, leading to degraded performance.
What type of enhancement is this?
Performance
What does the enhancement do?
In our current system, an index cache miss triggers the loading of the entire index file from s3, regardless of the actual amount of data required. This behavior causes significant inefficiencies in scenarios where only a small portion of the index file is needed for the query.
In some cases, only a small fraction of the index data is accessed. Despite this, the system downloads the entire 200 MiB index file. The download time dominates the query's total execution time, leading to degraded performance.
Implementation challenges
No response