Open rohitprg opened 2 months ago
hi @rohitprg - I've observed this with DuckDB as well (I think). Could you retry your test with just DuckDB (no Flight SQL) to see if the behavior is similar?
Thanks @prmoore77 - I will perform my tests on DuckDB and update you with the results. Do you know of any potential solutions to this issue?
DuckDB flight-sql-server appears to have an issue with its memory management, as it does not seem to reset its memory usage between each query, causing a continuous increase in memory consumption.
This issue surfaced during performance benchmarking when reading parquet files from Azure blob storage was the primary function. The benchmarking was performed using a Python script which utilized the Locust framework for load testing. This script replicated user behavior by running a sequence of SQL queries and noting the response time for each.
The problem lies in DuckDB flight-sql-server apparent failure to clear its memory usage between these queries, resulting in a steady build-up of memory usage with each query executed.
The series of tests started with 10 users and gradually escalated to 300 concurrent users. Throughout these tests, an Out of Memory error was consistently encountered. This persistent increase in memory usage indicates that DuckDB flight-sql-server is not resetting its memory use between queries. Even after a prolonged period of inactivity, exceeding eight hours, with no load being sent, the memory usage remained the same and did not decrease.