apache / datafusion

Apache DataFusion SQL Query Engine
https://datafusion.apache.org/
Apache License 2.0
5.48k stars 1.01k forks source link

Add high water mark to MemoryManager #2053

Open matthewmturner opened 2 years ago

matthewmturner commented 2 years ago

Is your feature request related to a problem or challenge? Please describe what you are trying to do. A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] (This section helps Arrow developers understand the context and why for this feature, in addition to the what)

Following conversations from slack it would be useful to have a public high water mark attribute for the memory used by a query. Then this would be able to be used by system that use datafusion, such as IOx or datafusion-tui.

Describe the solution you'd like A clear and concise description of what you want to happen. I havent had the chance to look into this yet so im not sure the practicality of it but maybe an attribute like last_query_peak_memory being added to MemoryManager.

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context Add any other context or screenshots about the feature request here.

matthewmturner commented 2 years ago

CC @yjshen @alamb