apache / datafusion-python

Apache DataFusion Python Bindings
https://datafusion.apache.org/python
Apache License 2.0
323 stars 64 forks source link

Configure memory usage #514

Open shner-elmo opened 9 months ago

shner-elmo commented 9 months ago

I saw a pull-request opened a while back that introduced the possibility of analyzing bigger than memory datasets by saving some of the state on disk, I was wondering if there is a way for the user to configure the maximum amount of memory the process can use?

Like duckdb has "SET max_memory='1GB';"