cmdlineluser
commented
2 months ago https://github.com/pola-rs/polars/issues/18235#issuecomment-2295187859
Our new streaming engine will be able to this. It will be able to multiplex and to multiple intermediate/output nodes.
Description
Currently there is no possibility to write data on disk without materializing it when dealing with many lazy frames which are nested from one lazy frame
The only possible way is to mimic it by using collect_all_async and hoping for the best