search

delta-io / delta-rs

A native Rust library for Delta Lake, with bindings into Python
https://delta-io.github.io/delta-rs/
Apache License 2.0
2.3k stars 404 forks source link

max_concurrent_tasks don't work with z_order #1656

Open marsupialtail opened 1 year ago

marsupialtail commented 1 year ago

Environment

Delta-rs version: 0.10.2

Binding: Python

Environment: Ubuntu 22:

Bug

see title.

marsupialtail commented 1 year ago

@wjones127

wjones127 commented 1 year ago

In order to fix this, we need to pass down a memory limit to DataFusion. Also we might want to consider passing down a limit on the number of threads used, though perhaps a memory limit is sufficient.