search

delta-io / delta-rs

A native Rust library for Delta Lake, with bindings into Python
https://delta-io.github.io/delta-rs/
Apache License 2.0
2.33k stars 411 forks source link

`max_spill_size` incorrect default value #2794

Closed mrjsj closed 3 months ago

mrjsj commented 3 months ago

Environment

Delta-rs version: 0.19.0

Binding: Rust

Environment:

Bug

What happened: The default value in crates/core/src/operations/optimize.rs::OptimizeBuilder::max_spill_size should be 20 GB, but it's nearly the double, as it's set to 20 1024 1024 * 2014

What you expected to happen: Should be: 20 1024 1024 * 1024

How to reproduce it:

More details: I will create PR with fix

mrjsj commented 3 months ago

take