Currently we write all jobs for one delta table in one write in a referencejob that references all jobs.
There seems to be a problem in the delta rust implementation that materializes all tables in memory before writing them to the destination:
Create multiple followup jobs per table and control the amount of jobs assigned in each via a setting. Ensure we only process one job per table in parallel if this is set. (loader_parallelism_strategy=table-sequential)
User other engine than rust, because rust seems to have this problem. Merge does not work there though atm.
Currently we write all jobs for one delta table in one write in a referencejob that references all jobs. There seems to be a problem in the delta rust implementation that materializes all tables in memory before writing them to the destination:
https://github.com/delta-io/delta-rs/issues/2968#issuecomment-2453034758
Possible ways to fix this: