Closed aberres closed 5 months ago
I tried with the below specs and performance is ok:
Needed 0.01s to read 100 rows.
Needed 0.02s to read 200 rows.
Needed 0.02s to read 300 rows.
Needed 0.03s to read 400 rows.
Needed 0.03s to read 500 rows.
Needed 0.05s to read 600 rows.
Needed 0.05s to read 700 rows.
Needed 0.05s to read 800 rows.
Needed 0.06s to read 900 rows.
Needed 0.08s to read 1000 rows.
Needed 0.1s to read 1250 rows.
Needed 0.11s to read 1500 rows.
@ylashin thanks! Closing this then as resolved
Environment
Delta-rs version: 0.10.1
Binding: Python
Environment:
Bug
What happened:
When evaluating
delta-rs
as a replacement for hand-managed Parquet files, I noticed that reading files with some thousand columns is unbearably slow. The number of rows does not matter; the issue can be reproduced with a single row.The runtime does not increase linearly. See the listing below.
Reading the created Parquet file directly does not have any performance issues.
What you expected to happen:
The reading performance should not massively degrade when the number of columns increases.
How to reproduce it:
An absolutely naive script, writing and reading back a frame with a single row.
More details:
~2000 rows did not finish within several minutes.~ Do not succeed at all, the process crashed with SIGBUS.
Snakeviz profile: