Closed RandomFractals closed 1 year ago
Using those flags brings DuckDB CSV data loading on par with PyArrow & Polars. It used to take about 16s to load that data without multi-threaded CSV data reader (#4)
See updated pic and other sections with data loading timings in docs:
https://github.com/RandomFractals/chicago-crimes#with-duckdb
to speed up CSV data loading with DuckDB in
chicago-crimes-duckdb.ipynb
example notebook created in #4.See DuckDB v0.6.0 update notes: https://duckdb.org/2022/11/14/announcing-duckdb-060.html
Use
SET preserve_insertion_order=false
to enable unordered insertion.Use
SET experimental_parallel_csv=true
for multi-threaded CSV data loading.