wireservice / csvkit

A suite of utilities for converting to and working with CSV, the king of tabular file formats.
https://csvkit.readthedocs.io
MIT License
5.98k stars 608 forks source link

How to utilize all available Cores with csvsql? #1205

Closed Gill-Bates closed 12 months ago

Gill-Bates commented 1 year ago

I have a big CSV-File (150 MB) which I want to import into Postgres. When I run csvsql, I can see, that the tool is not using all available cores:

image

Is there a trick how I can control csvsql to use all cores?

./csvsql -V
`csvsql 1.1.1`
uname -a
inux vmgrafana 6.1.0-11-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.38-4 (2023-08-08) x86_64 GNU/Linux
./python -V
Python 3.11.2
jpmckinney commented 1 year ago

I don't see much opportunity to use multiprocessing in csvsql – especially for an import. Where would it thread?

If you want the fastest import, just use COPY. https://www.postgresql.org/docs/current/sql-copy.html

jpmckinney commented 12 months ago

Closing as no response.