Is it possible to add a multiprocessing argument?

DocNow / twarc-csv

A plugin for twarc2 for converting tweet JSON into DataFrames and exporting to CSV.

MIT License

31 stars 10 forks source link

Is it possible to add a multiprocessing argument? #59

Open myrainbowandsky opened 1 year ago

myrainbowandsky commented 1 year ago

We could split a csv into pieces of csvs and use twarc-csv for each of them. Or maybe we could use joblib's n_jobs to read CPU-thread number of blocks of Json lines. A server usually has multi-cores. Is it possible to add such a parameter?

igorbrigadir commented 1 year ago

Yes, it's not implemented but possible, and welcome (with an added requirement of keeping things in the same order as the json, as this helps downstream tools).

The process of generating CSVs is already "embarrassingly parallel" as it splits the data into chunks here: https://github.com/DocNow/twarc-csv/blob/main/csv_writer.py#L82 so that would be the best place for it.

Joblib would be nice. Will accept good PRs for this.