neuml / paperetl

📄 ⚙️ ETL processes for medical and scientific papers
Apache License 2.0
342 stars 27 forks source link

Add multiprocessing support to files process #38

Closed davidmezzetti closed 2 years ago

davidmezzetti commented 2 years ago

Currently, the files etl process is single threaded. With #34, it is now much easier to parallelize this process. This change will use a multiprocessing pool to process files.