feat(harvard_pdfs): parallelize cap pdf download

freelawproject / courtlistener

A fully-searchable and accessible archive of court data including growing repositories of opinions, oral arguments, judges, judicial financial records, and federal filings.

https://www.courtlistener.com

Other

550 stars 151 forks source link

feat(harvard_pdfs): parallelize cap pdf download #4636

Closed quevon24 closed 3 weeks ago

quevon24 commented 3 weeks ago

Add argument to set number of workers (2 by default)
Add argument to set reporter to start from
Improve the readability of reporters names
Tests updated

Set workers number manage.py import_harvard_pdfs --crosswalk-dir /opt/courtlistener/cl/assets/media/crosswalk --max-workers 4 Go thru all reporters but starting from specific repoter:

manage.py import_harvard_pdfs --crosswalk-dir /opt/courtlistener/cl/assets/media/crosswalk --start-from-reporter "Hill"

quevon24 commented 3 weeks ago

The change is ready. I'll set it to auto-merge