search

stangurrr / congress_whowhat

1 stars 1 forks source link

parallelize webcrawl #3

Open mynameisvinn opened 6 years ago

mynameisvinn commented 6 years ago

new notebook parallelizes crawling. by defining N_THREADS, the number of threads/workers, you can fetch data from govinfo.gov much faster. (eg using four threads means 4x speedup.)