eliangcs / pystock-crawler

(UNMAINTAINED) Crawl and parse financial reports (XBRL) from SEC EDGAR, and daily stock prices from Yahoo Finance
MIT License
311 stars 99 forks source link

A batch size option for crawling reports #13

Closed eliangcs closed 10 years ago

eliangcs commented 10 years ago

2 only occurs when the symbol list is large (~5k symbols). To work around it, maybe I can add a --batch-size option to the pystock-crawler reports command. This option splits a large symbol list into smaller parts and merges them into one output file when crawling is done.