Examine performance of scrape_filing_docs.R

mccgr / edgar

Code to manage data related to SEC EDGAR

31 stars 15 forks source link

Examine performance of scrape_filing_docs.R #92

Open iangow opened 4 years ago

iangow commented 4 years ago

This seems slow. My guess is that the query needs improvement (perhaps an index somewhere).

iangow commented 4 years ago

This issue is particularly bad when filing_docs already contains everything in filings (i.e., there is nothing to scrape). The performance of the anti-join is very bad in this case and I'm not sure how to speed it up.