It seems that the logic of this script is to do incremental update of edgar.filing_docs.
Def14_a never gets updated, so the code will scrape the same set of files in the while loop endlessly.
SEC has traffic controls. Paralleling with 8 cores will cause the IP to be blocked. I tested and found that 2 cores plus sleeping for 0.5s works (at least on my server). The key is to avoid submitting more than 10 requests per second, otherwise SEC will block the IP for 10 min.
It seems that the logic of this script is to do incremental update of
edgar.filing_docs.Def14_anever gets updated, so the code will scrape the same set of files in the while loop endlessly.SEC has traffic controls. Paralleling with
8cores will cause the IP to be blocked. I tested and found that2cores plus sleeping for0.5s works (at least on my server). The key is to avoid submitting more than10requests per second, otherwise SEC will block the IP for 10 min.