search

leoliu0 / cik-cusip-mapping

provide cik to cusip links using 13G and 13D filings
151 stars 48 forks source link

Missing Data full_index.csv #4

Closed xu000112-bit closed 3 years ago

xu000112-bit commented 3 years ago

Hi Dr Liu,

Your cik-cusio-mapping database is so powerful and I am really impressed with it. However, when I try to run code python dl.py 13D 13D, it raises an error that I am missing 'full_index.csv' file. I am wondering whether you could share the full_index.csv file?

jonasniemeyer commented 3 years ago

If you still face the issue, run dl_idx.py first. This will create full_index.csv with all filing urls.

Note that the SEC requires you to declare user agent data, hence you need headers in your request statement. For that, add headers = { "Connection": "keep-alive", "Expires": "-1", "Upgrade-Insecure-Requests": "-1", "User-Agent": ( "Mozilla/5.0 (X11; CrOS x86_64 12871.102.0) AppleWebKit/537.36 " "(KHTML, like Gecko) Chrome/81.0.4044.141 Safari/537.36" ), } inside dl.py and dl_idx.py and put an additional headers=headers parameter in each requests.get function call.

leoliu0 commented 3 years ago

Thank you both. It seems SEC has changed the user-agent requirement. I will add that to the dl_index.py

leoliu0 commented 3 years ago

Added user agent, so the script should work now.