search

usgpo / bulk-data

User Guides for XML on the govinfo Bulk Data Repository. For information about Bill Status XML Bulk Data, see https://github.com/usgpo/bill-status.
https://www.govinfo.gov/bulkdata
266 stars 97 forks source link

Bulkdata crawling returns 406 error #67

Closed ogtega closed 3 years ago

ogtega commented 3 years ago

Is the 406 error I get when using python to crawl bulkdata JSON URLs intentional?

ogtega commented 3 years ago

It seems this is due to Cloudflare not supporting dynamic content, I had to add Accept: application/json to my headers.

jonquandt commented 3 years ago

@ogtega - sorry for the delayed response. Yes, the Accept headers are required. I'll make a note in the documentation.