Create a shared scraping function.

r-three / common-pile

Repo to hold code and track issues for the collection of permissively licensed data

MIT License

22 stars 6 forks source link

Create a shared scraping function. #62

Closed blester125 closed 6 months ago

blester125 commented 6 months ago

This PR adds a shared scraping function to the licensed pile shared library. It has a default user-agent string and smart retries. We should use it when we need to HTTP GET a resource from within python.