search

IATI / IATI-Datastore

An open-source datastore for IATI data with RESTful web API providing XML, JSON, CSV plus ETL tools
http://datastore.iatistandard.org/
Other
1 stars 0 forks source link

Consider setting the User-Agent to ‘IATI datastore’ or similar #262

Closed andylolz closed 3 years ago

andylolz commented 7 years ago

I.e. just here: https://github.com/IATI/IATI-Datastore/blob/5b871aa3/iati_datastore/iatilib/crawler.py#L163

At least one publisher blocks requests with a User-Agent containing ‘python-requests’ (!) But regardless of that, it’s useful for publishers to have their own record of when their data is being harvested by the datastore & other IATI tools etc.

andylolz commented 7 years ago

NB the registry archiver also does this: https://github.com/ViderumGlobal/ckanext-iati/blame/master/README.rst#L110-L111