openzim / python-scraperlib

Collection of Python code to re-use across Python-based scrapers
GNU General Public License v3.0
18 stars 16 forks source link

handle_user_provided_file should set a User-Agent #103

Closed rgaudin closed 4 months ago

rgaudin commented 1 year ago

Although this function is not used for mass downloads but just to pick a couple files per scrape, because this is good practice, and because files hosted on Wikimedia servers require it, we should set a User-Agent for requests emited through it.

https://meta.wikimedia.org/wiki/User-Agent_policy