search

oduwsdl / archivenow

A Tool To Push Web Resources Into Web Archives
MIT License
403 stars 42 forks source link

User configurable sessions #45

Closed shawnmjones closed 4 years ago

shawnmjones commented 4 years ago

These changes should allow a user to specify their own session objects, for setting their own request headers (like user agents) and more. It has only been tested with IA and archive.today.

maturban commented 4 years ago

I merged the work done by @shawnmjones. However, after some tests, I realized that the condition if 'user-agent' in session.headers:, added in each IA and IS handlers, will be always "True" because by default session.headers include User-Agent (e.g., 'User-Agent': 'python-requests/2.13.0')

shawnmjones commented 4 years ago

Thanks for catching this.