This change creates a requests Session object at the BaseDownloader level and loads it with the headers and cookies that are specified in a config file, as keyword arguments, or command line arguments. In part this keeps us from having to repeat ourselves in instances where we have multiple requests that require the same arguments, and it should slightly simplify the story for new downloaders.
All the old-style downloaders will continue to work, but should eventually be updated to use the Session established by the BaseDownloader. For this PR, I've updated the WSJ downloader to the new syntax in order to put together a fix for #178. I've also attempted to update the tests that it uses... we'll see!
This change creates a requests Session object at the BaseDownloader level and loads it with the
headersandcookiesthat are specified in a config file, as keyword arguments, or command line arguments. In part this keeps us from having to repeat ourselves in instances where we have multiple requests that require the same arguments, and it should slightly simplify the story for new downloaders.All the old-style downloaders will continue to work, but should eventually be updated to use the Session established by the BaseDownloader. For this PR, I've updated the WSJ downloader to the new syntax in order to put together a fix for #178. I've also attempted to update the tests that it uses... we'll see!