Rewrite user agent (URL fetching) code to Python

[pypt]

Issue to track rewriting user agent (LWP::UserAgent) to Python.

It goes without saying that Media Cloud relies heavily on fetching stuff from the web. Currently we use LWP::UserAgent to do the job, but we want to move to using Python's requests so that the new code which wants to fetch some stuff from web as well could be written directly in Python.

Feature branch is web_useragent_python.

Tasks:

• [x] Wrap usages of LWP::UserAgent into MediaWords::Util::Web::UserAgent to establish the user agent's API that we're using
• [x] Rename HTTP::HashServer to MediaWords::Test::HTTP::HashServer
• [x] Rewrite MediaWords::Test::HTTP::HashServer to Python to make it easier to test user agent code against a test web server
• [x] Make parallel_get() into a method of the MediaWords::Util::Web::UserAgent
• [x] Move alarabiya hack to MediaWords::Util::Web::UserAgent as all the requests would probably like to employ this hack
• [x] Cover MediaWords::Util::Web::UserAgent with Perl unit tests to help retain the same expected behavior after the Python rewrite:
  • [x] max. download size
  • [x] max. redirects
  • [x] User-Agent: header
  • [x] From: header
  • [x] UTF-8 response
  • [x] non-UTF-8 response
  • [x] invalid encoding
  • [x] timeouts
  • [x] retries via ::Determined of specific HTTP status codes
  • [x] retry timing via ::Determined, plus custom timing
  • [x] max. redirects
  • [x] whether or not cookies are being stored between redirects
  • [x] blacklisted URLs
  • [x] HTTP request log in data/logs/
  • [x] GET
  • [x] POST
  • [x] request(): custom METHOD
  • [x] request(): custom headers
  • [x] request(): custom content type
  • [x] request(): custom content (POST data) -- both hashref and string
  • [x] request(): authorization
  • [x] response: HTTP status code, message, status line
  • [x] response: HTTP headers
  • [x] response: content type
  • [x] response: decoded content (UTF-8, non-UTF-8, and invalid UTF-8)
  • [x] response: successful and unsuccessful responses
  • [x] response: redirects and previous
  • [x] response: get original request
  • [x] response: errors on client/server side
  • [x] get_string()
• [x] Make get() and post() use request() internally; run fix_common_url_mistakes(), is_http_url(), add HTTP auth every time
• [x] Rewrite MediaWords::Util::Web::UserAgent and its tests to Python
• [x] Add test for GB2312 (#185)

[pypt]

Covered existing LWP::UserAgent-based implementation with unit tests in https://github.com/berkmancenter/mediacloud/tree/web_useragent_python, time to rewrite it to Python now.

[pypt]

Good news - UserAgent class rewrite to Python is done in [web_useragent_python branch](), tested (both automatically and manually) and ready to go.

Summary:

• Did some minor refactoring:

  • Moved all of our current code to use MediaWords::Util::Web::UserAgent wrapper instead of plain LWP::UserAgent.
  • UserAgent now has get_follow_http_html_redirects(url: str) -> Response method which will follow all HTML redirects too (<meta /> refreshes, archive.org, archive.is, linkis.com, alarabiya hacks).
  • Moved parallel_get() to a method under UserAgent().

• Covered pretty much 100% of ::UserAgents behavior with Perl unit tests.

• Implemented UserAgent class in Python which uses requests behind the scenes.

• Rewrote UserAgent's Perl unit tests to Python.

• Made Perl's MediaWords::Util::Web::UserAgent into a proxy package which uses Python's UserAgent class behind the scenes.

• Fixed Recognize Chinese chars in sentence extraction because requests decodes GB2312 properly.

• Rewrote encode_json() and decode_json() from MediaWords::Util::JSON:: to Python.

• Added MediaWords::Util::URL::urls_are_equal() (implemented in Python as urls_are_equal(url1: str, url2: str) -> bool) to compare URLs (i.e. urls_are_equal('http://localhost:80///', 'HTTP://LOCALHOST') is True).

• Added is_urls() / isnt_urls() Perl test helpers to MediaWords::Test::URLs so now one can:

  use MediaWords::Test::URLs;
  
  sub test_urls()
  {
      is_urls( 'http://localhost', 'http://localhost/', 'URLs are expected to be equal' );
  }

Backstory for the unitiated (@ColCarroll, partially @rahulbot):

We want to gradually rewrite our core codebase from Perl to Python because of various reasons.

Our chosen approach of the rewrite is to do it gradually, rewriting pieces of code from the bottom up while removing some obsolete code at the same time, until we reach the dreamed of 100% Python codebase. We rewrite the codebase class-by-class and function-by-function, making both Perl and Python code use the very same class or function rewritten to Python.

In essence, Media Cloud is a piece of software which downloads some stuff from the web, stores it, fetches it back at some point and then mingles with said stuff in various ways. Because of that, the bottom-most code for the "bottom-up approach is":

• DatabaseHandler class – "store stuff, fetch stuff" – rewritten to Python some time ago
• UserAgent class – "download some stuff" – just rewritten to Python
• KeyValueStore:: classes – "store stuff, fetch stuff" – next in line to get rewritten

After rewriting those three, we can finally start rewriting the already implemented "mingle with stuff" code, also this opens us a way to write new mingling with stuff code in Python directly, e.g. if all three of the classes above were in Python, the recently deployed CLIFF / NYTLabels fetchers + taggers could have been implemented in Python directly.

While one can use requests directly to fetch some stuff here and there, I highly recommend for us to continue using the less-syntax-sugary but more adapted to our "business" needs MediaWords::Util::Web::UserAgent (Perl) / UserAgent (Python) class whenever we can, at least for fetching random stuff from the unpredictable deep web, because it (among other things):

• Sets the correct User-Agent and From HTTP request headers.
• Maintains a cookie session for a single UserAgent object which is required to fetch content from some websites.
• Limits response sizes by default, makes them configurable.
• Has default response timeout, makes it configurable.
• Limits HTTP redirects, makes it configurable.
• Supports retrying on certain HTTP cherry-picked status codes, makes it configurable.
• Supports resolving not only HTTP but HTML redirects too (e.g. <meta>).
• Detects encodings smarter than requests does by default.
• Automatically adds HTTP authentication to websites for which we have to go through paywalls.
• Supports fetching many URLs in parallel (and limiting the fetch by domain).
• Has URL blacklist for websites that have claimed to dislike us.
• Is tested with exotic encodings.
• ...

The best reference on how UserAgent works is perhaps its unit test. It closely follows how Perl's LWP::UserAgent is implemented (see HTTP::Request, HTTP::Response and HTTP::Message too).

[pypt]

Will deploy tomorrow unless something else comes up.

[pypt]

Oh, I rewrote HashServer to Python too, so now we can test fetching stuff from the web using both Perl and Python.

The rewritten version is more advanced in that it creates forks for every request, so one can test stuff such as parallel_get(), timeouts, crashes etc.

[pypt]

Deployed. There might be some small, easily fixable bugs left here and there (e.g. related to UTF-8 encoding), but in general everything seems to work (e.g. see Japanese query for "North Korea" in Dashboard).

Fixes #185.