Implement Scala Matchbox UDFs in Python.

[ruebot]

GitHub issue(s):

• 408

• 409

• 410

What does this Pull Request do?

Implement Scala Matchbox UDFs in Python.

• Resolves #408
• Alphabetizes DataFrameloader functions
• Alphabetizes UDFs functions
• Move DataFrameLoader to df packages
• Move UDFs out of df into their own package
• Rename UDFs (no more DF tagged to the end).
• Update tests as necessary
• Partially addresses #410, #409
• Supersedes #412.

How should this be tested?

• TravisCI
• Adapt this to local files.
• https://github.com/archivesunleashed/aut-docs/pull/62

Additional Notes:

1. I made a number of structural changes to the Scala side. @lintool, please let me know if you take strong issue with anything.

2. I'm going to punt on the hasX filters for right now, and loop back around to them. I hit a wall with trying to get them to run in PySpark, and part of me is tempted to just say that we should go with the natural PySpark (Python) implementation here. Basically:

from aut import *

languages = ["es", "fr"]

WebArchive(sc, sqlContext, "/data")\
  .webpages()\
  .filter(col("language").isin(languages))\
  .select("crawl_date")\
  .show(10, True)

or

from aut import *

languages = ["es", "fr"]

WebArchive(sc, sqlContext, "/data")\
  .webpages()\
  .filter(~col("language").isin(languages))\
  .select("crawl_date")\
  .show(10, True)

Instead of

from aut import *

languages = ["es", "fr"]

WebArchive(sc, sqlContext, "/data")\
  .webpages()\
  .filter(Udf.has_language("language", languages))\
  .select("crawl_date")\
  .show(10, True)

or

from aut import *

languages = ["es", "fr"]

WebArchive(sc, sqlContext, "/data")\
  .webpages()\
  .filter(~Udf.has_language("language", languages))\
  .select("crawl_date")\
  .show(10, True)

Basically, an argument I made in #425.

[codecov[bot]]

Codecov Report

Merging #463 into master will decrease coverage by 0.05%. The diff coverage is 94.54%.

@@            Coverage Diff             @@
##           master     #463      +/-   ##
==========================================
- Coverage   76.49%   76.43%   -0.06%     
==========================================
  Files          49       50       +1     
  Lines        1459     1460       +1     
  Branches      279      279              
==========================================
  Hits         1116     1116              
- Misses        213      214       +1     
  Partials      130      130              

[ruebot]

Yeah, I could clean that notebook up, and toss it in https://github.com/archivesunleashed/notebooks when we're done.