Upgrade to Hadoop 3

[anjackson]

This is a major change, updating everything to run on Hadoop 3 and take advantage of it. This transition is a long way off!

In the meantime, this branch will be run in parallel as we are running two clusters in parallel.

• [ ] Restore test that had to be taken out of the h020 branch in d41faf8