S3 verifier cleanup

[lkemmerer]

I'm still running this on iad.dev for a bit and I think I want to force some failures and make sure things recover, but here's a first run at refactoring the S3Verifier. It probably makes sense to walk through the PR commit-by-commit.

Main changes include:

• extracting executor services out and making them injectable; this let me write some test around slow queries and timeouts. I may want to add more testing, espeically around one query taking a long time, but the other one not...

• extracting hub properties out into a couple of configs + ConfigProviders.

• using QueryResults for stuff.

• moving some code around into new files and wrote unit tests around MissingItemFinder (where we do the queries and find...missing s3 items...).

• Changing the way we're doing async/threading:

  • Instead of using Runnables and modifying a set as part of it, it uses Consumable that return the Set of results.
  • Instead of using a CountdownLatch and whatnot, it uses CompletableFutures, which have similar timeout functionality. Added a finally block that cancels any futures that aren't done running.

I am not sure that CompletableFutures are the best tool for the job, but it seemed like it was heading in a more positive direction. Someone with deeper knowledge of Java threading, async stuff, and concurrency should probably give me some suggestions.

[lkemmerer]

Thanks for taking a look, @chriskessel! I'll make those changes soonish.