Better Watermarking and LAD monitoring

[olessia]

More logging for LAD

• Log counts of late-arriving data (e.g. late-arriving records for dir foo in /data/raw/kafka/foo/2017/11/01/17: 2)
• Report the fact that there was LAD to Datadog.

Configurable Watermarking delays

• Add delay-hours parameter that allows delaying watermarking.
• This allows us to delay watermarking by the specified number of hours.
• Defaults to 0 to preserve current behaviour. The parameter should be set to a sensible value to the Azkaban task in a separate PR.

LAD check based on paths touched in the last N executions

• For each Camus run, create a file listing all the directories it touched.
• LAD "runs" type check can pick up these files for the last N runs and check all the dirs.
• This feature can be activated in a separate PR by changing the Azkaban task.

[dterror-zz]

c.c @angelini @kmtaylor-github

[olessia]

@dterror It's not 🎩 in staging yet, the PR is onto staging, but I've 🎩 locally.