Periodically update schemas from mozilla-pipeline-schemas

[jklukas]

Currently, we fetch mozilla-pipeline-schemas from GitHub at ingestion-beam build time and include the content in the jar. This means we can only get schema updates by rebuilding, draining the existing Dataflow job, and instantiating a new job with the new code.

We could try to spin up a periodic task (every 5 minutes, perhaps?) to get the latest content from GitHub and update the collection of schemas. Perhaps this could be expressed nicely as a side input?

[jklukas]

I see two main options we could take to run a task periodically:

• Spin up our own Java Executor to handle running an additional thread
• Use a Beam @OnTimer function that will be called periodically based on processing time

I'm inclined to use the @OnTimer method, though documentation is a bit scarce about how it's actually executed. The Beam docs state "each instance of your function object is accessed by a single thread at a time on a worker instance, unless you explicitly create your own threads", so I'm inferring that the @OnTimer method will block processing of additional records while it runs. This is probably fine, but we'll want to put a reasonably small timeout on our call to GitHub so that we don't accidentally clog the pipeline. Fail fast, log the error, and continue without updating the schemas.

[jklukas]

I'd like to use the same script (bin/update-schema) to pull down mozilla-pipeline-schemas into resources at build time and to make files available for the @OnTimer method.

We'll maintain a mozillaPipelineSchemasTimestamp that tracks when the repo was last modified. The @OnTimer method will make a call to the GiHub API to see whether the repo has a newer last modified timestamp (or perhaps for the dev branch specifically). If it already has the newest, it end with no changes. If it gets a newer timestamp, then it runs the bin/update-schemas script and loads the schemas from those files.

If any of the calls to GitHub fail, we log and move on so we don't hold up processing of messages. We'll try again when the timer next fires.

[jklukas]

Rather than maintaining a timestamp, perhaps it's better to maintain the last commit id. We can use the get branch endpoint and extract commit.shafrom this url:

https://api.github.com/repos/mozilla-services/mozilla-pipeline-schemas/branches/dev

[jklukas]

Wrote up a bug with design thoughts for a solution where we publish schema changes to PubSub instead: https://bugzilla.mozilla.org/show_bug.cgi?id=1502057

[jklukas]

Based on discussion in https://github.com/mozilla/gcp-ingestion/issues/145, we're simply going to expect to issue a rebuild and update to get new schemas.