Create a GTFS RT daily validation notices table

[machow]

In a gtfs_rt_transform DAG, create a KubernetesPodOperator task that does the following:

To setup for validation:

• download the feed's protobuf files into a temporary directory, in the folder {tmp_dir}/rt
• download the feed's gtfs schedule data and zip it
• name each file {filename}_{date}, so we can tell the validator to sort by file name

Running and extracting validation results:

• Run validator via python subprocess command
• Save any files ending in .results.json to rt-processed/validation/{date}_{itp id}_{url number}.json

Loading results:

• We likely need to explicitly write out the schema of the JSON
• Using an external table that globs on the files in rt-processed

[machow]

Current DDL statement for loading results for vehicle positions...

CREATE OR REPLACE TABLE gtfs_rt.validation_report (
    errorMessage STRUCT<
      messageId INT64,
      gtfsRTFeedIterationModel STRING,
      validationRule STRUCT<
        errorId STRING,
        severity STRING,
        title STRING,
        errorDescription STRING,
        occurrenceSuffix STRING
      >,
      errorDetails STRING
    >,
    occurenceList ARRAY<
      STRUCT<
        occurenceId INT64,
        messageLogModel STRING,
        prefix STRING
      >
    >
)
OPTIONS (
)

TODO:

• merge basic validation -> gcs bucket script, along with a days worth of validations
• create issue for adding to airflow DAG, stub out corner cases (e.g. no gtfs schedule data for the day)
• create issue for DDL statement to load (may require some trial and error, generating validation results)

[machow]

proof of concept merged here: https://github.com/cal-itp/data-infra/pull/757

would need to compress results to complete. Note that the bulk of the size comes from rule explanations being embedded in the results: https://github.com/cal-itp/data-infra/issues/786#issuecomment-983945175

requires: #786