Track / load remaining files included in GTFS schedule spec

[machow]

I'm not sure if these were missed when I was loading 6 months ago, or were recently added to the docs, but there are 3 files we currently don't load into the warehouse.

• attributions.txt
• pathways.txt
• translations.txt

It looks like relatively few feeds use these files right now (but analyzing e.g. pathways will likely be increasingly important).

In order to load these, we need to...

• add them to the gtfs_schedule_history DAG (see e.g. the agencies task there; gtfs schedule spec for column types)
• add them to the calitp_included_gtfs_tables task
• delete our gtfs_schedule_type2 tables
  • gtfs_schedule_type2.{table_name}
• delete gs://gtfs-schedule-data/schedule/processed
• re-run the gtfs_loader DAG across history (only run a couple days at a time, to not choke airflow)
• re-run gtfs_schedule_history2.merge_updates

Completing this would resolve https://github.com/cal-itp/data-infra/issues/180

[machow]

Feel free to move all the file tracking tasks into their own dag (that depends on past)

helpful notes from Chris--there are 4 jobs being done here (should also have 4 dags):

1. put into cloud storage
2. understand files (and when they change)
3. load feeds with changed files into external tables
4. understand what has changed in each table (e.g. build SCD tables)

[machow]

@holly-g this would have gone to Chris C, so will have to wait until Jarvus brings in a data engineer (or I pick it up)

[machow]

TO document:

• data transfer job for schedule data runs once a week to dev: https://console.cloud.google.com/transfer/cloud/jobs/transferJobs%2F10464386066736031383/runs?authuser=0&project=cal-itp-data-infra
  • can run gtfs_loader directly as a result, without need to download
• generate_provider task needs to be fixed on dev #247