Pre-release analytics for 5.0

MobilityData / gtfs-validator

Canonical GTFS Validator project for schedule (static) files.

https://gtfs-validator.mobilitydata.org/

Apache License 2.0

266 stars 100 forks source link

Pre-release analytics for 5.0 #1703

Closed emmambd closed 3 months ago

emmambd commented 4 months ago

Before publishing the release, we need to set up the pre-release and run analytics on the impact on rules as a last check on performance impacts and feed affects.

Should resemble last release's report. plus a check on feeds that failed due to out of memory issues from this list in https://github.com/MobilityData/gtfs-validator/issues/1611.

Should also include

the number of components per feed and the list of feeds that has them, like https://airtable.com/app98IfnmNPV1EUWw/pagBlhmmPCJ7uKrx7
The total number and list of all notices, not just the difference. Should include list of feeds affected.

Version of this done in : https://drive.google.com/drive/folders/1FEemgqhamtY_-ZBuvUl_LfaFiftR04Fb

cka-y commented 3 months ago

Tasks:

[ ] Document process in Notion
[ ] Run pre-release analytics

emmambd commented 3 months ago

@jcpitre One thing I forgot this morning - this task is actually bigger because we also want to do our organization-wide analytics as part of the pre-release analytics. Should include

Not just the difference in notices between releases, but the overall list of notices triggered.
Number of components per feed, with the feed list.

We could re-task this tomorrow if needed.

jcpitre commented 3 months ago

The analytics is done via this repo: gtfs-datasets-metrics-poc Here is the documentation we have for going version from 4.1.0 to 4.2.0: Readme.md

Instead of using Notion, I will continue documenting in a README.md file.

jcpitre commented 3 months ago

Here is the sheet containing the results: https://docs.google.com/spreadsheets/d/10zGOwbqbpdc-rhZ1UIDAqC8N9dnjh5fnViF1rBYOEwg

This does not include what was added here: https://github.com/MobilityData/gtfs-validator/issues/1703#issuecomment-1988949747

Some feeds were not properly processes by version 5.0.0:

au-victoria-public-transport-victoria-ptv-gtfs-657
be-bruxelles-capitale-societe-des-transports-intercommunaux-de-bruxellesmaatschappij-voor-het-intercommunaal-vervoer-te-brussel-stibmivb-gtfs-1088
fr-bretagne-societe-deconomie-mixte-des-transports-de-lagglomeration-nantaise-tan-gtfs-1025
gb-unknown-chiltern-railways-gtfs-1311
us-california-san-francisco-municipal-transportation-agency-sfmta-gtfs-50
us-washington-metro-transit-gtfs-1080

Checked the logs for the last one and it seems to be a memory issue.

jcpitre commented 3 months ago

Documentation updated. See gtfs-datasets-metrics-poc repo README.md