Proposal: Modify Validator to allow for custom validation

[isabelle-dr]

This is an issue for planning the introduction of custom validation, It will be updated as we plan and develop the work. Please share with us below your needs and ideas! 💡 If you'd like to contribute to this release, please get in touch with isabelle at isabelle@mobilitydata.org🙏

Goal for the release 🎯

In addition to the canonical set of rules (based on the official GTFS specification), we want to allow introducing additional rules and create custom validation sets in this tool. This aims at responding to the following user needs:

• as a consumer, I can create a custom validation set for my own criteria:
  • I can modify the schema for a GTFS file
  • I can upgrade or downgrade canonical rules
  • I can silence certain canonical rules
• as a producer, I can validate a GTFS dataset against a custom validation set:
  • a GTFS extension that isn't adopted yet
  • a specific consumer's set of criteria
• as a producer, I can validate my feed against several validation sets at once
• as a developer, I can add rules that aren't based on the GTFS specification or the MobilityData Best Practices

Considerations

• we need to allow for different schemas in this feature
• all the "non-canonical" rules need to be publicly available for everyone to use
• if we validate against several validation sets at once, we need to have a way to not run the same code twice
• we need to think of a governance process for the "non-canonical" rules to ensure the stability of the custom validation sets
• we need to think of a way to flag the "non-canonical" rules that are used widely by the community and "canonalize" them
• we need a process to make sure the custom validation sets aren't getting further from the official specification as the canonical validation set evolves

Materials

The Product Requirement Document: https://bit.ly/gtfs-validator-3-1-0. It is a working document, feel free to leave comments directly in it.

This feature has been discussed/could help in the following issues:

• https://github.com/MobilityData/gtfs-validator/pull/1039
• https://github.com/MobilityData/gtfs-validator/issues/927
• https://github.com/MobilityData/gtfs-validator/issues/918
• https://github.com/MobilityData/gtfs-validator/issues/882
• https://github.com/MobilityData/gtfs-validator/issues/471
• https://github.com/MobilityData/gtfs-validator/issues/215

[isabelle-dr]

Here is a summary of what was discussed during our first community meeting to confirm the needs and look at how to change the validator architecture for the 3.2 release. Thank you for your participation!

Candidates for the Custom Validation feature we have so far. This list will be updated as other interests arise.

Custom validation for	What's different?	Interested in contributing
GTFS-Fares v2	New files, extended and modified existing files, new custom rules to add	@lionel-nj
GTFS-Flex v2	New files, extended and modified existing files, new custom rules to add	@lionel-nj
OpenTripPlanner's 1.x criteria	To be defined	@markstos
Cal-ITP's guidelines	GTFS-Fares v2 is included, shapes.txt and pathways.txt is required, stops.wheelchair_boarding, trips.wheelchair_accessible and stops.tts_stop_name fields are required, some severities of canonical rules are different	@evansiroky
Google's extended route types (discussed here)

[bdferris-v2]

As an update, my colleague @asvechnikov2 has put together a proposal doc that describes potential high-level functionality in support of custom validation:

https://bit.ly/gtfs-validator-custom-validation

Happy to have feedback (heads-up @KClough @themightychris).