Proposal: Modify Validator to allow for custom validation

isabelle-dr commented 3 years ago

This is an issue for planning the introduction of custom validation, It will be updated as we plan and develop the work. Please share with us below your needs and ideas! 💡 If you'd like to contribute to this release, please get in touch with isabelle at isabelle@mobilitydata.org🙏

Goal for the release 🎯

In addition to the canonical set of rules (based on the official GTFS specification), we want to allow introducing additional rules and create custom validation sets in this tool. This aims at responding to the following user needs:

as a consumer, I can create a custom validation set for my own criteria:
- I can modify the schema for a GTFS file
- I can upgrade or downgrade canonical rules
- I can silence certain canonical rules
as a producer, I can validate a GTFS dataset against a custom validation set:
- a GTFS extension that isn't adopted yet
- a specific consumer's set of criteria
as a producer, I can validate my feed against several validation sets at once
as a developer, I can add rules that aren't based on the GTFS specification or the MobilityData Best Practices

Considerations

we need to allow for different schemas in this feature
all the "non-canonical" rules need to be publicly available for everyone to use
if we validate against several validation sets at once, we need to have a way to not run the same code twice
we need to think of a governance process for the "non-canonical" rules to ensure the stability of the custom validation sets
we need to think of a way to flag the "non-canonical" rules that are used widely by the community and "canonalize" them
we need a process to make sure the custom validation sets aren't getting further from the official specification as the canonical validation set evolves

Materials

The Product Requirement Document: https://bit.ly/gtfs-validator-3-1-0. It is a working document, feel free to leave comments directly in it.

This feature has been discussed/could help in the following issues:

isabelle-dr commented 2 years ago

Here is a summary of what was discussed during our first community meeting to confirm the needs and look at how to change the validator architecture for the 3.2 release. Thank you for your participation!

Candidates for the Custom Validation feature we have so far. This list will be updated as other interests arise.

Custom validation for	What's different?	Interested in contributing
GTFS-Fares v2	New files, extended and modified existing files, new custom rules to add	@lionel-nj
GTFS-Flex v2	New files, extended and modified existing files, new custom rules to add	@lionel-nj
OpenTripPlanner's 1.x criteria	To be defined	@markstos
Cal-ITP's guidelines	GTFS-Fares v2 is included, `shapes.txt` and `pathways.txt` is required, `stops.wheelchair_boarding`, `trips.wheelchair_accessible` and `stops.tts_stop_name` fields are required, some severities of canonical rules are different	@evansiroky
Google's extended route types (discussed here)

bdferris-v2 commented 2 years ago

As an update, my colleague @asvechnikov2 has put together a proposal doc that describes potential high-level functionality in support of custom validation:

https://bit.ly/gtfs-validator-custom-validation

Happy to have feedback (heads-up @KClough @themightychris).

MobilityData / gtfs-validator