search

ethereum-optimism / superchain-registry

An index of chains which serves as the source of truth for who’s in the Superchain Ecosystem
MIT License
84 stars 99 forks source link

Validate all chains on PRs which modify the validation test code / expectations #689

Open alfonso-op opened 1 week ago

alfonso-op commented 1 week ago

We need to strengthen the validations that run on the superchain-registry to catch breaking changes such that the validate all chains CI job is triggered for all PR changes with the one exception being when a new chain is added.

See George's comment below for more details.

geoknee commented 1 week ago

The problem is rather that we aren't validating all chains in all the cases that we should be. We are being too eager to skip validation on all chains. On PRs we already skip validating any chain which didn't have its TOML files changed. But if the PR changes the standard config TOML files, or the Go files which compose the validation suite itself, this could change the validation status of any chain in the repo.

I think the rule should be that if you only add a chain, that is add the TOML files for a new chain, and don't touch anything else, then only your chain gets validated. Otherwise, all chains get validated. This preserves the nice property we already enjoy, that if a PR only adds a new chain, it won't get blocked on validation failures from other chains.