hyperlane-xyz / hyperlane-monorepo

The home for Hyperlane core contracts, sdk packages, and other infrastructure
https://hyperlane.xyz
Other
295 stars 312 forks source link

Adding a new chain to the scraper should not require manual psql tomfoolery #3324

Open tkporter opened 6 months ago

tkporter commented 6 months ago

Problem

Atm, adding a new domain to the scraper involves:

  1. manually inserting the new domain into the domains db using pqsl, which is error prone and time consuming
  2. For consistency, adding the domain to the scraper migration Rust file: m20230309_000001_create_table_domain.rs. We never actually run this migration in prod!

Solution

  1. The scraper migrations should set the domains for the same set of chains specified in the agent configs (which will ultimately come from the registry). This should be done automatically and consistency should be ensured at CI time if necessary
  2. When deploying the scraper, the domains table is updated. We should prefer to run domains table migrations as a way to update the table when deploying the scraper, as that sounds like the intention behind migrations - but if this proves complex, we can fallback to automated psql stuff
tkporter commented 2 months ago

after chatting with @paulbalaji, agreed that this isn't a super cumbersome part of the deployment process and we shouldn't prio this

Mo-Hussain commented 1 month ago

I wonder if our rust ORM supports generating migration files? We could use that to add the chains to the scraper with one migration file per deployment. We would have directional history of all migrations and the ORM tooling should help roll back a specific migration if needed