New ETL channel (or namespace) to track dependencies of external repositories

[pabloarosado]

Context & motivation

• Currently, we track dependencies of things that are used in Grapher charts; we will get an error if we delete a dependency
• But, we don't track all downstream uses:
  • Explorers
  • External repositories

We would love to get an error early, rather than breaking something.

Proposal

Inspired by this thread, I thought we could have a new ETL channel (or, as an easier alternative, a new namespace), e.g. repository, which hosts latest versions of steps that are loaded by external repositories, like poverty-data, covid-19-data, co2-data, energy-data, and also owid-grapher.

With this, if anyone accidentally deletes or archives a dependency of these repository steps, we will get an error by VersionTracker. We could also show these dependencies in the ETL Dashboard (and treat them as always active, even if they don't have charts).

Specifically, Sophia said it would be useful for her to have latest versions of grapher/worldbank_wdi/2023-05-29/wdi/wdi#ny_gdp_pcap_pp_kd and for grapher/demography/2023-03-31/population/population#population_historical. They could be easily created in the new channel or namespace.

Technical notes

• Creating a new channel would be ideal, but it requires much more additional work. So, a much cheaper alternative would be to create a new namespace.

Scope

• Decide on a special reserved namespace
• Make a new dataset for Sophia (for example), using a special namespace
• Modify the version tracker to always consider that namespace used

[danyx23]

Another option for how to do this would be to add an alias system that we use to keep what are probably basically our MiMs in an easily accessible namespace.

I.e. the aliases would not be an ETL channel or something like this but a new kind of URI scheme that is mapped to full ETL paths. Maybe something to discuss in our offsite.

[sophiamersmann]

Grapher has the following indicator IDs currently hardcoded:

• Continents: 123 (probably ok)
• Population: 597929 (used in the admin as default size dimension for scatter plots)
• More population variables: 525709, 525711, 597929, 597930 (used to exclude the population indicator from being displayed in the sources modal in certain cases)

In the future, the entity selector has these two variables hardcoded (not yet merged, but implemented in https://github.com/owid/owid-grapher/pull/3466)

• Population: 597930 (used to sort entity names)
• GDP per capita: 735665 (used to sort entity names)

[pabloarosado]

Now that the external channel is created, I've created https://github.com/owid/etl/issues/2609 to add all existing dependencies (including the ones listed by Sophia above).