Improve the way we find and load a dataset/table/indicator

pabloarosado commented 6 months ago

Summary

We need a better way to quickly access an ETL dataset/table/indicator from within Python.

Goal: Improve our catalog.find* function(s) (or create other another function).

Minimum requirements: That function makes it easier to find and load a specific dataset or table.
Better: That function would lets you search for indicators.
Optimal: That function lets you search for a dataset/table/indicator, based on exact or fuzzy matches, or semantic similarity of ETL paths as well as metadata.

Current workflow to search and load data

Currently, every time I need to access an ETL table (for example, to do a quick check) I need to go through one or more of the following steps:

Scrolling through the ETL dag files or snapshots/step files (which is quite inconvenient).
Going to the dataset admin.
Going to our home page to use the search bar. This is nice, but once you find the chart, then you have to click on "edit", then select the indicator, and copy the ETL path.
The indicator search is powerful, although in practice I have not found it particularly useful so far.

Then, once you know which dataset/table/indicator you need, you have to write quite a lot of code to load it, e.g.:

from etl.paths import DATA_DIR
from owid.catalog import Dataset
ds = Dataset(DATA_DIR / "garden/energy/2023-12-12/owid_energy")
tb = ds["owid_energy"].reset_index()

Note that here there are multiple things that need to be manually given (the namespace, version, short name of the dataset, and short name of the table). Then, if I need a specific indicator, I'd need to do all this, plus then something like [c for c in tb.columns if "solar" in c] and then figure out which indicator I need.

Overall, it takes a few minutes to "quickly" access some data, which is not ideal.

Issues with `catalog.find*`

We currently have a catalog.find (and other related functions) that are supposed to help with this, but this function has issues. From the top of my head:

It finds tables, not datasets.
It doesn't let you load snapshots.
It's not very transparent in whether it loads from the remote catalog or local ETL.
It has problems (don't remember exactly what) when loading tables with similar short names (e.g. "population" and "population_density").
Also, a user gave us feedback on this function: https://github.com/owid/etl/issues/2616

So, in practice, catalog.find is not useful. Therefore, we need a better function to quickly find and access our data. Superusers would also benefit from these improvements.

larsyencken commented 4 months ago

Some background on why find() is not as great as it could be:

find() is defined on the owid.catalog module, but is really RemoteCatalog.find()
The RemoteCatalog can have methods on it like __getitem__ that you can't define on the module

In principal, we can make whatever we like here, so we should just get the interface right and then solve the (trivial) code to make it work smoothly.

stale[bot] commented 2 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Marigold commented 2 months ago

I'd keep this open for a bit longer, although it doesn't seem to be a priority at the moment.

Marigold commented 2 months ago

We still want this, though it might be a part of a larger API redesign.

stale[bot] commented 1 week ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

owid / etl