Consolidate data source metadata using a Pydantic model

zaneselvans commented 2 years ago

Where is the Data Source metadata now

pudl/metadata/sources.py (where we're trying to consolidate it)
pudl-zenodo-storage/zs/metadata.py
Some subcomponents are defined in pudl/metadata/constants.py (licenses, contributors, etc.)
pudl/workspace/datastore.py::ZenodoFetcher (DOIs for Zenodo archives)
pudl/settings.py (working partitions)
pudl/metdata/resources.py (working tables -- to be extracted dynamically based on etl_group)

Classes / Data Models:

Implemented using Pydantic.

`EtlSettings`

Needs to ensure that the ETL parameters it's gotten are valid.
This means validating requested data partitions (years, states, etc).
Needs to contain a reference to the relevant DataSource classes to do that.
Should it enumerate the tables which are to be produced?
Should the tables to be produced be user defined, or automatically generated based on the data sources which are being processed?
The set of all tables enumerated in / passed to / generated by the EtlSettings object will define what tables are ultimately produced and which ETL functions are run.
Each DataSource has its own specific EtlSettings class defined. There's another class that contains several of these individual EtlSettings objects and defines a whole PUDL ETL run.

`DataSource`

Stores almost all metadata related to the data sources.
Can output this metadata in various formats: RST for docs, YAML for Datasette.
Contains a dictionary of working_partitions describing the chunks of data that can be processed.
Can output the list of available (valid) tables within the DataSource by looking up which Resources have the DataSource.name in their etl_group parameter.

`Resource` & `Package`

Enumerates the tables and tracks which etl_group they belong to.
Each etl_group value corresponds to a different DataSource definition.
Given a set of etl_group / DataSource IDs, the corresponding Resource definitions can be identified, and used to generate a Package. That Package could generate the list of all valid tables across all of the specified DataSources... if that's useful. But it would be simpler if we could keep the validation of tables specific to the individual DataSources and the DataSource specific EtlSettings classes.

Questions

EIA-860m

How does something like the eia860m fit into all this. Is it a standalone DataSource, or is it bolted onto the side of the eia860?
In some contexts it has to be separate -- e.g. Zenodo Archives, specifying its working partition(s).
In others right now it's being treated as part of eia860 -- extracted and concatenated together, if eia860m==True.
This is a bit messy. It seems like the more atomic treatment of data sources up through Extract (which seems to be how Airbyte does things) would keep the API cleaner, and then various sources can be combined a bit later on (early on in Transform).
Airbyte takes this a step further and seems to treat every data partition as an individual source, which seems like... a lot. But would allow a more uniform API that can deal with things file formats (and thus extraction processes) changing over time, or the need to concatenate monthly / year-to-date provisional data releases onto the bulk data, when that provisional data has a different kind of partitioning or structure.
For the moment, is there a simple (if potentially hacky) way to avoid disentangling eia860m from eia860 but still create clean metadata structures that can be used both in the Zenodo archives and in the EtlSettings context?

Implicit Tables

How do we handle tables that don't belong to one of the "real" DataSource classes? The static tables, the glue tables? Do we always load them? Are they always valid?
Are the static / glue tables that are part of an ETL run derived from this list of other requested data sources / tables? If so, then the set of valid tables ceases to be a property of an individual DataSource, and becomes a property of the collection of all DataSources.
This can be avoided if users aren't allowed to specify particular tables, or if some subset of the tables are dealt with implicitly, as indicated by the requested tables. E.g. if you're processing ferc1 then you automatically get the (static) ferc_accounts table. If you're processing ferc1 and eia923 then you automatically get the eia_ferc1 glue tables.
The simplest option seems to be to limit the EtlSettings to specifying which data partitions and DataSources are being processed, and always assume all available tables within the data source will be provided, generating that list of tables internally.
For development purposes we should be able to override this at some point in the process and specify one or more particular tables rather than doing everything every time, but this doesn't need to be exposed as part of the API.

Related Issues / PRs

1409
1410
1419
1424
1425

bendnorman commented 2 years ago

I think we can create an EIA 860m DataSource definition and retain the settings validation that is happening in Eia860Settings.check_860m_date(). It could look something like this (I don't know what DataSource model will look like):

class Eia860Settings(GenericDatasetSettings):
    working_partitions: ClassVar = DataSource.from_source("eia860").working_partitions
    eia860m_date: ClassVar[str] = DataSource.from_source("eia860m").working_partitions
    working_tables: ClassVar = DataSource.from_source("eia860").working_tables

    years: List[int] = working_partitions["years"]
    tables: List[str] = working_tables
    eia860m: bool = False

    @validator("eia860m")
    def check_860m_date(cls, eia860m: bool) -> bool:
           ...

zaneselvans commented 2 years ago

This consolidation is done, with the exception of the Zenodo archiver which has a bunch of its own (not mostly duplicative) metadata, see #1419 and this draft PR

catalyst-cooperative / pudl