catalyst-cooperative / pudl

The Public Utility Data Liberation Project provides analysis-ready energy system data to climate advocates, researchers, policymakers, and journalists.
https://catalyst.coop/pudl
MIT License
456 stars 106 forks source link

Convert EIA ETL to assets #2083

Closed bendnorman closed 1 year ago

bendnorman commented 1 year ago

Functions that produce valuable interim tables will be converted software defined assets. These steps include:

eia_assets = load_assets_from_modules([eia860, eia923])

@multi_asset(
    ins={
        asset_key.to_python_identifier(): AssetIn()
        for eia_asset in eia_assets
        for asset_key in eia_asset.asset_keys
    }
)
def eia_transform(**eia_transformed_dfs):
    ...
bendnorman commented 1 year ago

The EIA ETL has been converted to use dagster assets. This work hasn't been merged into dev yet but is completed on the dagster-asset-etl branch.