Adjust PUDL ETL to allow different FERC tables to contain different years

catalyst-cooperative / pudl

The Public Utility Data Liberation Project provides analysis-ready energy system data to climate advocates, researchers, policymakers, and journalists.

https://catalyst.coop/pudl

MIT License

456 stars 106 forks source link

Adjust PUDL ETL to allow different FERC tables to contain different years #463

Closed zaneselvans closed 4 years ago

zaneselvans commented 4 years ago

Currently we require all the FERC tables to include data from all the same years in the PUDL DB, but it's certainly possible, and might be useful, to allow different years in different tables, since it takes time to do the cleaning.

zaneselvans commented 4 years ago

GIven that the row-mapping process for all the old tables works well, it seems like we can safely do all-or-nothing processing of each FERC table, without adding this complexity.