search

catalyst-cooperative / pudl

The Public Utility Data Liberation Project provides analysis-ready energy system data to climate advocates, researchers, policymakers, and journalists.
https://catalyst.coop/pudl
MIT License
468 stars 107 forks source link

Prototype integration of FERC's new XBRL format #1530

Closed zschira closed 2 years ago

zschira commented 2 years ago

Background

For background, see 1321 and 1440. This epic will track the integration of this new XBRL format.

Forward work

Arelle seems to be the best tool for working with XBRL that is not proprietary, and does not require a subscription fee. Arelle's documentation is lacking, but it seems to be widely used in the XBRL world, and it's still being actively maintained.

To make integrating new filings with historical data as seamless as possible, we will attempt to develop a process convert XBRL data to a SQLite db the same schema as the db produced by the ferc1_to_sqlite script. While the tables in the new XBRL taxonomy do not correspond to the tables in the old Foxpro db, they do follow the pages of the raw form, so we should be able to create this mapping to perform the conversion. As a first step, we will attempt to recreate the tables in the raw FERC db required for the (FERC PUDL tables](https://catalystcoop-pudl.readthedocs.io/en/dev/data_sources/ferc1.html#pudl-data-tables).

zschira commented 2 years ago

Closing, XBRL work is being tracked in #1568 and #1574.