-
It'd be nice if a search for FBI brought back results for Federal Bureau of Investigations, but it's a bit weird if a search for HSC brings back results for the less-popular Homeland Security Council.…
-
Working on like 5 PRs simultaneously today, I had to dig into why our integration tests have suddenly gotten sooo slooow.
Running the same command as the tests use:
```
tox -e integration -- --…
-
In order to publish raw XBRL derived databases, it must be able to ingest datapackage descriptors output by `ferc-xbrl-extractor`. These can then be converted to metadata structures that can be used w…
-
To be able to publish extracted XBRL data from all FERC Forms, the `ferc1_to_sqlite` script needs to be made more generic (`ferc_to_sqlite`), so PUDL can create all of the raw databases.
## Tasks
…
-
Our current dagster refactor plan doesn't describe how to persist interim tables and how to apply dagster concepts to the output and analysis layers.
## Data Persistence questions
- Is it possibl…
-
When attempting to run the `ferc1_to_sqlite` script processing all tables for years 2020 and 2021 (so, 1 year each from the two FERC data sources) I get a bunch of errors about files not existing (but…
-
### `iso_rto_code` and other variables missing from `plants_eia860` table in `pudl-v0.6.0-2022-03-12` release
Below, the list of variables in the `plants_eia860` table from the most recent `pudl-v0.6…
-
Some sites have error pages that don’t respond with actual error status codes (i.e. they have a `200` status code instead of `>= 400`). For example: https://www.eia.gov/tools/models/datatools.cfm
S…
-
Trying to get the nightly builds to pass for #1681 / #2077 I discovered that the `fuel_ferc1` outputs were really different locally vs. in the nightly builds. The expected row count validation is fail…
-
Convert EPA CEMS ETL to yield software defined assets. There are couple ways we can use dagster concepts here:
- [Dynamic graphs](https://docs.dagster.io/concepts/ops-jobs-graphs/dynamic-graphs): Have…