catalyst-cooperative / ferc-xbrl-extractor

A tool for converting FERC filings published in XBRL into SQLite databases
MIT License
11 stars 1 forks source link

Add table/instance filtering to speed up debugging process. #148

Closed jdangerx closed 1 year ago

jdangerx commented 1 year ago

This makes it so you can do something like:

> xbrl_extract $PUDL_INPUT/2021/ferc1-xbrl-2021.zip foo.sqlite --instance-pattern "PJM" --requested-tables "electric_operations_and_maintenance_expenses_320_duration" "identification_001_duration" --metadata-path foo-md.json

And take a look-see at what the extractor is putting out, as well as stuff breakpoints in places without having to add conditional breakpoints like "if the table is blah and the instance is blah, then breakpoint()".

Some timings just for fun:

> time xbrl_extract $PUDL_INPUT/2021/ferc1-xbrl-2021.zip foo.sqlite --instance-pattern "PJM" --requested-tables "electric_operations_and_maintenance_expenses_320_duration" "identification_001_duration" --metadata-path foo-md.json
2023-09-27 13:01:52 [    INFO] catalystcoop.ferc_xbrl_extractor.xbrl:265 Parsing taxonomy from https://eCollection.ferc.gov/taxonomy/form1/2022-01-01/form/form1/form-1_2022-01-01.xsd
2023-09-27 13:01:56 [    INFO] catalystcoop.ferc_xbrl_extractor.xbrl:149 Finished batch 1/1

________________________________________________________
Executed in    5.89 secs    fish           external
   usr time    7.20 secs    0.31 millis    7.20 secs
   sys time    2.07 secs    1.75 millis    2.07 secs
codecov[bot] commented 1 year ago

Codecov Report

All modified lines are covered by tests :white_check_mark:

Comparison is base (61525e5) 93.09% compared to head (2ca562e) 93.29%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #148 +/- ## ========================================== + Coverage 93.09% 93.29% +0.20% ========================================== Files 8 8 Lines 594 597 +3 ========================================== + Hits 553 557 +4 + Misses 41 40 -1 ``` | [Files](https://app.codecov.io/gh/catalyst-cooperative/ferc-xbrl-extractor/pull/148?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=catalyst-cooperative) | Coverage Δ | | |---|---|---| | [src/ferc\_xbrl\_extractor/cli.py](https://app.codecov.io/gh/catalyst-cooperative/ferc-xbrl-extractor/pull/148?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=catalyst-cooperative#diff-c3JjL2ZlcmNfeGJybF9leHRyYWN0b3IvY2xpLnB5) | `94.54% <100.00%> (+0.20%)` | :arrow_up: | | [src/ferc\_xbrl\_extractor/datapackage.py](https://app.codecov.io/gh/catalyst-cooperative/ferc-xbrl-extractor/pull/148?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=catalyst-cooperative#diff-c3JjL2ZlcmNfeGJybF9leHRyYWN0b3IvZGF0YXBhY2thZ2UucHk=) | `99.35% <100.00%> (+0.64%)` | :arrow_up: | | [src/ferc\_xbrl\_extractor/xbrl.py](https://app.codecov.io/gh/catalyst-cooperative/ferc-xbrl-extractor/pull/148?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=catalyst-cooperative#diff-c3JjL2ZlcmNfeGJybF9leHRyYWN0b3IveGJybC5weQ==) | `91.56% <100.00%> (+0.10%)` | :arrow_up: |

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

jdangerx commented 1 year ago

Sort of seems like I probably broke some of this behavior that used to exist, since there was some old logic to hook into. But I guess it's fixed now! I'm gonna add a test to make codecov happy and then merge.