catalyst-cooperative / pudl-scrapers

Scrapers used to acquire snapshots of raw data inputs for versioned archiving and replicable analysis.
MIT License
3 stars 3 forks source link

Xbrl integration #56

Closed zschira closed 2 years ago

zschira commented 2 years ago

This PR adds support for scraping FERC XBRL data from their RSS feed. The feed is segmented into their latest feed with the most recent 650 filings in it, and month specific feeds with all older filings. The ferc_xbrl.py script indexes all available feeds, and downloads/archives filings for all years of data. It also archives XBRL taxonomies along with the actual filings.

codecov[bot] commented 2 years ago

Codecov Report

Base: 63.4% // Head: 61.4% // Decreases project coverage by -1.9% :warning:

Coverage data is based on head (f832cd1) compared to base (237f6bb). Patch coverage: 53.3% of modified lines in pull request are covered.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #56 +/- ## ======================================= - Coverage 63.4% 61.4% -2.0% ======================================= Files 18 19 +1 Lines 637 801 +164 ======================================= + Hits 404 492 +88 - Misses 233 309 +76 ``` | [Impacted Files](https://codecov.io/gh/catalyst-cooperative/pudl-scrapers/pull/56?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=catalyst-cooperative) | Coverage Δ | | |---|---|---| | [src/pudl\_scrapers/spiders/ferc60.py](https://codecov.io/gh/catalyst-cooperative/pudl-scrapers/pull/56/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=catalyst-cooperative#diff-c3JjL3B1ZGxfc2NyYXBlcnMvc3BpZGVycy9mZXJjNjAucHk=) | `56.5% <0.0%> (ø)` | | | [src/pudl\_scrapers/helpers.py](https://codecov.io/gh/catalyst-cooperative/pudl-scrapers/pull/56/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=catalyst-cooperative#diff-c3JjL3B1ZGxfc2NyYXBlcnMvaGVscGVycy5weQ==) | `42.1% <11.1%> (-27.9%)` | :arrow_down: | | [src/pudl\_scrapers/bin/ferc\_xbrl.py](https://codecov.io/gh/catalyst-cooperative/pudl-scrapers/pull/56/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=catalyst-cooperative#diff-c3JjL3B1ZGxfc2NyYXBlcnMvYmluL2ZlcmNfeGJybC5weQ==) | `56.1% <56.1%> (ø)` | | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=catalyst-cooperative). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=catalyst-cooperative)

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.