Refactor VCE RARE asset checks

catalyst-cooperative / pudl

The Public Utility Data Liberation Project provides analysis-ready energy system data to climate advocates, researchers, policymakers, and journalists.

https://catalyst.coop/pudl

MIT License

489 stars 112 forks source link

Refactor VCE RARE asset checks #3926

Open jdangerx opened 1 month ago

jdangerx commented 1 month ago

### Tasks
- [x] Review asset checks and see if their memory footprint can be reduced
- [x] Remove any asset checks that seem low value and resource intensive
- [x] Accumulate an error report and return it if any of the piecemeal checks fail
- [ ] Fix the expected row-count asset check so that it can pass in the fast ETL

zaneselvans commented 1 week ago

Most of these issues were addressed in #3959 but the row-count check still only passes if all data is processed, meaning the fast_etl will always fail if asset checks are enabled, which they should be.