Due to a swrs schema change for the 2022 reporting year, emissions from EIO facitlities are not being parsed into the Emission table & therefore cannot be queried in metabase.
2021 reports had an xml structure of:
\
-- \
---- \
------ \
The 2022 reports for EIOs now have a structure of:
\
-- \
---- \
------ \
------ \
This structure change is why the emissions from EIOs are not being parsed. The ETL looks for the \ tag when parsing the XML reports into rows in the Emission table. It does not know about the new \ tag.
CAS has expressed a desire to have the emissions from the new schema parsed in the same place (into the emission table rather than separated).
How to fix it:
We did something similar last year when adding emissions from R3 reports
[x] add a new materialized view to parse the \ tag with xpath
[x] update the load() function to include the new materialized view in the set of views to parse
[x] modify the Emission transform function to add the data from the new materialized view into the emission table (EIOEmissionAmount should go into the Quantity column)
[x] add pgTap tests to validate the functionality of the new materialized view & the load functions
Probability (how likely the bug is to happen, scored from 1-5): 5
Effect (how bad the bug is when it does happen, scored from 1-5): 3
Steps to reproduce the behaviour:
Query for any emissions from an EIO facility in metabase for 2022 and there will be no results
Describe the Bug:
Due to a swrs schema change for the 2022 reporting year, emissions from EIO facitlities are not being parsed into the Emission table & therefore cannot be queried in metabase.
2021 reports had an xml structure of: \
-- \
---- \
------ \
The 2022 reports for EIOs now have a structure of: \
-- \
---- \
------ \
------ \
This structure change is why the emissions from EIOs are not being parsed. The ETL looks for the \ tag when parsing the XML reports into rows in the Emission table. It does not know about the new \ tag.
CAS has expressed a desire to have the emissions from the new schema parsed in the same place (into the emission table rather than separated).
How to fix it: We did something similar last year when adding emissions from R3 reports
Probability (how likely the bug is to happen, scored from 1-5): 5
Effect (how bad the bug is when it does happen, scored from 1-5): 3
Steps to reproduce the behaviour: