Parse "other venting sources" from swrs xml reports for viewing in metabase

[dleard]

From the metabase channel: Need details about venting - There is a free text field available in the SWRS report to enter details about "other venting sources". The tags in the XML contain the string "VentingDetails". These don't appear to be parsed in and would be useful for a current project. Would it be possible to add? What we really want is a list of all of the entries (unique values at least) so that we can learn what "other types" of venting there are.

Compliance has asked us to parse data about "other venting sources" & display that data in metabase for a current project. We will need to gather some requirements about what they need in order for that data to be useful. Questions:

• Does each source need separate identification by tag?
• Should it be tied to a specific fuel (as opposed to just a list by report)?
• Should the tag always contain the word 'other'? (there are tags that contain "ventingdetails" but not the word "other" in some reports)

Todo:

• Gather requirements from compliance
• find the consistent location of these tags in the xml reports
• decide which table this data should live in or if it needs a new table entirely
• create an xpath matcher (and potentially a new materialized view) to capture the data from the matching tags in the xml
• update or create a load function to get the results into a table (depending on if a new table was needed)
• add tests to ensure the new data is properly being parsed
• release GGIRCS to prod (so the data can be viewed in metabase)

Sample report id containing these tags: 20055 There is a query result available in our scrum team channel with reports likely to contain these tags

Development Checklist:

• [x] swrs_transform schema has an 'other_venting' materialized view to parse the proper tags
• [x] swrs schema has an 'other_venting' table { id, eccc_xml_file_id, report_id, detail_tag, detail_value }
• [x] swrs_transform schema as a 'load_other_venting' function
• [x] the 'load_other_venting' function joins on swrs_transform.final_report and swrs_transform.report
• [x] swrs_transform.load() is reworked to include the new function in the mv_array

[LindsayMacfarlane]

Hey team! Please add your planning poker estimate with Zenhub @pbastia @dleard @Sepehr-Sobhani @gurjmatharu @mikevespi

[LindsayMacfarlane]

Please add your planning poker estimate with Zenhub @sam-warren

[dleard]

After meeting with CAS:

• Need this within ~ 1 month (draft has already been submitted, this will be used in an addendum)
• Needs a list of unique values and containing tags
• Tag must include the word 'other' (there are some tags that have 'venting details' but not 'other', these should be excluded) rough idea for table: { id, report_id, detail_tag, detail_value }

[mikevespi]

@LindsayMacfarlane I'm not sure if you want to do any QA on this, it would require going into metabase once it's in test and querying this. I will likely do this myself once it's there, I just put this in Pending PO for the day in case.

[LindsayMacfarlane]

Thank you @mikevespi - I feel the business area will be better able to test this once in TEST. Feel free to merge into TEST.