Automated publishing of redacted Mines Act inspection reports

[RFK250]

As a NRCED user interested in mining operations, I want to be able to view the inspection reports for mines so that I can understand how the mine is operating and feel confident that government is holding mining companies accountable.

As a Mines inspector, I want ONLY inspection reports that are reviewed and redacted to be published to NRCED so that we remain compliant with the Mines Act.

Additional Context

• EMLI inspections are stored on NRIS
• It is possible to import those reports using the NRIS Web Service, but they are currently not being published
• It is not acceptable to simply publish just any attachments associated with an EMLI NRIS record - only reviewed, redacted and finalized attachments should be published
• Review and redaction of inspection reports is currently conducted by EMLI staff. It's a fully manual process that ends with a redacted copy of the original inspection report being created.
• There is no function in NRIS that would allow an NRIS user to differentiate publishable attachments from non-publishable attachment
• An acceptable initial increment would be to publish only attachments that have filenames that match a TBD naming convention - however, I'm open to alternative ideas.
• For an initial increment it should be assumed that no systems changes to NRIS or the NRIS Web Service are possible, as those applications are contracted to a third party and would have no timeline for completion

Acceptance Criteria

• GIVEN There is a finalized, redacted inspection record on EMLI's NRIS application
• WHEN the NRIS-EMLI Integration runs
• THEN ONLY that finalized, redacted inspection record is imported and published to NRCED

CODE SNIPPETS TO BE UPDATED: Logic for NRIS-EMLI import: https://github.com/bcgov/NRPTI/blob/master/api/src/integrations/nris-emli/datasource.js

API called for obtaining nris-emli records: "https://api.nrs.gov.bc.ca/nrisws-api/v1/emprInspections?inspectionStartDate=2020-01-01&inspectionEndDate=2020-01-15" inspectionStartDate: 2020-01-01 (Hardcoded) inspectionEndDate: 2 weeks from start date, for every subsequent call this is incremented 7 days until end date reaches current date.

The following checks are done to find valid complete records that are >= 7 days:

//Only import assessmentSubStatus in 'Closed', 'Response Received' or 'Report Sent' const validassessmentSubStatus = ['Closed', 'Response Received', 'Report Sent'];

//For 'Response Received' and 'Report Sent' only import records that are 45 days old if (validassessmentSubStatus.slice(1).includes(record.assessmentSubStatus) && reportDateAge < 45) return false;

//reject if inspection type isn't Mine Inspection: if (record.inspection.inspectionSubType !== 'Mine Inspection') return false;

//reject if assessmentSubType isn't Compliance Review or Inspection if (record.assessmentSubType !== 'Compliance Review' && record.assessmentSubType !== 'Inspection') return false;

//reject if inspectionSubType is audit if (record.inspection.inspectionType[0] === 'Audit') return false;

This is how a valid nris-emli record looks like that is obtained via the api call.

valid-record.json

[LolandaE]

Waiting for NRIS updates from CS at Mines

[sggerard]

Currently uploading of record attachments in the NRS-EMLI integration are disabled using the environment variable NRIS_EMLI_DOCUMENT_BINARIES_ENABLED. This environment variable does not appear to be set in openshift so would default to false. See relevant commit: https://github.com/bcgov/NRPTI/pull/852/commits/6daf09af7fb95cc9f7cebdcf9c64c88746638b52

Edit: Nevermind, env variable is set here and is currently enabled: https://github.com/bcgov/NRPTI/commit/1fd5c3efc5d611fbd0b077099c1da9c763455b61

Currently logic only allows 'Final Report' to be published. At this time are we only looking to publish Inspection reports and not other attachments? Need to filter attachments by attachmentComment of "INSPECTION Report", fileType = "Report" for example.

Short term/initial increment solution might be to check attachment filePath (file name) and/or potentially the attachComment field (depending on where else the field flows) for specific keyword/s such as PUBLIC, REDACTED or FINALIZED.

Note: After a specific date all inspection records will be redacted and contain no personal information.

Long term solution would be to implement a publish or public field on the attachment in NRIS that flows through the API. Can add support for this on the NRPTI end awaiting future implementation on the NRIS end. Robin Indicated this is being worked on already

[Keegnan]

Can work off June 1st, 2024

[acatchpole]

merged and deployed to test

[sggerard]

Hard to test as the EMLI importer isn't currently running due to what appears to be a major outage

[ColinSquirrell]

@sggerard - Sorry, last minute change. Due to a recent data migration in inspection categories here some updated qualifiers for which records are valid:

Change in valid records: if (record.assessmentSubType !== 'Inspection - Desktop' && record.assessmentSubType !== 'Inspection - Site Visit') return false;

This condition is no longer required (all inspectionSubType are acceptable to post) //reject if inspectionSubType is audit if (record.inspection.inspectionType[0] === 'Audit') return false;

New Condition Only import records for Visit Date >= July 1, 2024

New Condition Can we add an additional condition that only imports a file if it's a PDF. (no Word, or image, etc.)

[sggerard]

Additional condition Publicly Post the Inspection Report = Yes

[sggerard]

@ColinSquirrell - I have some follow up questions:

• I can't see an exposed field on the /emprInspections endpoint for the 'Publicly Post Inspection Report' dropdown Yes/No? Is this deployed to production and if so what is the field called?
• For the condition "Visit Date >= July 1, 2024" can I use the attachmentDate field on the attachment or is there another field I should be using?
• The existing logic in NRPTI for importing attachments was for Final reports. Just wanted to confirm that these reports should still be imported and if any of the updated/new logic (such as only allowing pdf's) should be applied too.

For allowing only pdf's I have added a condition: attachment.attachmentMediaType == 'application/pdf'

[ColinSquirrell]

@sggerard Thanks Sam! Good questions.

1) Publicly Post Inspection Report is in production - a recent addition so perhaps while captured in NRIS it was not incorporated into the API? If so we would be blocked and required action by CGI in the next NRIS release to expose? @jakemorr ?

2) Here's the front-end view of Visit Date (date the inspector visited the inspection). Probably most reliable field for capturing post July 1, 2024 date.

2) File types to upload. Only PDFs and file category is "Report" or "Final Report"

[RFK250]

It's amazing to see this work still continuing. What a great milestone this will be.

[sggerard]

Blocked awaiting action by CGI

[ColinSquirrell]

July 25, 2024 - Request raised with NRIS interim Product Owner Shyam.Chandran@gov.bc.ca. Adding this functionality to NRIS will be blocked behind other priority NRIS work. Expecting multi-month delay before NRIS API work can proceed to unblock this ticket.

[RFK250]

That is disappointing.

Edit: I realize that is not the most helpful comment. I am happy to contribute more productive comments, but I kind of feel like this isn't really my space to be contributing right now. :) But, you know I love this product and am willing to help if I can!