Automate Removal of Personal Data from ALC-CSV Importer

[LolandaE]

As an ALC administrative user of NRPTI, I want the ALC inspection data to be imported and published automatically so that I don't need to take manual steps to import data and can use my time more effectively.

As a user of NRCED, I want to see the most recent available data possible so that I have confidence that government is conducting the compliance and enforcement activities necessary to protect the environment.

As a member of the media using NRCED, I want to see the most recently available data possible so that I can include the relevant NRCED hyperlinks to records in the articles I write.

Acceptance Criteria GIVEN I am a NRPTI user with ALC user permissions AND I have drag and dropped a valid ALC inspections CSV file into the "Drop files..." UI WHEN I click the "Start Job" button THEN The ALC Inspections CSV importer runs as it currently does EXCEPT follows the below criteria when getting the entity type:

IF A record's "Inspected Property Owner" value ends with " Ltd.", " Inc.", or " Corp." (not case sensitive, with or without the trailing period) THEN Entity Type = Company ELSE Entity Type = Individual

Note

• ALC inspectors will be trained to enter data into the source system (NRIS) according to the above condition; ALC staff will perform QA/QC on historical records in NRIS prior to using the ALC CSV uploader in production.

Additional Context

• Currently, an ALC administrative user conducts manual queries of NRIS and then uploads them to NRCED. This is only done periodically, meaning that an inspection record created in NRIS today may not show up on NRCED for several months. This impacts the recency and relevancy of the data on NRCED.
• ALC inspection data is stored on NRIS. However, the NRIS Web Service applet does not have a configuration for exporting ALC inspection data the same way that it does for EPD inspections and EMLI inspections.
• The FLNR-NRIS-CSV model for importing from NRIS is likely the best analogue: a microservice (FME Server) sits in Zone B and queries NRIS (CVIS) nightly. It then pushes algorithmically transformed data to NRPTI in OpenShift. This eliminates the security concerns associated with a pull approach.
• The data transforms necessary are coded into the ALC-CSV importer.
• Currently, ALC does a manual transform before they drag and drop their CSV file. They do this to remove the names of individuals from "Inspection Property Owner." Then the importer (see getEntityType) parses blanks/empty values as EntityType=Individual, and non-blank/empty as EntityType=Company. This manual transform will need to be hard coded.

[RFK250]

@LolandaE Updated with revised AC for developer review. ALC will be acting quickly to get their data entry operations and standards in line with this ASAP. I'm happy to answer any questions they may have.

Devs will probably ask the question of whether or not there is a way to add a "Company/Individual" flag to the source system database (CVIS) and UI (NRIS) as it would be simpler and more reliable, but the answer is essentially no. It would be disproportionately expensive for ALC and would take a long time due to the long backlog and limited capacity of the contracted maintenance vendor.

[Christopher-walsh22]

I have changed the logic for how the importer determines Personal vs Company to match the criteria accordingly. Using a template provided by SO initial testing has been completed. I have altered the test data to match testing for this ticket and added it to the DEV Chat files to be used for future ALC tickets.