search

bcgov / NRPTI

Natural Resources Public Transparency Initiative
Apache License 2.0
5 stars 14 forks source link

Automate ALC Inspection Importing #1038

Open RFK250 opened 1 year ago

RFK250 commented 1 year ago

As an ALC administrative user of NRPTI, I want the ALC inspection data to be imported and published automatically so that I don't need to take manual steps to import data and can use my time more effectively.

As a user of NRCED, I want to see the most recent available data possible so that I have confidence that government is conducting the compliance and enforcement activities necessary to protect the environment.

As a member of the media using NRCED, I want to see the most recently available data possible so that I can include the relevant NRCED hyperlinks to records in the articles I write.

Notes:

Questions for Kyle

ACTION: Kyle to work with ALC to solidify what needs to be done. To make sure their inspected property owner value (INC, LTD) - these are currently being mixed up. Nothing exists to determine the difference between company and individual names. - June 21 This has been addressed with their inspectors, who will stop the practice of mixing company and individual names in the Inspected Property Owner field. ALC staff will also retroactively update Inspected Property Owner values in the source system so that existing records are not mixed up either.

RFK250 commented 1 year ago

@jakemorr Over to you for prioritization. There are a few importers that could use automating, so it's probably worth creating an Epic for all of them. This one is likely the simplest because there's a working model to follow.

RFK250 commented 1 year ago

Whoever tackles #1054 will likely gain the knowledge needed to write a ticket for this story.

acatchpole commented 7 months ago

NRIDS Service ticket for GTS access https://apps.nrs.gov.bc.ca/int/jira/servicedesk/customer/portal/1/SD-108006 Possible link to person who can grant GTS access https://apps.nrs.gov.bc.ca/int/jira/browse/SD-72030

sanjaytkbabu commented 7 months ago

FME Web portal access creds are present in relevant links

acatchpole commented 6 months ago

Unknowns :

RFK250 commented 6 months ago

I think when my answer for the first bullet was transcribed, some context was lost in the translation. A better way of putting it is: whatever documentation is in the NRPTI GitHub is what's available - which doesn't necessarily mean it's all there.

That being said: if you are replicating the NRIS-FLNR-CSV approach (i.e. using FME server), then you should focus on the FME server side of things. That's what queries the data from NRIS, puts it into a CSV file, and pushes that file to the S3 bucket.

Thus (and I think you've already figured this out) if you configure FME server to create a CSV file with the same syntax that the current ALC-CSV importer expects, then you can just reuse the ALC-CSV importer as is - just point it to the location of the CSV file that FME server creates, and set up a cron job to trigger the importer once a night. Make sense?

acatchpole commented 6 months ago

Special views for NRO inspections were created for FLNR for a variety of business needs. Such views do not exist for ALC records. When querying for these records, it will be from a table that holds ALL records. Will need to filter to ALC and Inspection records. Data will also come as XML. Also, FMES will need new credentials to allow access to this view.

CVIS Record View (probably) : ENFPROD1 -- CORS.CORS_CV_ASSESSMENTS_XVW Previous research: image

Process for getting FMES access to ALC inspections in CVIS:

  1. Identify all required fields
  2. Identify and get approval from data custodian for ALC records in NRIS
  3. Create service request for access

KL&OS will see if it is appropriate for them to create a test document of what the extracted XML might look like so work on the necessary transformations within FME can begin.

acatchpole commented 5 months ago

SR Created -> https://apps.nrs.gov.bc.ca/int/jira/servicedesk/customer/portal/1/SD-112444

The current plan is to give the existing proxy account the additional permissions it needs to access the required views.

There may yet be some push back to being granted full access to CVIS. If there is, we will need to figure out how to request new views created that serve ONLY the ALC inspection records. In that case, there might also be some costs associated with this work.

acatchpole commented 2 months ago

The following views for CVIS have been created and deployed:

Each has a corresponding role to support access:

PROXY_FME_NRIS is the proxy account that FME Server uses to access the CVIS database. For the purposes of this ticket, the CORS_CV_ALC_REPORTING_RLE role has been granted to that account, so FME Server can now query the CORS_CV_ALC_REPORTING_VW view.

Now it is a matter of setting up the extraction with FME. You will need GTS access which is access to a Windows VM with FME Workshop installed. To request access, go here and select Geospatial Services -> GTS / ArcGIS / SpatialFiles - Access and Changes and request access to Kamloops 10.8 desktop which is the system where FME Workshop is installed. This is the software you need create/edit .fme files (which define the ETL process).

Using the existing FLRNO FME file as a template(found here ), will need to edit so that:

This branch has changes that should handle the import from the S3 bucket, but the model for the ALC csv needs to be confirmed.

The links included further back in this ticket should provide all the other context needed for this ticket. especially the FME Confluence pages and Service Request 112444.

There is also an example ALC .csv (the kind they currently manually create and import) in the NRPTI Teams files (here). Note that not all of the fields in that file are actually consumed by NRPTI.