NRPTI Automated Import and Publish of data to NRCED

[RFK250]

Overarching User Stories

As an administrative user of NRPTI who manually adds and/or imports my agency's records, I want my data to be imported and published automatically so that I don't need to take manual steps to import data and can use my time more effectively.

As a user of NRCED, I want to see the most recent available data possible so that I have confidence that government is conducting the compliance and enforcement activities necessary to protect the environment.

As a member of the media using NRCED, I want to see the most recently available data possible so that I can include the relevant NRCED hyperlinks to records in the articles I write.

Description

Before NRPTI, data was manually aggregated, transcribed, transformed and published to an old version of NRCED in a complicated process that involved many people and had a long lead time. Once published, data was already outdated (often 6-12 months old) and was often rife with typos and inconsistencies that could not be fixed. Furthermore, many parts of government would not participate in the old database due to the high level of work involved.

NRPTI solved those many problems through automated aggregation, bulk uploading, algorithmic transformations, and instant publishing. However, not all records are fully automated. Many records reside on other databases and are imported through bulk CSV uploads, but were never automated because funding ran out before automation could be tackled.

This epic covers the tickets for automating the import, transforms and publishing of records.

Notes

• Generally, it will be fastest and easiest to automate the imports of records where (a) a CSV importer exists, (b) an existing approach to importing can be used (e.g. using an existing materialized view for a different subset of data), and (c) no interventions (e.g. manual data transforms, reviews, or approvals) are performed by business area operations staff as part of the process.

[RFK250]

@jakemorr was this closed by accident?

[jakemorr]

@RFK250 closed accidentally by one of our new devs! reopened