Describe the task
Modify the NRIS-EPD importer to support both a full import and a smaller dataset daily import.
Purpose
The amount of data the NRIS-EPD importer needs to process keeps growing larger and larger, causing the importer to take longer and longer. (Currently around 10 hours). This is not sustainable, consumes unnecessary resource and increases the likelihood of failure.
Acceptance Criteria
Daily import job is quick and handles all the most recent changes (Process the latest year's worth of data)
A full run takes place at a regular occurrence (weekly or monthly) to ensure all records are up to date
Additional context
Could potentially add a way to manually import a record (i.e by ID)
Describe the task Modify the NRIS-EPD importer to support both a full import and a smaller dataset daily import.
Purpose The amount of data the NRIS-EPD importer needs to process keeps growing larger and larger, causing the importer to take longer and longer. (Currently around 10 hours). This is not sustainable, consumes unnecessary resource and increases the likelihood of failure.
Acceptance Criteria
Additional context