Overview
Rather than download and update the resource.csv file, we should be able to generate it from the contents of the log.csv file. This means we only have one file to fetch, and don't have to handle the case of log.csv being obtained without resource.csv, or vv.
We have also had as issue where changing the organization codes (from local-authority-eng:xxx to local-authority:xxx) which resulted in the resourse.csv now containing a mix of both old and new codes.
Pull Request(PR):
Tech Approach
Don't obtain resource csv from S3
Update the code to generate it from the contents of log.csv, plus the collection files.
This will need to be a mix of what we do for new resources, but for all the entries in log.csv
Unit and integration tests
Acceptance Criteria/Tests
as a check, the generated and updated files should only differ in expected ways (local-authority-eng: to local-authority: in the organization field). It should be possible to manually patch an updated file to get the expected output for a generated one.
resource.csv is not downloaded from, or uploaded to S3 (and the existing files in S3 are removed)
resource.csv is successfully generated from the log.csv file
processing is not impacted
Overview Rather than download and update the resource.csv file, we should be able to generate it from the contents of the log.csv file. This means we only have one file to fetch, and don't have to handle the case of log.csv being obtained without resource.csv, or vv.
We have also had as issue where changing the organization codes (from local-authority-eng:xxx to local-authority:xxx) which resulted in the resourse.csv now containing a mix of both old and new codes.
Pull Request(PR):
Tech Approach
Don't obtain resource csv from S3 Update the code to generate it from the contents of log.csv, plus the collection files. This will need to be a mix of what we do for new resources, but for all the entries in log.csv Unit and integration tests
Acceptance Criteria/Tests
as a check, the generated and updated files should only differ in expected ways (local-authority-eng: to local-authority: in the organization field). It should be possible to manually patch an updated file to get the expected output for a generated one.
resource.csv is not downloaded from, or uploaded to S3 (and the existing files in S3 are removed) resource.csv is successfully generated from the log.csv file processing is not impacted
Resourcing & Dependencies
Developer. S3 access to remove old files.