Update dr19813-FeralScan fetcher and RASD to handle sensitive data

AtlasOfLivingAustralia / data-management

Data management issue tracking

7 stars 0 forks source link

Update dr19813-FeralScan fetcher and RASD to handle sensitive data #971

Closed peggynewman closed 3 months ago

peggynewman commented 10 months ago

Update the FeralScan fetcher to:

Pull all sensitive (including deer, pig, wild dog) data from FeralScan that we can access securely via the API
Retain a copy of the raw data on S3 ala-uploaded-datasets
Obfuscate the dataset as required and publish the DwCA to production

After this is done, work with Feralscan to set up a RASD metadata statement and a process for providing authorised access to the data

rosemaryjoconnor commented 9 months ago

Code complete and tested in databox-dev and databox. Pull request initiated with Mahmoud as reviewer. Have yet to contact Feralscan re RASD metadata statement.

rosemaryjoconnor commented 9 months ago

Additional coding required to output DWCA with complete dataset.

rosemaryjoconnor commented 7 months ago

Code completed and in github develop branch as at: 24/10/2023 Not yet in master branch/production

rosemaryjoconnor commented 6 months ago

19/12/2023

FeralScan now in production with successful write of raw data to S3 bucket: s3://ala-uploaded-datasets/ala/dr19813
Data checked. Obfuscations and exclusion Deer, Pig and Wild dog correctly
DWCA on S3 confirmed as full dataset

peggynewman commented 5 months ago

Waiting on me to check on with Cam

peggynewman commented 5 months ago

Superceded by https://github.com/AtlasOfLivingAustralia/data-management/issues/1008