dr365-dr366 Merge Data Load : BDBSA SA Flora and Fauna Files to ALA (03/2023)

[timhicks-ala]

From https://support.ehelp.edu.au/a/tickets/170612

The SA Department for Environment and Water has created an updated version of the SA Flora (BDBSA) and SA Fauna (BDBSA) records on the ALA FTP site. Please update the datasets metadata with the month and year of this extraction - 03/2023

• [ ] After creation, add this issue to the Data Management project board https://github.com/orgs/AtlasOfLivingAustralia/projects/9

Metadata

• [ ] Locate the existing data resource in the collectory and check metadata and parameters: https://collections.ala.org.au/dataResource/show/drNNNN
• [ ] Check the unique identifier matches that in the new dataset

Data Prep

• [ ] Map dataset to Darwin Core Terms in a CSV if necessary http://rs.tdwg.org/dwc/terms/. Package the CSV into a Darwin Core Archive
• [ ] Ensure the dataset has correctly specified unique identifier without duplicates, and matches the unique identifier in the collectory connection parameters
• [ ] Ensure images are specified in an images file in the DwCA with the identifier field populated
• [ ] Check that species, location, times are adequately specific. Check back with the data provider if necessary
• [ ] Download existing DwCA from hdfs and merge the two DwCAs

Data Load

• [ ] Upload the new DwCA to the dmgt server
• [ ] Run the Ingest_dataset job on Jenkins, setting the data resource UID (DRNNNN) parameter http://aws-spark-quoll-master.ala:9194/job/Ingest-dataset/
• [ ] Note the statistics for total and new records from the end of the log file for the load by pasting the relevant lines here:

  [Log entries..... ]

  QA

• [ ] Wait overnight for next complete reindex
• [ ] Check that the number of records accessible match the loaded numbers using https://biocache.ala.org.au/occurrence/search?q=data_resource_uid:drNNNN
• [ ] Ask the data provider to review the new records using the same URL

[rosemaryjoconnor]

Jenkins job running. Full project name: Data-resource-loads/dr365,dr366-BDBSA-quarterly This job is currently being triggered manually.

SA BDBSA ftp their files into uploads.ala.org.au here /mnt/data/dewnr-sa/upload/dewnr-sa every quarterly. The datestamp on the filename needs to be matched in order to process the correct files.

New Fetcher to be written before next data load

[rosemaryjoconnor]

Datasets loaded successfully 27/03/2023

[rosemaryjoconnor]

BDBSAFetcher.py, fetcher written. Need to check with Mahmoud/Patricia re where to upload code on S3 to test on databox.

• preingestion bucket on ala-databox-avro or ala-databox-dev?