Merge Data Load : Frogwatch - 12/4/2023

[timhicks-ala]

Helpdesk request is https://support.ehelp.edu.au/a/tickets/173977

Data has been uploaded to the resource: https://collections.ala.org.au/dataResource/show/dr10043

Previous (example) Frogwatch data upload ticket is here: https://github.com/AtlasOfLivingAustralia/data-management/issues/786

• [ ] After creation, add this issue to the Data Management project board https://github.com/orgs/AtlasOfLivingAustralia/projects/9

Metadata

• [ ] Locate the existing data resource in the collectory and check metadata and parameters: https://collections.ala.org.au/dataResource/show/drNNNN
• [ ] Check the unique identifier matches that in the new dataset

Data Prep

• [ ] Map dataset to Darwin Core Terms in a CSV if necessary http://rs.tdwg.org/dwc/terms/. Package the CSV into a Darwin Core Archive
• [ ] Ensure the dataset has correctly specified unique identifier without duplicates, and matches the unique identifier in the collectory connection parameters
• [ ] Ensure images are specified in an images file in the DwCA with the identifier field populated
• [ ] Check that species, location, times are adequately specific. Check back with the data provider if necessary
• [ ] Download existing DwCA from hdfs and merge the two DwCAs

Data Load

• [ ] Upload the new DwCA to the dmgt server
• [ ] Run the Ingest_dataset job on Jenkins, setting the data resource UID (DRNNNN) parameter http://aws-spark-quoll-master.ala:9194/job/Ingest-dataset/
• [ ] Note the statistics for total and new records from the end of the log file for the load by pasting the relevant lines here:

  [Log entries..... ]

  QA

• [ ] Wait overnight for next complete reindex
• [ ] Check that the number of records accessible match the loaded numbers using https://biocache.ala.org.au/occurrence/search?q=data_resource_uid:drNNNN
• [ ] Ask the data provider to review the new records using the same URL

[cha801p]

Loaded on prod: https://ap-southeast-2.console.aws.amazon.com/emr/home?region=ap-southeast-2#/clusterDetails/j-2RTS72DAB8LW6

[cha801p]

https://collections.ala.org.au/public/show/dr10043

[peggynewman]

@timhicks-ala this data has been loaded. The last update date problem in collectory is an outstanding bug and we apologise for that. could you let the data provider know the data has been loaded and close the ticket?

[timhicks-ala]

@peggynewman Done - I'll let you know if the submitter raises any further issues.