Merge Data Load : dr345

[cha801p]

• [x] After creation, add this issue to the Data Management project board https://github.com/orgs/AtlasOfLivingAustralia/projects/9

Metadata

• [x] Locate the existing data resource in the collectory and check metadata and parameters: https://collections.ala.org.au/dataResource/show/dr345
• [x] Check the unique identifier matches that in the new dataset

Data Prep

• [x] Map dataset to Darwin Core Terms in a CSV if necessary http://rs.tdwg.org/dwc/terms/. Package the CSV into a Darwin Core Archive
• [x] Ensure the dataset has correctly specified unique identifier without duplicates, and matches the unique identifier in the collectory connection parameters
• [x] Ensure images are specified in an images file in the DwCA with the identifier field populated
• [x] Check that species, location, times are adequately specific. Check back with the data provider if necessary
• [ ] Download existing DwCA from hdfs and merge the two DwCAs

Data Load

• [ ] Upload the new DwCA to the dmgt server
• [x] Run the Ingest_dataset job on Jenkins, setting the data resource UID (DRNNNN) parameter http://aws-spark-quoll-master.ala:9194/job/Ingest-dataset/
• [x] Note the statistics for total and new records from the end of the log file for the load by pasting the relevant lines here:

  [Log entries..... ]

  QA

• [ ] Wait overnight for next complete reindex
• [ ] Check that the number of records accessible match the loaded numbers using https://biocache.ala.org.au/occurrence/search?q=data_resource_uid:dr345
• [ ] Ask the data provider to review the new records using the same URL

[cha801p]

Ticket Update: January 15, 2023 (3 PM)

Issue: Data refresh for QVMAG (dr345)

Actions Taken:

• [x] Duplicate check ran - no duplicates found
• [x] Column names checked
• [x] Image URLs were not working - communicated with data provider and received working links
• [x] Created a DwCA locally
• [x] Loaded the data on databox successfully
• [x] Ran Preingestion DAG on prod with Load_images: true
• [x] Verified images.ala to confirm images are getting loaded
• [x] Verified the logs

Logs: 24/01/16 03:50:55 INFO ALAUUIDMintingPipeline: newUuids: 1482.0, preservedUuids: 116025.0, orphanedUniqueKeys: 57.0 24/01/16 03:50:55 INFO ALAUUIDMintingPipeline: Percentage UUID change: 1, allowed percentage: 50, override percentage check: false

Expected Count: 117507

Status: Waiting for indexing to reflect the changes

[cha801p]

Ticket Update: January 22, 2023 (9 AM) Data Loaded Successfully.