Merge Data Load : APC platypus & rakali records for ALA Sep-Nov 2023

[cha801p]

• [ ] After creation, add this issue to the Data Management project board https://github.com/orgs/AtlasOfLivingAustralia/projects/9

Metadata

• [ ] Locate the existing data resource in the collectory and check metadata and parameters: https://collections.ala.org.au/dataResource/show/dr8128
• [ ] Check the unique identifier matches that in the new dataset

Data Prep

• [ ] Map dataset to Darwin Core Terms in a CSV if necessary http://rs.tdwg.org/dwc/terms/. Package the CSV into a Darwin Core Archive
• [ ] Ensure the dataset has correctly specified unique identifier without duplicates, and matches the unique identifier in the collectory connection parameters
• [ ] Ensure images are specified in an images file in the DwCA with the identifier field populated
• [ ] Check that species, location, times are adequately specific. Check back with the data provider if necessary
• [ ] Download existing DwCA from hdfs and merge the two DwCAs

Data Load

• [ ] Upload the new DwCA to the dmgt server
• [ ] Run the Ingest_dataset job on Jenkins, setting the data resource UID (dr8128) parameter http://aws-spark-quoll-master.ala:9194/job/Ingest-dataset/
• [ ] Note the statistics for total and new records from the end of the log file for the load by pasting the relevant lines here:

  [Log entries..... ]

  QA

• [ ] Wait overnight for next complete reindex
• [ ] Check that the number of records accessible match the loaded numbers using https://biocache.ala.org.au/occurrence/search?q=data_resource_uid:dr8128
• [ ] Ask the data provider to review the new records using the same URL

[cha801p]

Ticket Update: December 6, 2023 (1 PM)

Issue: Refresh of APC data for September to November

Resolution: The new dataset has been successfully loaded into biocache. (Verified by loading on databox)

Actions Taken:

• [x] The data has been loaded onto prod
• [x] Column renamed
• [x] Date format fixed
• [x] General checks run - duplicate records, No scientific name missing and column with no data deleted (assocoatedMedia)
• [x] Verified the UUID count and will be ensured it reflects on prod tomorrow after indexing.

Log Records: INFO [2023-12-06 01:37:10,325+0000] [main] au.org.ala.pipelines.beam.ALAUUIDMintingPipeline: newUuids: 214.0, preservedUuids: 7038.0, orphanedUniqueKeys: 108.0 INFO [2023-12-06 01:37:10,325+0000] [main] au.org.ala.pipelines.beam.ALAUUIDMintingPipeline: Percentage UUID change: 2, allowed percentage: 50, override percentage check: false INFO [2023-12-06 01:37:10,325+0000] [main] au.org.ala.pipelines.beam.ALAUUIDMintingPipeline: Backing up existing UUIDs to /data/pipelines-data/dr8128/1/identifiers/ala_uuid_backup_1701826630325 INFO [2023-12-06 01:37:10,325+0000] [main] au.org.ala.pipelines.beam.ALAUUIDMintingPipeline: Pipeline complete. INFO

Useful Links: Prod: https://collections.ala.org.au/public/show/dr8128

Challenges Encountered: Modified "catalogueNumber" to "catalogNumber". Fixed date format; This problem may have arisen due to the xlsx file format.

Useful Prod Stats before indexing (Testing purposes): 6,998 records returned of 7,038 for Data resource: Australian Platypus Conservancy Exclude duplicate records [(27 records excluded)] Exclude records with unresolved user annotations [(13 records excluded)]

[cha801p]

Prod count: 7,252 records Link: https://biocache.ala.org.au/occurrences/search?q=data_resource_uid:dr8128

Data refresh done successfully.