Merge Data Load : APC data for December 2023 to February 2024

[cha801p]

Ticket Update: March 6, 2024 (2 PM)

Issue: Refresh of APC data for December 2023 to February 2024

Resolution: Load the dataset onto the databox

Actions Taken:

• [x] The data has been loaded and verified onto the databox (multiple attempts)
• [x] Column renamed
• [x] Renamed "catalogueNumber" to "catalogNumber".
• [x] Fixed date format; This problem may have arisen due to the xlsx file format.
• [x] General checks run - duplicate records, No scientific name missing and column with no data deleted (assocoatedMedia)
• [x] Verified the UUID count and will be ensured it reflects on prod tomorrow after indexing.

Prod Log Records: INFO [2024-03-06 03:15:02,860+0000] [main] au.org.ala.pipelines.beam.ALAUUIDMintingPipeline: Checking the percentage change in new UUIDs: INFO [2024-03-06 03:15:02,861+0000] [main] au.org.ala.pipelines.beam.ALAUUIDMintingPipeline: newUuids: 149.0, preservedUuids: 7252.0, orphanedUniqueKeys: 108.0 INFO [2024-03-06 03:15:02,861+0000] [main] au.org.ala.pipelines.beam.ALAUUIDMintingPipeline: Percentage UUID change: 2, allowed percentage: 50, override percentage check: false INFO [2024-03-06 03:15:02,861+0000] [main] au.org.ala.pipelines.beam.ALAUUIDMintingPipeline: Backing up existing UUIDs to /data/pipelines-data/dr8128/1/identifiers/ala_uuid_backup_1709694902861 INFO [2024-03-06 03:15:02,862+0000] [main] au.org.ala.pipelines.beam.ALAUUIDMintingPipeline: Pipeline complete.

Links:

• Prod: https://collections.ala.org.au/public/show/dr8128
• Databox: https://collections-test.ala.org.au/public/show/dr8128

Challenges Encountered:

• [x] Databox ingest failed with following logs: INFO [2024-03-05 01:31:04,251+0000] [main] au.org.ala.pipelines.beam.ALAUUIDMintingPipeline: newUuids: 149.0, preservedUuids: 0.0, orphanedUniqueKeys: 6956.0 INFO [2024-03-05 01:31:04,251+0000] [main] au.org.ala.pipelines.beam.ALAUUIDMintingPipeline: Percentage UUID change: 100, allowed percentage: 50, override percentage check: false WARN [2024-03-05 01:31:04,251+0000] [main] au.org.ala.pipelines.beam.ALAUUIDMintingPipeline: The number of records with new identifiers has changed by 50%. This update will be blocked.

• [x] incremental_load set to true

• [x] It was identified that the unique id's had an issue and could not merge the incoming data

• [x] DwCA from prod was uploaded to collector-test to fix the issue

Useful Prod Stats before indexing (Testing purposes): 7,206 records returned of 7,252 for Data resource: Australian Platypus Conservancy Exclude duplicate records 33 records excluded Exclude records with unresolved user annotations [(13 records excluded)]

Status: Waiting for indexing