New Data Load : SREBA invertebrate data

[cha801p]

https://collections-test.ala.org.au/dataResource/show/dr22377

[cha801p]

Ticket Update: February 16, 2024 (12:30 PM)

Issue: New dataset to load.

Solution: Successfully load the new dataset into biocache

Actions Taken: Successfully loaded the data on the test environment

• [x] Data review
• [x] Deleted empty columns
• [x] Duplicate check ran
• [x] Date format fixes
• [x] DwcA created locally
• [x] Loaded the data on biocache-test
• [x] Sent a link to the data provider for review

Issues Encountered: The issue with the column name was identified This caused ingest_small_dataset to fail twice The column name was fixed and DwcA was created locally Loaded the data on biocache-test

Loaded data for review: https://biocache-test.ala.org.au/occurrences/search?q=data_resource_uid:dr22377

Status: Waiting for the customer review

[cha801p]

Issue with "Individual count" column flagged by data provider - I think the column is formatted as float, it should be plain text or int

[cha801p]

Ticket Update: February 28, 2024 (1 PM)

Issue: New dataset to load.

Solution: Successfully load the new dataset into biocache

Actions Taken:

• [x] New dr created - https://collections.ala.org.au/dataResource/show/dr25133
• [x] Updated the metadata
• [x] Reviewed the Data
• [x] Deleted empty columns
• [x] Duplicate check ran
• [x] Date format fixes
• [x] Individual count column fixed
• [X] DwcA created locally - eml.xml removed
• [X] Loaded the data on collectory
• [X] Ran preingestion
• [X] UUID logs checked

Logs: INFO [2024-02-28 00:54:47,422+0000] [main] au.org.ala.pipelines.beam.ALAUUIDMintingPipeline: Running the pipeline INFO [2024-02-28 00:54:48,147+0000] [main] au.org.ala.pipelines.beam.ALAUUIDMintingPipeline: As this is an initial load for this dataset, UUID change checks are not ran. INFO [2024-02-28 00:54:48,147+0000] [main] au.org.ala.pipelines.beam.ALAUUIDMintingPipeline: Pipeline complete. INFO [2024-02-28 00:54:48,147+0000] [main] au.org.ala.pipelines.beam.ALAUUIDMintingPipeline: Checking for backups to prune..../data/pipelines-data/dr25133/1/identifiers INFO [2024-02-28 00:54:48,148+0000] [main] au.org.ala.pipelines.beam.ALAUUIDMintingPipeline: Writing metrics..... INFO [2024-02-28 00:54:48,149+0000] [main] org.gbif.pipelines.common.beam.metrics.MetricsHandler: Trying to write pipeline's metadata to a file - /data/pipelines-data/dr25133/1/uuid-metrics.yml INFO [2024-02-28 00:54:48,150+0000] [main] org.gbif.pipelines.common.beam.metrics.MetricsHandler: Added pipeline metadata - newUuidsAttempted: 1845,

Loaded data for review: https://collections.ala.org.au/public/show/dr25133

Status: Waiting for indexing

[cha801p]

Ticket Update: February 29, 2024 (12:30 PM)

Issue: The "individual count" column issue continues to persist after the indexing

Solution: Reload the data

Actions Taken:

• The issue continues to persist after indexing
• The data was reformatted
• occurrence.csv was regenerated
• Loaded and tested on collectory-test
• Reloaded the data on prod
• Preingestion triggered
• Preingestion ran successfully

Loaded data for review: https://collections.ala.org.au/public/show/dr25133

Status: Waiting for indexing

[cha801p]

Status: Data loaded on Prod and links sent to the data provider for review