Merge Data Load : Bdbsa SA Flora and Fauna 09/2022

timhicks-ala commented 2 years ago

From https://support.ehelp.edu.au/a/tickets/155658:

The SA Department for Environment and Water has created an updated version of the SA Flora (BDBSA) and SA Fauna (BDBSA) records on the ALA FTP site. Please update the datasets metadata with the month and year of this extraction - 09/2022

[ ] After creation, add this issue to the Data Management project board https://github.com/orgs/AtlasOfLivingAustralia/projects/9

Metadata

[ ] Locate the existing data resource in the collectory and check metadata and parameters: https://collections.ala.org.au/dataResource/show/drNNNN
[ ] Check the unique identifier matches that in the new dataset

Data Prep

[ ] Map dataset to Darwin Core Terms in a CSV if necessary http://rs.tdwg.org/dwc/terms/. Package the CSV into a Darwin Core Archive
[ ] Ensure the dataset has correctly specified unique identifier without duplicates, and matches the unique identifier in the collectory connection parameters
[ ] Ensure images are specified in an images file in the DwCA with the identifier field populated
[ ] Check that species, location, times are adequately specific. Check back with the data provider if necessary
[ ] Download existing DwCA from hdfs and merge the two DwCAs

Data Load

[ ] Upload the new DwCA to the dmgt server
[ ] Run the Ingest_dataset job on Jenkins, setting the data resource UID (DRNNNN) parameter http://aws-spark-quoll-master.ala:9194/job/Ingest-dataset/
[ ] Note the statistics for total and new records from the end of the log file for the load by pasting the relevant lines here:
```
[Log entries..... ]
```
QA
[ ] Wait overnight for next complete reindex
[ ] Check that the number of records accessible match the loaded numbers using https://biocache.ala.org.au/occurrence/search?q=data_resource_uid:drNNNN
[ ] Ask the data provider to review the new records using the same URL

patkyn commented 2 years ago

There're offending duplicates in the catalogNumbers in SAflora-01092022.DwC.csv that has caused the ingestion and indexing to fail for dr366

catalogNumber SUP10981-11-DISCARDED SUP12050-1-DISCARDED SUP12629-1-DISCARDED SUP13054-26-DISCARDED SUP13100-14-DISCARDED SUP13148-11-DISCARDED SUP13200-15-DISCARDED SUP13201-29-DISCARDED SUP13274-5-DISCARDED SUP13912-35-DISCARDED SUP13922-36-DISCARDED SUP13934-28-DISCARDED SUP13935-13-DISCARDED SUP13937-37-DISCARDED SUP13937-37-DISCARDED SUP13937-54-DISCARDED SUP14928-13-DISCARDED SUP15168-30-DISCARDED SUP15173-27-DISCARDED SUP15182-9-DISCARDED SUP15189-24-DISCARDED SUP15234-7-DISCARDED SUP15272-35-DISCARDED SUP15283-26-DISCARDED SUP15383-22-DISCARDED SUP8690-25-DISCARDED SUP8770-19-DISCARDED SUP8777-40-DISCARDED SUP8777-40-DISCARDED SUP8792-23-DISCARDED SUP8857-38-DISCARDED SUP8916-59-DISCARDED

peggynewman commented 1 year ago

Resolved long ago

AtlasOfLivingAustralia / data-management

Merge Data Load : Bdbsa SA Flora and Fauna 09/2022 #819