Ecollaboraton Frog Monitoring event core

[peggynewman]

As per freshdesk ticket https://support.ehelp.edu.au/a/tickets/181965

[cha801p]

Ticket Update: September 4, 2023 (6 PM)

Problem: New Event dataset to load.

Solution: Successfully load the new dataset into biocache.

Actions Taken: Data was thoroughly reviewed and restructured to extract event.csv and occurrence.csv files.

Communication: A sample of the data was sent to the data provider, along with a request to provide data in a similar format.

Questions:

• [x] Is it possible for an event to exist without any associated occurrences? - No
• [x] Should all of the following fields [Site, Night, Date, Time] be used to create the eventID?
• [x] What is the method for generating occurrenceIDs?

Next Step: Once eventID and occurrenceIDs are generated, create event.csv, measurementsOrFacts.csv, and occurrence.csv from the data.

[cha801p]

Ticket Update: September 11, 2023 (5 PM)

Problem: New Event dataset to load.

Solution: Successfully load the new dataset into biocache.

Actions Taken: Data Loaded on databox.

Communication: A sample link was shared on datamanagement-internal group for review (https://biocache-test.ala.org.au/occurrences/search?q=data_resource_uid:dr21990)

Issued Faced:

Set Unique Key as - occurrenceID

• [x] 1. encoding = {key: convert_values(result[key]) for key in vars(self.csv_encoding)} KeyError: 'csv_eol' - Reset Connection parameters to Protocol: Darwin Core Archive in the collectory setting
• [x] 2. KeyError: "There is no item named 'event.csv' in the archive" Command exiting with ret '1' - Csv named changed and aligned with meta.xml
• [x] 3. KeyError: "There is no item named 'occurrence.csv' in the archive" Command exiting with ret '1' - Csv named changed and aligned with meta.xml
• [x] 4. Exception in thread "main" java.lang.IllegalArgumentException: Illegal character in path at index 11: Observation Type - Removed spaces in the column name where column name is not mapped to a Darwin Core Term
• [x] 5. Loading issue with measurementsOrFacts.csv - restructure the CSV and reload

[peggynewman]

The event hierarchy looks odd - @charvolant could you have a look at this with Raj?

Also, just re some data: https://biocache-test.ala.org.au/occurrences/c03e9a2a-ce49-4f68-aaa6-6d37825f8d2a

• no CC license
• samplingProtocol looks weird in biocache: 1. See 2. Field Observation
• no event date
• no measurementOrFacts

[cha801p]

Ticket Update: October 17, 2023 (2 PM)

Issue: Required to load a new event dataset.

Resolution: Successfully imported the new dataset into biocache.

Actions Taken:

1. Data uploaded to databox and events.test.
2. Updated the CC license.
3. Updated metadata.
4. Corrected some irregular samplingProtocol values, although some anomalies still remain.
5. Adjusted event date.
6. Updated measurementOrFacts.csv and reloaded the data.

Challenges Faced: Dealing with the manual inconsistencies in the RAW data required manual Excel adjustments.

Next Step:

• [x] Request for @peggynewman to review reloaded data.
• [ ] Once reviewed, proceed with loading the data on the production.

Communication: Regarding the structure of the data-

1. I have conducted a comprehensive review of the data, and it is properly organized and structured.
2. @charvolant and I have meticulously examined the data during a call, and we did not discover any irregularities.
3. This issue has been reported within the extended-data-model context but has not yet been addressed.

[cha801p]

Ticket Update: November 28, 2023 (10 AM)

Issue: Required to load a new event dataset.

Resolution: Successfully imported the new dataset into biocache.

Actions Taken:

• [x] Follow-up email sent to the data provider asking for a data update. (As discussed in the previous mail chain)

Status: Waiting for the customer

[cha801p]

Ticket Update: December 14, 2023 (3 PM)

Issue: Required to load a new event dataset on biocache

Resolution: Successfully imported the new dataset into biocache.

Actions Taken:

• [x] Loaded the dataset on events-test and Biocache-test
• [x] Sent a test link to the data provider for validation
• [x] Informed the data provider about Raj's leave

Problems Encountered: extendedMeasurementOrFacsts are not appearing on biocache records

Possible Solution: Restructure extendedMeasurementOrFacsts.csv according to the sample file sent by Doug

Status: Work Pending

[cha801p]

Status: Databox link sent to data provider, waiting for approval Next Step: Once verified by data provider load the data on Prod

[cha801p]

https://collections.ala.org.au/dataResource/show/dr24697

[cha801p]

Ticket Update: February 8, 2024 (5 PM)

Issue: Required to load a new event dataset on biocache

Resolution: Successfully imported the new dataset into biocache.

Actions Taken:

• [x] Data verified by the data provider and meta changes requested
• [x] Verified the data on events-test and databox
• [x] New dr created on prod
• [x] New metadata incorporated to prod
• [x] DwCA uploaded to collectory
• [x] Preingest dataset Ran
• [x] UUID logs checked: INFO [2024-02-08 05:13:24,758+0000] [main] au.org.ala.pipelines.beam.ALAUUIDMintingPipeline: As this is an initial load for this dataset, UUID change checks are not ran.

Status: Waiting for full_solr_index

[peggynewman]

There are 600 records with no state. Looking at one record, the site was Sippy Downs, which I think is in Qld. Is there no location info for that site?

The title of the DR is "Sunshine Coast..." so even if there is no lat/long, we could set the stateProvince field to Queensland. Do you see any problems with doing this?

[cha801p]

Ticket Update: February 27, 2024 (5 PM)

Issue: Required to load a new event dataset on prod

Resolution: Successfully imported the new dataset into prod/events.

Actions Taken:

• [x] The location issue mentioned above has been fixed and deployed on prod
• [x] There was a request to delete a record from the data provider - successfully deleted a record
• [x] This has been informed to the data provider

Data load successful