BSBSA SA Flora and Fauna Files to ALA - 09/2024

[timhicks-ala]

The SA Department for Environment and Water has created an updated version of the SA Flora (BDBSA) and SA Fauna (BDBSA) records on the ALA FTP site. Please update the datasets metadata with the month and year of this extraction - 09/2024

https://support.ehelp.edu.au/a/tickets/210607

[rosemaryjoconnor]

03/09/2024

SA Flora: dr366 can be processed now SA Fauna: dr365 will need a fetcher update for the new functionality to extract FrogWatch data in BDBSA to fr10043 code has been written needs testing with new dataset

Associated DM Issue: https://github.com/AtlasOfLivingAustralia/data-management/issues/1042

[rosemaryjoconnor]

02/09/2024

Multiple versions of files uploaded on same day and SA-Flora archive corrupt.

Emailed client via suppport request:

01/09/2024 multiple saflora_all and safauna_all files - see below listed in descending order - most current first in each list

SA-Flora

• saflora_all_04062024.zip - September 1, 2024, 09:14:26 - will not open at all is corrupt zip file
• saflora_all_01062024.zip - September 1, 2024, 09:14:23
• saflora_all_01092024.zip - September 1, 2024, 09:14:23

SA-Fauna

• safauna_all_01062024.zip - September 1, 2024, 09:14:23
• safauna_all_01092024.zip - September 1, 2024, 09:14:23
• safauna_all_04062024.zip - September 1, 2024, 09:14:23

Fetcher picks up the latest version for each as per final upload time on the server. Which in this scenario would mean that possibly incorrect or inconsistent versions would be loaded.

The issue was found when running the load for SA-Flora. File sa_flora_all_04062024.zip failed to load due to being a corrupt zip file. I have downloaded it manually and the same issue occurs when trying to load it manually.

Have requested they re-upload the latest files after testing that they can be opened successfully.

[rosemaryjoconnor]

04/09/2024

• dr366 - SAFlora new file contains 11 records with NaN eventDate.
• Fetcher will need updated as it does not cater for NaN numeric value when assigning to verbatimeEventDate and so crashes.

This can be done along with the FrogWatch updates.

dr366 - SA Flora load will be delayed until the new fetcher can be released

[rosemaryjoconnor]

06/09/2024

• SA Flora load failed in databox due to records missing eventDate.
• Emailed LM - indicated that these shouldn't be in the dataset.
• New dataset for SA Flora due next week

Testing SA Fauna/Frogwatch in databox.

[rosemaryjoconnor]

10/09/2024

New SA Flora file provided with incorrect records removed

• [x] Databox successful
• [x] Prod successful - new record count 2,068,010
• [x] Collectory dates updated in Prod

[timhicks-ala]

Data has been additionally updated: https://support.ehelp.edu.au/a/tickets/211371

[rosemaryjoconnor]

11/09/2024

BDBSA fetcher updates

Record counts

• [x] dr365 - SA Fauna - Old: 1,809,015 New: 1, 826,443
• [x] dr10043 - Frogwatch - Old: 8,646 New: 8,698

Databox loads

• [x] dr365 - SA Fauna

• [x] dr10043 - Frogwatch

• [x] Fetcher pull request: https://github.com/AtlasOfLivingAustralia/preingestion/pull/262

To do: Production Loads

• [ ] dr365 - SA Fauna - Old: 1,809,015 New: 1, 826,443
• [ ] dr10043 - Frogwatch - Old: 8,646 New: 8,698
• [ ] Collectory dates updated

[rosemaryjoconnor]

14/09/2024

• [x] Frogwatch - UUID catalogNumber updates in databox

[rosemaryjoconnor]

27/09/2024

BDBSA Fetcher in production and working as expected after release

Refer to: https://github.com/AtlasOfLivingAustralia/data-management/issues/1055