dr344-QM New Vernon Endpoint

[rosemaryjoconnor]

• [ ] QM data to be extracted from new endpoint

• [ ] Implementation of OAI-PMH on Vernon Browser as an upgrade to sustain the mobilisation of QM data to the ALA.

Mtg 27/03/2023

[rosemaryjoconnor]

Information emailed after mtg:

The harvesting would be via the /opacobjects endpoint. By default this just returns the ids, with a subsequent call to each record being used to retrieve the individual records. However, this endpoint can be changed to retrieve full records.

Here is an example from our demo site (mostly social history and art objects), retrieving multiple full records with each request. The demo site is set to return 12 records per page, but this is configurable:

https://browser.vernonsystems.com/api/v3/opacobjects?view=detail

Subsequent pages are retrieved using the offset page, when the offset is in multiple of the page sizes.

e.g. Page 2 on our demo site, with 12 records per page is https://browser.vernonsystems.com/api/v3/opacobjects?view=detail&offset=12

[rosemaryjoconnor]

QM fetcher updated and tested locally.

[rosemaryjoconnor]

Waiting on databox

[rosemaryjoconnor]

dr344 latest update complete. Fetcher to be scheduled weekly. New fetcher requires data endpoints to be provided

[rosemaryjoconnor]

Details of new API have been provided. 13/07/2023

[rosemaryjoconnor]

Code for new endpoint looking good. Issue with extracting data for fields from API. JSON sting contains mix of single and double quotes json.loads errors

[rosemaryjoconnor]

Data extracted successfully. QM following up on missing collectionCodes

[rosemaryjoconnor]

Problems with collectionCode being absent. This may be an ongoing problem. Need to discuss re new key fields and UUID implications

[rosemaryjoconnor]

900K+ records now available in the database Running on databox to test records Will run fetcher on databox-dev also to test. Check of duplicate and missing records to finalise

[rosemaryjoconnor]

900K+ records need multi-threading to extract. Mahmoud has updated the fetcher ready for testing, but the extract will take over 4 hours. Meeting with QM to be scheduled to discuss

[rosemaryjoconnor]

• QMFetcher is now functioning well - downloading in approximately 4-5 hours.
• Minor issue still with inconsistencies in number of records downloaded vs number written to occurrences CSV.
• Provide QM with duplicates list and list of records missing key fields. This has been addressed at their end and we have drop duplicates in place in fetcher.
• Analysis of image name updates - comparing API with current Prod is underway

[rosemaryjoconnor]

Databox:

• 900,267 occurrence records ingested
• 20,403 images uploaded (as per images-test)
• biocache-test only 38 occurrence records showing as having images associated

[rosemaryjoconnor]

As at 01/11/2023 There were: 4,936 occurrence records with associated images over 3000 unique duplicates in the dataset

[rosemaryjoconnor]

Waiting on update of data with new fields to populate othercatalogNumbers and fieldNotes

[peggynewman]

Great work on this! Found some unhandled collection codes:

Like this one, institutionCode = QM and collectionCode = 'fossil' ... may need an update to provider maps: https://biocache-test.ala.org.au/occurrences/66409d3c-ef31-4004-a217-ad277e50e757

[rosemaryjoconnor]

Provider maps are in Prod, but will double check again and set up any that are missing. Unable to create in Test: Create ProviderMap Property [userLastModified] of class [Collection] cannot be null Property [Most southern
latitude] of class [Collection] cannot be null Property [Most northern
latitude] of class [Collection] cannot be null Property [Most eastern
longitude] of class [Collection] cannot be null Property [Name] of class [Collection] cannot be null Property [uid] of class [Collection] cannot be null Property [Most western
longitude] of class [Collection] cannot be null

[rosemaryjoconnor]

09/02/2024 QM Fetcher live