search

AtlasOfLivingAustralia / biocache-service

Occurrence & mapping webservices
https://biocache-ws.ala.org.au/ws/
Other
9 stars 26 forks source link

Download fails when including Wildnet Taxon ID #891

Open timhicks-ala opened 3 months ago

timhicks-ala commented 3 months ago

Steps to reproduce:

  1. See this record search for a small area in remote northern NSW, about 25k records: https://biocache.ala.org.au/occurrences/search?&q=qid%3A1710128021567&disableAllQualityFilters=true#tab_mapView
  2. Perform a custom download and include the last Miscellaneous field, Wildnet Taxon ID
  3. The download will start, but end quickly. An email will be sent saying the download has failed, but the website will say it finished successfully and provide a link to the download. That zip file will contain a very small CSV file with a handful of records, no citation information, and no DOI will be minted/communicated.

It looks as though this list is used for matching data for the Wildnet Taxon ID field: https://lists.ala.org.au/speciesListItem/list/dr15515

It currently lists 72 unmatched taxa. It's possible the issue is caused by attempting a custom download with Wildnet IDs including a species that doesn't match on this list. Currently this is affecting any custom download where Wildnet Taxon ID is selected, whether individually or as part of a "select all" action, where the search includes a problem species in the list.

There are multiple issues here:

  1. Missing matches in this list should probably result in empty fields (or a note about not finding a match) in the download and a graceful failure of that particular field rather than a download error.
  2. It's causing downloads to fail in a particularly ungraceful manner. It looks as though it's processing records for download, stops when it hits an unrecognised one in the Wildnet list, sends the error email but then publishes the nearly-empty zip file to the download webpage. This is an unusual edge case in download failures.
  3. The above list was last updated in 2020. Possibly there is an updated list available.

Raised in https://support.ehelp.edu.au/a/tickets/197135

adam-collins commented 3 months ago

This issue is due to Kvp arrays differing in length.

Must normalise the length of Kvp arrays in ListsService::getKvp

adam-collins commented 2 months ago

For testing use this test url)#tab_mapView)

timhicks-ala commented 2 months ago

For testing use this test url)#tab_mapView)

I've attempted to test this but the WildNet Taxon ID column doesn't appear in a custom download CSV for me in test. Other Miscellaneous fields do appear, but not the WildNet one.

adam-collins commented 2 months ago

Please test again. The WildNet list was not in the test environment. I have added it.

timhicks-ala commented 2 months ago

Can confirm WildNet Taxon IDs are now appearing in custom downloads in test.

wildnet