recordedById duplicates

[kylie-m]

When there are multiple identifiers, each identifier is duplicated. dwc:identifiedByID does not suffer from this problem.

In the ‘original vs processed’ window, dwc:recordedByID is only in the ‘original’ column, including the duplicated identifiers.

[adam-collins]

• raw_recordedByIds is not used https://biocache.ala.org.au/ws/occurrences/search?q=raw_recordedByIds:*
• recordedByID is a multivalue field. It appears it does contain duplicates https://biocache-ws-test.ala.org.au/ws/occurrence/5623cce5-98af-4723-a332-517e0b575d94
• raw_identifiedByIds is not used .
• identifiedByID is a multivalue field but it does not appear to be used in that way e.g. https://biocache.ala.org.au/ws/occurrence/798c62b1-9196-49f2-8fc4-c69a59a1d95f

Please confirm what needs fixing. I interpret the request as:

1. Continue to not use raw_identifiedByIds and raw_recordedByIds
2. Treat recordedByID the same as identifiedByID, i.e. only put the raw value into this field once instead of the current twice.

[adam-collins]

@djtfmartin, @peggynewman please check that I understand the intention of this request as noted above.

The code indicates there was some work this year to move towards using the raw_ fields as well as using the processed value in the ID fields.

branch https://github.com/gbif/pipelines/tree/848_duplicates_in_recordedbyID

[adam-collins]

Changed my mind. Searches will work better if you can search for individual IDs.

[adam-collins]

Pull request https://github.com/gbif/pipelines/pull/983

[adam-collins]

in version 2.18.0-SNAPSHOT

[adam-collins]

Test with https://biocache-ws-test.ala.org.au/ws/occurrence/5623cce5-98af-4723-a332-517e0b575d94, no duplicate recordedByID values https://biocache-ws.ala.org.au/ws/occurrence/5623cce5-98af-4723-a332-517e0b575d94, duplicate recordedByID values

[kylie-m]

Thanks Adam, working correctly for me. Also looking great in the UI - Testing passed!

[peggynewman]

LGTM