dwc:establishmentMeans and dwc:degreeOfEstablishment

[kylie-m]

There is now a TDWG controlled vocabulary for dwc:establishmentMeans, which has fewer terms than the GBIF vocabulary that was used previously.

dwc:degreeOfEstablishment was split off from dwc:establishmentMeans in early 2020.

Pipelines enforces the TDWG vocabulary for dwc:establishmentMeans, so, if the provided value is ‘naturalised’, ‘adventive’ or ‘managed’ (or any value that is not in the vocabulary) there will be no processed value. However, they can still be downloaded as raw_establishmentMeans.

ALA does not support dwc:degreeOfEstablishment yet, so providers who do the right thing do not get their data forwarded to GBIF correctly and people cannot download the data.

The ‘Establishment means’ facet has become less useful, because of the missing values, although it has never been very useful, because establishmentMeans and degreeOfEstablishment are rather spottily recorded (this is in the nature of the data, as these are assertions that can often not be made by the recorders or people who enter the data).

Add degreeOfEstablishment everywhere that establishmentMeans appears: SOLR schema, fields, facet search, downloads. Check for pipelines processing of the terms.

• raised by @peggynewman

[adam-collins]

pull request https://github.com/gbif/pipelines/pull/980

[adam-collins]

In version 2.18.0-SNAPSHOT

[adam-collins]

Test that data exists for this query https://biocache-ws-test.ala.org.au/ws/occurrences/search?q=degreeOfEstablishment:*&facets=degreeOfEstablishment

[kylie-m]

Thanks Adam, that's working for me. FYI @peggynewman

[nielsklazenga]

This issue is not fixed:

• https://biocache-ws-test.ala.org.au/ws/occurrences/search?q=degreeOfEstablishment:*&facets=degreeOfEstablishment works, but https://biocache-ws.ala.org.au/ws/occurrences/search?q=degreeOfEstablishment:*&facets=degreeOfEstablishment gives an error:

  {
  "message": "Error from server at null: undefined field: \"degreeOfEstablishment\"",
  "errorType": "Query syntax invalid",
  "statusCode": 400
  }

• degreeOfEstablishment does not display on record details pages
• neither degreeOfEstablishment nor raw_degreeOfEstablishment can be downloaded. I could download raw_establishmentMeans before, so we've gone a step backwards.

[adam-collins]

@nielsklazenga Thank you for testing.

I cannot download raw_establishmentMeans in test or production. raw_degreeOfEstablishment is the same because it is consistent. Luckily not going backwards. I'll create a new issue so as not to confuse this pipelines change with downstream changes required.

[nielsklazenga]

Thanks @adam-collins. I used to be able to get raw_establishmentMeans and raw_degreeOfEstablishment through the offline downloads, see https://github.com/vicflora/vicflora-mapper/blob/504563365d4084eff1580e415e32514a072d4896/app/Actions/DownloadOccurrenceData.php.

They probably disappeared at an earlier stage though, as I found out two weeks ago that while I have been downloading the AVH data every week, it was not uploaded in my system, and now raw_eventDate cannot be downloaded anymore either, so a new issue is probably indeed the way to go.

[peggynewman]

@adam-collins I can't see the degreeOfEstablishment field in a record either on the record or api page. Looked in the first one of these results: https://biocache-ws-test.ala.org.au/ws/occurrences/search?q=degreeOfEstablishment:*&facets=degreeOfEstablishment Happy to be wrong

[adam-collins]

It is absent. As with the text_ field added by the collection of pipelines changes in test, the intention was to wait for pipelines changes to get to production before creating the next set of issues.

[peggynewman]

Ok so we keep this open until next round then.