Fix handling of protected records

[JimBacon]

There are workflow rules which cause occurrences of certain taxa to be marked confidential or sensitive when recorded. There is also an outstanding issue to be able to mark samples as private.

• Private is intended for protecting locations (i.e. at the sample level)
• Sensitive is intended for protecting taxa (i.e. at the occurrence level)
• Confidential is intended for protecting communication around an occurrence.

Making records Private or Sensitive causes the location information used for public display to be blurred. Confidential records should not be publicly displayed at all.

Known issues

• When a sample contains multiple occurrences, some of which are sensitive, you can view the location of the non-sensitive occurrences at full resolution and get a clue about where the sensitive species probably are.
• The request to be able to mark samples private also asks them to be hidden from public view.
• Samples having confidential records cannot be viewed, even by the recorder, and are excluded from downloads including by scheme organisers.
• There are records of taxa which are not confidential but should be.
• There are taxa that should be protected for which no workflow rule exists.

[szabadfalvi]

I am not sure, i got the point of these intricate masking categories. Wouldn't be enough to use Sensitive Site (Site level) + Sensitive Sp. (occurrence level), in parallel with masking the corresponding tracks (for single sp. 15 min surveys)?

[JimBacon]

The three categories are part of the standard recording software that we have used to build the EBMS website. It has evolvled over many years to handle all the possibilites we have encountered. EBMS may not need them all.

If we are clear about the different ways we want to restrict records then it should be clear which features are best to use. Currently we have workflow rules which make some occurrences Sensitive and some Confidential.

[JimBacon]

To review all the current pages where occurrences can be accessed, to see how the pages handle sensitivity and confidentiality, I created training records in both the transect survey and the 15-minute survey for

• Coenonympha phryne which will be given a sensitivity of 10000 by a workfolw rule
• Euphydryas maturna which will be marked confidential by a workflow rule
• Aglais io which is not restricted.

The general findings were:

• Pages driven by Elasticsearch (My Samples, My Downloads, Explore online data, Summary by Country & Year, Verification, Downloads) never include confidential occurrences or samples having confidential occurrences.
• Some Elasticsearch pages (Explore online data, Verification) list sensitive records twice, once blurred, once at full resolution.
• Elasticsearch pages respect the training flag
• Pages that use reports querying the database, rather than Ealsticsearch, do not respect the training flag. This issue has already been raised.

[JimBacon]

Continuing investigation, there are 3 Elasticsearch endpoints in use: es-able, es-able-admin, es-able-samples-admin.

These map on to 3 aliases on Elasticsearch indexes having filter criteria in their definitions as follows:

es-able Maps to alias occurrence_search_able on the index occurrence_brc1_v1

metadata.confidential:false AND 
metadata.release_status:R ((metadata.sensitivity_blur:B) OR (!metadata.sensitivity_blur:*)) AND
metadata.website.id: [118,140]

es-able-admin Maps to aliasoccurrence_search_able_admin on the index occurrence_brc1_v1

metadata.confidential:false AND 
metadata.release_status:R AND 
((metadata.sensitivity_blur:F) OR (!metadata.sensitivity_blur:*)) AND
metadata.website.id: [118,140]

es-able-samples-admin Maps to alias occurrence_search_able_samples_admin on the index sample_brc1_v1

metadata.confidential:false AND 
metadata.release_status:R AND 
((metadata.sensitivity_blur:F) OR (!metadata.sensitivity_blur:*)) AND
metadata.website.id: 118

Various things leap out.

• All endpoints exclude confidential records, and unreleased records.
• The admin endpoints are for showing sensitive records at full precision,
• The non-admin endpoint is for showing sensitive records with reduced resolution,
• There is a missing AND in the non-admin endpoint which will be causing the duplicate records.
• The endpoints to the occurrence index include EBMS and EU Pollinator Monitoring Scheme records. This arises from https://github.com/BiologicalRecordsCentre/ABLE/issues/509#issuecomment-1467651371

However, removing the filter on confidential records does not make them magically appear. This is because client_helpers/ElasticsearchProxyHelper.php::buildEsQueryFromRequest() contains the following:

    if (!self::$confidentialFilterApplied) {
      // Unless explicitly specified in a filter, hide confidential.
      $bool['must'][] = ['term' => ['metadata.confidential' => FALSE]];
    }

This is requiring a filter to be created explicitly giving the user permission to see confidential records.

[JimBacon]

Another thing to look into. Some records of a species that have a workflow rule to mark them sensitive have the flag set and some do not. For example compare

• https://butterfly-monitoring.net/mydata/details/record?occurrence_id=30104383
• https://butterfly-monitoring.net/mydata/details/record?occurrence_id=29635010

[CrisSevilleja]

Why are they different on the sensitivity flag? same species, in the same area

[JimBacon]

I have resolved the duplication of sensitive records and the vanishing confidential records by making the following changes:

• The ElasticSearch filter on alias occurrence_search_able was changed to

  ((metadata.sensitivity_blur:B) OR (!metadata.sensitivity_blur:*)) AND
  metadata.website.id: [118,140]

• The ElasticSearch filter on alias occurrence_search_able_admin was changed to

  ((metadata.sensitivity_blur:F) OR (!metadata.sensitivity_blur:*)) AND
  metadata.website.id: [118,140]

• The ElasticSearch filter on alias occurrence_search_able_samples_admin was changed to

  ((metadata.sensitivity_blur:F) OR (!metadata.sensitivity_blur:*)) AND
  metadata.website.id: 118

• The ElasticSearch endpoint for the verification page was changed from es-able to es-able-admin
• The code fix to allow filtering of ElasticSearch sources by the confidential field was deployed to the website.
• The Form Structure was updated to include the following filter on the pages
  • My Data > My Samples,
  • My Data > My Downloads,
  • Scheme Admin > Downloads
  • Scheme Admin > Verification
  • Scheme Admin > Summary by country and year

    <input type="hidden" class="es-filter-param" data-es-bool-clause="must" data-es-field="metadata.confidential" data-es-query-type="term" value="all" />

[DavidRoy]

@CrisSevilleja - the main requirements of this are now met. Can you confirm with @szabadfalvi and others. We'll then close this issue and open new issues for related tasks

[szabadfalvi]

Hi All, Yes, it seems that sensitive spp. (oedippus + maturna i checked) are now included in the downloads (as well as the Life Stage). This issue seems "swell" now :-) &

On Wed, Aug 30, 2023 at 11:22 AM David Roy (CEH) @.***> wrote:

@CrisSevilleja https://github.com/CrisSevilleja - the main requirements of this are now met. Can you confirm with @szabadfalvi https://github.com/szabadfalvi and others. We'll then close this issue and open new issues for related tasks

— Reply to this email directly, view it on GitHub https://github.com/BiologicalRecordsCentre/ABLE/issues/594#issuecomment-1698808619, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHE6CHM5BBW5YNMECGXEYDXX4A53ANCNFSM6AAAAAAZW4DJYE . You are receiving this because you were mentioned.Message ID: @.***>