Sensitive species 2: additional columns for download

[kitenetter]

Add new functionality to the csv download process for iRecord, for both ‘standard’ and ‘simple’ downloads:

• [ ] Retain the blurred grid ref in the output grid ref column
• [ ] The location name should default to “<sensitive record, location hidden>” (using “<>” prefix/suffix so that this location can can be clearly distinguished from on-sensitive sites when ordering alphabetically by location name; could use a different prefix/suffix if preferable, e.g. “ sensitive record, location hidden”)
• [ ] Add a new column “sensitive record”, set to TRUE if the record is flagged as sensitive due to taxon or user-specified sensitivity, otherwise FALSE
• [ ] Add a new column “sensitive location” for the entered location name, only populated if the record is sensitive
• [ ] Add a new column “sensitive output grid ref” for the unblurred grid ref, only populated if the record is sensitive
• [ ] Add a new column “sensitive VC” for the VC based on the precise location, only populated if the record is sensitive

To enable the above, we will need to add indexing based on the true location, as well as on the blurred.

ES data already supports a blurred and an unblurred view, will need to add a database field to enable this in Postgres.

[kitenetter]

Question: once the above is available in the standard download, will it automatically feed through to the survey-specific downloads, or will they need additional editing?

[johnvanbreda]

once the above is available in the standard download, will it automatically feed through to the survey-specific downloads, or will they need additional editing?

Yes, the survey specific downloads are the same as the standard downloads, with the addition of extra columns for custom attributes.

Some questions:

1. In the standard download, there is an existing location field called site name, so shall I call the new field sensitive site rather than sensitive location?
2. On the simple download, there is a Site field which has some special handling - replacing the value with 'unnamed site' if the location name is empty. This is one of the adaptations made so that the Simple download format would be MapMate compatible. Presumably we can alter this field and populate the value with 'sensitive record, location hidden' if the record is sensitive without breaking MapMate imports?
3. Does the sensitive location/site field get populated for all users, or only verifiers (and people with sensitive data access)?

[kitenetter]

1. Yes, let's stick to using site
2. Yes, 'sensitive record, location hidden' should work for MapMate, but note request to have a non alphabetic wrapper around the name
3. The sensitive data should be available to users downloading their own records (= anyone using 'My records' from the 'Records to access' list); to verifiers (i.e. anyone with a verification role and/or using a verification filter from the 'Records to access' list); and to LERCs (the only definition we have for these is the use of a download filter that is prefixed "LERC" in the 'Records to access' list). The difficulty is that we have other custom downloads set up for which we do not want sensitive details to be exposed, or at least not without checking first. Happy to go back to giving LERC accounts a defined role if that will make this easier to set up.

[johnvanbreda]

More questions:

1. When you don't have sensitive records access, the system blocks access to the site name at a low level, so the code can't currently know whether there is a site name or not. Therefore we can't state that the site name is being hidden as it might just not exist at all (and on the Simple download version we can't differentiate between "unnamed site" and "[sensitive record, location hidden]"). One possible solution would be to store ! (or some other similar indicator) to indicate a site name being withheld because of sensitivity and store an empty string if there was no site name, so we can differentiate. Would that be OK? This would affect the outputs on normal explore pages - the site name for a sensitive record would show ! if there is a withheld site name.
2. For the LERC download, to avoid making the permissions in Indicia yet more complex I would suggest we have a separate clone of the Download page which you give access to using an LERC role in Drupal. This would show their downloads (perhaps we need to filter to those starting LERC...) and use a custom Elasticsearch connection so that it gets the full precision version of each record. Is that OK?

[kitenetter]

1. Main concern over using "!" is that it isn't obvious what it means (e.g. to someone visiting the Explore page). But it sounds like we don't have a lot of options here, and the use of "!" does at least match the red symbol that we already use for sensitive records, so this seems workable.
2. I think that is okay. We have a role "regional collator" that doesn't currently have a function - as far as I can see that role does not grant any permissions beyond those of a standard user, so we could go back to using that role for the LERCs. But maybe it would be clearer to create a new role (or rename that old one).

All downloads for LERCs are prefixed "LERC" so that will work for filtering.

[johnvanbreda]

I've done all the coding required for this. Once deployed, I'll need to set up the special version of the download page with custom ES alias for LERCs.

[kitenetter]

@johnvanbreda let me know if you need me to start assigning a new role to the LERC user accounts.