Dataset with embargo (isPublic=false) : download button, data available, ...

[mathieuLU]

2 cases when the download button should not be displayed :

• no archive to be downloaded (how to test ? curl ?)
• dataset is saying isPublic=false

display the embargo date instead ?

in the snippet, the embargo should be mentionned

Make sure that the API cannot provide to public when isPublic=false ? how to make it available in the client then ??

maybe a subset should be provided ?

in any case, the "total records" in the dataset listing is supposed to show the real number of records of the dataset. Not what's in the database

[mathieuLU]

ok so for a public dataset (no emargo, isPublic=true), we have this now : snippet => detail =>

and so when dataset with embargo : snippet (no button + embargo date) => detail =>

does that look better @DeboraArlt @blacksparrowhawk

[DeboraArlt]

looks ok for me. perhaps would be better with an explicit statement about the data set not being currently public, but we can go for this easy solution now. and see what users say.

[mathieuLU]

ok so the question is still open. How do we deal with the datasets with embargo ? (isPublic = false) Will the data be in the database anyway with some authentication/filter needed ? How do we use the visualisation tools ? Do we only let a sample data available to download ? etc...

several options :

1- all the data is in the database, but not available to download/retrieve from the API. Which means that the web client (which is using the API) can't use the data for the visualisation tool. SO we need to specify a way for the client to access the data anyway. And do we want the client to access the first data available as usual ? (maybe a risk to display protected data ?)

2- only a sample available in the database. This way, the database remains 100% public. No need to authenticate/filter or anything. But it means that a sample has to be created when we publish the dataset. And then it has to be clearly specified somewhre that only a sample is available. Which means that we may display the "total number of records of the dataset" (for real) and the "total of records AVAILABLE in the database"

3- maybe a mix of these 2 ? hidden data in the database + public sample available for download/API ?

@blacksparrowhawk what are you views on that ?

[mathieuLU]

i add to the discussion the comment that Debora wrote :

` I don’t know exactly how it works, but for SOS they talk about the public index, end a sensitive/protected index. I thinks it’s a bit like two separate databases. Normally one get’s only data from the public index. But if you have permission you also get data from the protected index (according what your permissions allows access to).

For me option 2 works too. The private/embargo dataset provides a subset that is available public. But it seemed like Johan did not quite like this option. But let’s say the subset is only one animal. One could also select just part of the total time series, e.g. just half a year instead of a full annual cycle, i.e. a simple subset for illustrative purpose. I cannot see that anyone would/could misuse that data. `

[DeboraArlt]

From the discussion today 24 Sept (Mathieu and Debora) it sounds like the easier option is having only public data in our database. That's the data we dipslay and that one can access via the API. No permissions etc. needed. It would be good though to have a public sample data from the non-public (embargo) datasets. This sample data could be explored (graphs) and also downloaded. The data owner decides what he/she wants to release as a sample - could be data for one animal only, and for a selected time period and/or selected location only, i.e. data that the owner deems as unsensitive to show in graphs and have downloadable via the API. I suggest to add to the metadata - so metadata include information about total no. of records in the dataset, and some information about what the selected public sample is. I will have a look at our metadata form and see where this could be added.

[mathieuLU]

has been decided :

• we keep the field isPublic for the dataset. It tells if the data can be downloaded or not, and accessible or not through the API. For the whole dataset
• we add a field isPublic for each animal (default = true). when animalIsPublic =false, then the data linked to that animal can't be downloaded (excluded from the archive !!) and can't be fetched through the API.
• the field totalRecords keeps telling how many records are in the dataset, including the non-public ones
• we add a field dataAvailability at the dataset level, text field, that explains if/why there are hidden data in the dataset, and why there could be a difference between the totalRecords and the realiti in the downloadable file.

[mathieuLU]

ok so :

• field isPublic (datasetlevel) is now used as a filter on records/events in the API => https://github.com/biodiversitydata-se/biologging-sensor-API/commit/350b5456eea84346551eb6746df4dcd3fb818b80
• field isPublic is added on the organism level (and as well at the event/record level, inherited from organism) => https://github.com/biodiversitydata-se/biologging-sensor-API/commit/7b1eb94ee1c04449ef2333292c886c48fa02e8ee and https://github.com/biodiversitydata-se/biologging-sensor-API/commit/61e991902d79e28e60ed6f5ca8b0833ab44007cc
• field dataAvailability is added as well, and displayed in the snippet and the details of a dataset => https://github.com/biodiversitydata-se/biologging-sensor-client/commit/2457a8e1a1b54a5bf6c8ca27c214a8d68b40a056