Users' data download report to include other peoples' samples for their plots

[sacrevert]

A user’s data download options could be amended to incorporate samples not made by them for their plots (this is for when a user inherits plots and wants to have a summary of what was recorded). Possibly could also be dealt with by #65

[andrewvanbreda]

AVB notes for meeting. Things to consider about this.

1. The current report allows download of occurrences related to samples (visits) you created. This may be confusing if it becomes a mixture. One solution is to perhaps have 2 downloads?

2. Indicia has permissions on its easy_download forms that prevent users from downloading other people's data unless they have the required permission. This could probably be bypassed, but need considering.

3. Make sure you only let users download data that you will always be happy for them to have going forward, as once download is made available, rights to retract the rights to that data are also lost.

4. Is a download necessary, would perhaps a better way forward be to make extra information about the samples viewable on screen?

[sacrevert]

Preferred solution is a drop-down filter under My Data menu option. page title is currently "Download my records for reporting" but can change to "Plot data downloads" Help text "Choose whether you wish to download your records, or records that other surveyors have made for your plots" Options would be "My records", "Other peoples' records".

[sacrevert]

@andrewvanbreda to do as Priority

[andrewvanbreda]

Hi @sacrevert @NPMSSupport ,

I have had a look at this issue and the existing download form.

This page currently uses Indicia's built-in Easy Download form which I don't think can be made to work in the suggested way.

It the current page is a form that just has one report, and it is the Warehouse that decides what rights the user has to view records. This is how admins and users can use same page.

So what we are after is far away from the way the page currently works, I cannot just change the form, also I can't really add such project specific functionality to a general Indicia form as it will make the form a mess for other projects.

So I have two thoughts on this, with option 1 being my preference.

1. Rework/Extend with My Visits area. Perhaps call it "My Visits and Records". Include 4 grids (perhaps on separate tabs, or accessed using a link from the page). So My visits My occurrences Other peoples' visits for my plots Other peoples' occurrences for my plots

(obviously the terminology can be varied).

The grids can have more columns shown in the download files than in the grid on screen.

One limitation here would be that there would only be a CSV file type

2. Write a completely new screen which looks more like the existing download page but works in the way you originally specified. This would be a lot of extra effort (budget) though.

Andy

[sacrevert]

hi @andrewvanbreda sorry for delay responding, I am happy with option 1. @NPMSSupport can you confirm that you are happy to? We have this item in our 2023/24 workplan, so we do need to decide this and deliver fairly soon, ideally before March 2024 of course. Thanks!

[andrewvanbreda]

Hi @sacrevert @NPMSSupport ,

I have gone ahead and created a beta version of the download page.

Note although I have done some testing on it, I am still expecting there to be niggles.

I decided to do this for a few reasons.

1. This is Priority issue and I think there is a danger of it getting left and other tasks then using up the budget.

2. I had to create most of reports anyway, regardless of which suggestion you went for. Setting up the map, grids is the least problematic bit.

I did also try to take into account the column suggestions here, although I did get bit confused about the suggestion Output map ref. https://github.com/BiologicalRecordsCentre/NPMS/issues/226

Note that on the occurrence grids their are more columns in the download file than on the grids due to space.

Let me know thoughts. Here is the page.

https://avb-downloa-brc-npms-d10.pantheonsite.io/content/my-visits

Andy

[sacrevert]

See the comment about output/original grid ref here: https://github.com/BiologicalRecordsCentre/NPMS/issues/226#issuecomment-1016581090 Maybe that is irrelevant now due to changes to indicia fields, I don't know.

[andrewvanbreda]

H @sacravert,

Yes saw the comment.

But I see now why I got confused as I misread it.

Ignore that for now and just comment on the rest of the page if you wish.

I would suggest having the Output map ref and Output map projection in the download file and leave the Plot column on the grids (which is more consistent with the formatted display we see on the other grids but would cause unwanted html tags in the download files).

[sacrevert]

Also see comment here https://github.com/BiologicalRecordsCentre/NPMS/issues/226#issuecomment-1845154538 Maybe we can close one of these, as it's getting confusing! I don't mind which one.

[andrewvanbreda]

Copying comment from Oli into this thread

This looks nice to me @andrewvanbreda ; however, I think that the labelling of the different tabs is misleading. "Other species for my plots" and "Other visits for my plots" are not in fact those things, but are actually "Other species for other peoples' plots in my square" and "Visits to other plots in my square". I checked this for a square assigned to me, SU4287 (location_id=38937), and the information that appears under those tabs does not relate to the plots that I created, but to other plots in different locations in the square.

Maybe the current set-up includes both of these types of things, i don't know as I don't have an example to check (by which I mean, it would also include species/visits to MY plots if such visits by other people existed).

I don't know whether we want both types of species/visits listed or not. On the one hand it could confuse people (although of course they can see the existence of other plots on the https://www.npms.org.uk/my-squares-and-plots plage), on the other it would probably be of interest to them. I think we need input from others like @NPMSSupport and @Sam-Amy here before we make a decision

[andrewvanbreda]

@sacrevert I have changed the tab names for now as the logic matches what you have suggested for the tab name. They were badly named. The names are perhaps a bit long now, but we can think of a way to shorten them.

[andrewvanbreda]

Hi @sacrevert The logic I have coded is as follows. Perhaps this can be changed if you require

My Species = Any occurrence you created on NPMS, regardless of whether you are still assigned the square for the plot anymore.

My Visits = Same as My Species but for visits (samples)

"Other species for other peoples' plots in my square" = Any occurrences not created by you, but are associated with plots in squares assigned to you

"Visits to other plots in my square" = Any sample not created by you, but are associated with plots in squares assigned to you (same as above but for visits).

I think actually there may have been a confusion by what you mean by "my plot" and what I have interpreted it to mean. I think you are using it to refer to ones a user created, but I refer to it to mean a plot a user has rights to enter data for. This is also why the tabs were named incorrectly.

Note that I don't think the following situation is currently covered. If you create a plot, but then the square is assigned to someone else, then that new owner creates data for a plot you created under that square, that will not currently be shown in my report for the originally plot creator.

[Sam-Amy]

I don't know if this is different for NPMS, but worth noting that in general for iRecord only verifiers or administrators of activities to which records were submitted, have the ability to download any records other than their own (hence the restrictions you monetioned Andy). I don't personally think there is a huge amount of difference in reality to being able to see someone's records online (i.e. as an iRecord user), and being able to download them, but possibly soemthing to consider clarifying in the privacy notice? Presumably you have got round this issue now, otherwise I was going to ask whether the functionality of acitivites might be useful, i.e., each square could be like an activity and the current asignee becomes admin.

I will have a look at the page now anyway!

[sacrevert]

@Sam-Amy The main point of this functionality is specifically to allow recorders to see records created by other people; as you say, worth us making sure that the privacy notice covers this, but this is not something that Andy needs to worry about. Records with recorder names can already be downloaded from the NBN anyhow.

[sacrevert]

Note that I don't think the following situation is currently covered. If you create a plot, but then the square is assigned to someone else, then that new owner creates data for a plot you created under that square, that will not currently be shown in my report for the originally plot creator.

I think that's fine, we don't need to cover that eventuality, as "future" records for plots that someone created but then no longer surveys are not something that needs to be communicated to a past user.

Otherwise the logic you describe in fine for the moment. If it is agreed that we would like to restrict it in some way we will let you know.

[andrewvanbreda]

@sacrevert OK, now the logic is agreed, I will do more detailed tests on it because I need to do that even if @NPMSSupport say they prefer the other style of page.

[Sam-Amy]

The title "My Species" suggests a species list to me. Why is it not "My Records" or "My species records"?

The title "Other species for other peoples' plots in my square" suggests to me that only 'other species' (i.e. those not recorded by you) appear here, but if this is all occurences recorded by other people, why not "Records made by other people", or just "All records" and include both here, with a description at the top that this may include records made by other people who have previously surveyed the square? I can imagine it might be help full to see the two together in a table so you could, for example, search on a species you have recorded and see if the same was recorded previously.

And the same for visits; "My visits" and "All visits"? "Visits to other plots in my square" suggests this will be different plots (i.e. locations) to the ones you are looking at, which is confusing. If you did want to keep a longer heading, wouldn't "Other visits to plots in my square" be more appropriate?

[sacrevert]

I think we can just simplify the latter two headings to: "Others' records in my square" and "Others' visits in my square"

As Sam suggests, the first tab is perhaps also better as "My Records"

We will add some descriptive text to the page when it goes live @andrewvanbreda . We'll wait for @NPMSSupport to comment too of course

[andrewvanbreda]

Labels fixed

[andrewvanbreda]

Hi @BirenRathod Would you be able to pull the NPMS reports folder when you can a chance please. Thanks.

[BirenRathod]

@andrewvanbreda This should be updated by now.

[andrewvanbreda]

@sacrevert @NPMSSupport I have now completed my own testing/fixes for this page. The page can be found here. I will wait to hear from you about any plans regarding putting it live. https://avb-downloa-brc-npms-d10.pantheonsite.io/content/my-visits

There are just a couple of notes

1. At the time of writing I know of one bug. In the download file of "Others' records in my square", the file displays the Record Key incorrectly. I have fixed this, and this will probably be available on the site by the site you look at it. (the bug is very obvious, as you will see '#warehouse_id#' appear in every key when it is wrong)

2. New explanatory text would need to be provided by yourselves for the page.

I will await feedback.

[NPMSSupport]

Thanks @andrewvanbreda This is looking good. And I am happy as long as we are not providing any additional info than folk would otherwise see on iRecord, and as @Sam-Amy says, we have triple checked the data policy. @sacrevert If you are happy to provide the text that would be great thanks.

Apologies if I have missed this, but what will this mean for the current Data download page?

[andrewvanbreda]

@NPMSSupport It will be best to keep it, but restricted to admin roles. There is currently a Northern Ireland download page that is not replicated by the new one.

There isn't any sensitive data on the reports I dont think, as it is just the same data you would see if you owned a square. Perhaps it might be an idea for me to remove the Recorder Name column from the Other Samples/Records tabs and downloads? That is open data but it could make people nervous that other users are seeing their name on the report? Depends if that column is considered useful

[sacrevert]

@andrewvanbreda I think we can leave Recorder Name. It is covered by the data policy, and it is standard on other sites (e.g. the front page of iRecord gives recorder names)

[sacrevert]

Here is some header text (including either some paragraph spacing or the blank line as you prefer). Please check that you agree with what I have written before posting:

The grid below lists all the samples that you have personally created, even if you are no longer assigned to a square ("My Visits", "My Records").

The other tabs below list records and visits to any squares currently assigned to you, but made by other recorders. Depending on your square, this could include both historic records made within the plots that you currently survey, and/or records made by others with their own, separate, plots within the square.

[andrewvanbreda]

@sacrevert Have added the text to that test site (which will be cloned onto live once we are all happy).

[sacrevert]

@andrewvanbreda sorry, just made a minor edit, can you re-copy the text? Thanks!

[andrewvanbreda]

@sacrevert Done

[sacrevert]

I think that we are all happy for this to be cloned to live when you have a chance. Cheers

[andrewvanbreda]

@sacrevert @NPMSSupport This is live now.

The old My Visits page has been taken out of the menu, set as administrator access only, and renamed as "My visits (no longer used)" (i.e. it completely unused, but not deleted in case we need to check what the page used to do).

The old Data Downloads page has been left in the menu but set as administrator access only, renamed as "Data download (legacy version)".

Please close issue if happy.