sensitive counts not visible for recorder on website, also not in admin download

chrisvanswaay commented 4 weeks ago

@kazlauskis @DavidRoy Last Sunday (9 June) Irma Wynhoff did five counts, and marked one of them as sensitive. She uploaded all of them, and this is what the situation is now:

She sees five counts as uploaded in the app on her phone: one starting 10:38h, one 11.26h, one 12:41h, one 13:12h and one starting 14:05h
the one from 13:12h is marked as sensitive by her.
This one cannot be seen when visiting the website for others (which is fine).
On the website after logging in, she only sees four counts on that day.
A personal download by her also only shows data for the four other counts, not the one starting 13:12h.
My download as admin (or however we call it) only show four counts that day, the one from 13:12h is not in my download files. That means I cannot use this count for anything.

I don't think this is a righ way of dealing with sensive records. Sensitive records should be:

visible for the recorder in the app (this works).
visible for the recorder on the website when searching their own records (this is not the case)
in a personal download from the recorder (which is not the case).
in a download made by an admin (who has to work with the data).

This is also important for my colleages at Vlinderstichting, as we have marked some visits as sensitive, but now it means we cannot get to them anymore. That makes 15min counts useless for our professional work. I haven't tested how this works for the api.

If you agree to my interpretation of sensitive counts, could this please be corrected. And if not, then we might discuss how sensitive records should be treated.

kazlauskis commented 4 weeks ago

@johnvanbreda probably some reports need tweaking. We set privacy_precision=0 from https://github.com/BiologicalRecordsCentre/ABLE/issues/468

chrisvanswaay commented 3 weeks ago

@johnvanbreda I hope this issue can be solved (I hope it is just a simple change of a few queries) as we need the sensitive data too. Not sure how much it has been used, but most likely sensitive sites contain rare and threatened species, just what we need the 15 min counts for.

johnvanbreda commented 2 weeks ago

I've changed the required configuration so that the private samples should now be visible. For the occurrences, they are ready, for the samples it may take another hour to update. Also note that the samples I could find for Irma were indexed in France so won't appear on your scheme download.

Please close if OK.

chrisvanswaay commented 2 weeks ago

@johnvanbreda Thanks. I have access to all 15 min counts via download. I think we are almost there.

I can see all observations of that visit in the occurrences download. As there is a rare species involved, I could search for that.
However at the moment I cannot find the sensitive sample in the download of the samples. The Visit Sample ID==26100171
Irma also does not see this visit in her https://butterfly-monitoring.net/mydata/samples

chrisvanswaay commented 1 week ago

@johnvanbreda We also discovered that sensitive species are not included in personal downloads by recorders (so a recorder does a 15 min count, sees a species marked as sensitive, and this does not show up in the download). @DavidRoy This influences our professional work. Let me know if this cannot be fixed soon, because then we have to find another way (or back to paper). We started relying on the app.

Irmawyn commented 1 week ago

If possible, this problem needs to be solved immediately. The sensitive species have finally started to appear and need to be recorded. We need the data quite fast. Thanks in advance for working on this problem

Verzonden vanuit Outlook voor iOShttps://aka.ms/o0ukef

Van: Chris van Swaay @.> Verzonden: Wednesday, June 26, 2024 9:01:34 PM Aan: BiologicalRecordsCentre/ABLE @.> CC: Subscribed @.***> Onderwerp: Re: [BiologicalRecordsCentre/ABLE] sensitive counts not visible for recorder on website, also not in admin download (Issue #691)

@johnvanbredahttps://github.com/johnvanbreda We also discovered that sensitive species are not included in personal downloads by recorders (so a recorder does a 15 min count, sees a species marked as sensitive, and this does not show up in the download). @DavidRoyhttps://github.com/DavidRoy This influences our professional work. Let me know if this cannot be fixed soon, because then we have to find another way (or back to paper). We started relying on the app.

— Reply to this email directly, view it on GitHubhttps://github.com/BiologicalRecordsCentre/ABLE/issues/691#issuecomment-2192436727, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AK6ABCQL4GXC5PAKYYOKYWTZJMFY5AVCNFSM6AAAAABJECINWKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJSGQZTMNZSG4. You are receiving this because you are subscribed to this thread.Message ID: @.***>

BELANGRIJK: Deze e-mail is afkomstig van buiten de organisatie. Klik niet op links of open geen bijlagen tenzij u de afzender herkent en zeker weet dat de inhoud veilig is.

johnvanbreda commented 1 week ago

Am working on this now...

Irmawyn commented 1 week ago

Thank You!

Dit bericht is verzonden door / This message was sent by

Dr. Irma Wynhoff

De Vlinderstichting / Dutch Butterfly Conservation

Butterfly Conservation Europe

Postbus 506 / P.O. Box 506

NL - 6700 AM Wageningen

Mennonietenweg 10, Wageningen

[T] (+31) 0317 467346 / direct (+31) 0317 467320 / mob (+31) 06 44 67 46 49

[E}: @.**@.>,

[I]: vlinderstichting.nlhttp://www.vlinderstichting.nl/ bc-europe.ehttp://www.bc-europe.org/u facebook.com/bluesinthemarshes/http://www.facebook.com/bluesinthemarshes/

Aan de inhoud van dit e-mailbericht kunnen geen rechten worden ontleend. De informatie is uitsluitend bestemd voor de geadresseerde. De Vlinderstichting staat niet in voor de juiste en volledige overbrenging van de inhoud van een verzonden e-mailbericht noch voor tijdige aankomst daarvan.

No rights may be claimed from the contents of this e-mail. The information is only intended for the addressee. Dutch Butterfly Conservation cannot be held responsible for the correctness and completeness of the contents of an e-mail, nor for its arrival on time.

Van: John van Breda @.> Verzonden: maandag 1 juli 2024 12:56 Aan: BiologicalRecordsCentre/ABLE @.> CC: Irma Wynhoff @.>; Comment @.> Onderwerp: Re: [BiologicalRecordsCentre/ABLE] sensitive counts not visible for recorder on website, also not in admin download (Issue #691)

Am working on this now...

— Reply to this email directly, view it on GitHubhttps://github.com/BiologicalRecordsCentre/ABLE/issues/691#issuecomment-2199838739, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AK6ABCRKQLRF4OWEZZBZDWDZKEYWVAVCNFSM6AAAAABJECINWKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJZHAZTQNZTHE. You are receiving this because you commented.Message ID: @.***>

BELANGRIJK: Deze e-mail is afkomstig van buiten de organisatie. Klik niet op links of open geen bijlagen tenzij u de afzender herkent en zeker weet dat de inhoud veilig is.

johnvanbreda commented 1 week ago

FYI there is a need to populate the private samples into the Elasticsearch to make them available for reporting - this is running now but will take a few hours to complete.

Irmawyn commented 1 week ago

Thanks! You make my students very happy, and me as well! I will test it tomorrow.

Dit bericht is verzonden door / This message was sent by

Dr. Irma Wynhoff

De Vlinderstichting / Dutch Butterfly Conservation

Butterfly Conservation Europe

Postbus 506 / P.O. Box 506

NL - 6700 AM Wageningen

Mennonietenweg 10, Wageningen

[T] (+31) 0317 467346 / direct (+31) 0317 467320 / mob (+31) 06 44 67 46 49

[E}: @.**@.>,

[I]: vlinderstichting.nlhttp://www.vlinderstichting.nl/ bc-europe.ehttp://www.bc-europe.org/u facebook.com/bluesinthemarshes/http://www.facebook.com/bluesinthemarshes/

Aan de inhoud van dit e-mailbericht kunnen geen rechten worden ontleend. De informatie is uitsluitend bestemd voor de geadresseerde. De Vlinderstichting staat niet in voor de juiste en volledige overbrenging van de inhoud van een verzonden e-mailbericht noch voor tijdige aankomst daarvan.

No rights may be claimed from the contents of this e-mail. The information is only intended for the addressee. Dutch Butterfly Conservation cannot be held responsible for the correctness and completeness of the contents of an e-mail, nor for its arrival on time.

Van: John van Breda @.> Verzonden: maandag 1 juli 2024 16:04 Aan: BiologicalRecordsCentre/ABLE @.> CC: Irma Wynhoff @.>; Comment @.> Onderwerp: Re: [BiologicalRecordsCentre/ABLE] sensitive counts not visible for recorder on website, also not in admin download (Issue #691)

FYI there is a need to populate the private samples into the Elasticsearch to make them available for reporting - this is running now but will take a few hours to complete.

— Reply to this email directly, view it on GitHubhttps://github.com/BiologicalRecordsCentre/ABLE/issues/691#issuecomment-2200244088, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AK6ABCSEWGGRBX3HIBUUZALZKFOV3AVCNFSM6AAAAABJECINWKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBQGI2DIMBYHA. You are receiving this because you commented.Message ID: @.***>

BELANGRIJK: Deze e-mail is afkomstig van buiten de organisatie. Klik niet op links of open geen bijlagen tenzij u de afzender herkent en zeker weet dat de inhoud veilig is.

Irmawyn commented 6 days ago

Hi, I tested it yesterday and today again. The 15 min counts with sensitive species are still not visible on the website, but I found them in the downloads (and they are visible on my cell phone). Is it possible to also make them visible on the website? Thanks, Irma

Dit bericht is verzonden door / This message was sent by

Dr. Irma Wynhoff

De Vlinderstichting / Dutch Butterfly Conservation

Butterfly Conservation Europe

Postbus 506 / P.O. Box 506

NL - 6700 AM Wageningen

Mennonietenweg 10, Wageningen

[T] (+31) 0317 467346 / direct (+31) 0317 467320 / mob (+31) 06 44 67 46 49

[E}: @.**@.>,

[I]: vlinderstichting.nlhttp://www.vlinderstichting.nl/ bc-europe.ehttp://www.bc-europe.org/u facebook.com/bluesinthemarshes/http://www.facebook.com/bluesinthemarshes/

Aan de inhoud van dit e-mailbericht kunnen geen rechten worden ontleend. De informatie is uitsluitend bestemd voor de geadresseerde. De Vlinderstichting staat niet in voor de juiste en volledige overbrenging van de inhoud van een verzonden e-mailbericht noch voor tijdige aankomst daarvan.

No rights may be claimed from the contents of this e-mail. The information is only intended for the addressee. Dutch Butterfly Conservation cannot be held responsible for the correctness and completeness of the contents of an e-mail, nor for its arrival on time.

Van: Irma Wynhoff @.> Verzonden: maandag 1 juli 2024 16:13 Aan: BiologicalRecordsCentre/ABLE @.> Onderwerp: Re: [BiologicalRecordsCentre/ABLE] sensitive counts not visible for recorder on website, also not in admin download (Issue #691)

Thanks! You make my students very happy, and me as well! I will test it tomorrow.

Dit bericht is verzonden door / This message was sent by

Dr. Irma Wynhoff

De Vlinderstichting / Dutch Butterfly Conservation

Butterfly Conservation Europe

Postbus 506 / P.O. Box 506

NL - 6700 AM Wageningen

Mennonietenweg 10, Wageningen

[T] (+31) 0317 467346 / direct (+31) 0317 467320 / mob (+31) 06 44 67 46 49

[E}: @.**@.>,

[I]: vlinderstichting.nlhttp://www.vlinderstichting.nl/ bc-europe.ehttp://www.bc-europe.org/u facebook.com/bluesinthemarshes/http://www.facebook.com/bluesinthemarshes/

Aan de inhoud van dit e-mailbericht kunnen geen rechten worden ontleend. De informatie is uitsluitend bestemd voor de geadresseerde. De Vlinderstichting staat niet in voor de juiste en volledige overbrenging van de inhoud van een verzonden e-mailbericht noch voor tijdige aankomst daarvan.

No rights may be claimed from the contents of this e-mail. The information is only intended for the addressee. Dutch Butterfly Conservation cannot be held responsible for the correctness and completeness of the contents of an e-mail, nor for its arrival on time.

Van: John van Breda @.> Verzonden: maandag 1 juli 2024 16:04 Aan: BiologicalRecordsCentre/ABLE @.> CC: Irma Wynhoff @.>; Comment @.> Onderwerp: Re: [BiologicalRecordsCentre/ABLE] sensitive counts not visible for recorder on website, also not in admin download (Issue #691)

FYI there is a need to populate the private samples into the Elasticsearch to make them available for reporting - this is running now but will take a few hours to complete.

— Reply to this email directly, view it on GitHubhttps://github.com/BiologicalRecordsCentre/ABLE/issues/691#issuecomment-2200244088, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AK6ABCSEWGGRBX3HIBUUZALZKFOV3AVCNFSM6AAAAABJECINWKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBQGI2DIMBYHA. You are receiving this because you commented.Message ID: @.***>

BELANGRIJK: Deze e-mail is afkomstig van buiten de organisatie. Klik niet op links of open geen bijlagen tenzij u de afzender herkent en zeker weet dat de inhoud veilig is.

johnvanbreda commented 4 days ago

Hi @Irmawyn - please can you let me know which page you are expecting to see them on the website? Do you mean https://butterfly-monitoring.net/mydata/samples or somewhere else?

Irmawyn commented 4 days ago

Hi @John van Breda! I mean exactly that page on the website!

Verzonden vanuit Outlook voor iOShttps://aka.ms/o0ukef

Van: John van Breda @.> Verzonden: Friday, July 5, 2024 4:42:39 PM Aan: BiologicalRecordsCentre/ABLE @.> CC: Irma Wynhoff @.>; Mention @.> Onderwerp: Re: [BiologicalRecordsCentre/ABLE] sensitive counts not visible for recorder on website, also not in admin download (Issue #691)

Hi @Irmawynhttps://github.com/Irmawyn - please can you let me know which page you are expecting to see them on the website? Do you mean https://butterfly-monitoring.net/mydata/samples or somewhere else?

— Reply to this email directly, view it on GitHubhttps://github.com/BiologicalRecordsCentre/ABLE/issues/691#issuecomment-2211000879, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AK6ABCUIZMWLFPIC7BU46A3ZK2WF7AVCNFSM6AAAAABJECINWKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJRGAYDAOBXHE. You are receiving this because you were mentioned.Message ID: @.***>

BELANGRIJK: Deze e-mail is afkomstig van buiten de organisatie. Klik niet op links of open geen bijlagen tenzij u de afzender herkent en zeker weet dat de inhoud veilig is.

johnvanbreda commented 1 day ago

Hi @Irmawyn - thanks for confirming. Can you try again please - I think it should be OK now?

BiologicalRecordsCentre / ABLE