search

BiologicalRecordsCentre / ABLE

Assessing ButterfLies in Europe project repository
2 stars 3 forks source link

Downloads>Samples - Weird "Recorder Name" changes #491

Open xaviermestdagh opened 1 year ago

xaviermestdagh commented 1 year ago

Hello @JimBacon

I have a weird wrong recorder name in the "sample (visit)..." download csv. image

"Hellers, Marcel" is the good name and well spelled in his user details "Helelrs, Marcel" is found in the sample (visit) download, but only for these two visits.

The good name is however used when looking in the other option to export records (https://butterfly-monitoring.net/annual-report?locationID=branch:81&userID=all&year=2017) https://butterfly-monitoring.net/annual-report?locationID=branch:81&userID=all&year=2017 As well as when looking in the editing view of the records....

xaviermestdagh commented 1 year ago

Likely linked to reported here above: From the sample (visit) download this visit is reported as recorded by "De Sousa, Tiago" (which I beleive is the good value) image

When editing the visit, the visit is recorded by me image

And when looking in the annual report, there is no Recorder Name image

johnvanbreda commented 1 year ago

@JimBacon I'll have a quick look...

johnvanbreda commented 1 year ago

The problem underlying the first issue is that this visit had been deleted from the system, but the process which updates the reporting data (Elasticsearch/Logstash) was not configured to handle deletions in the samples reports. I've now added the missing configuration so that deleted samples (visits) will get removed.

The second issue relates to the sample/visit not having an explicit recorder defined, so the system uses the logged in user. Presumably the annual report needs to be tweaked to reflect this and the data entry form configured to not set a default for the explicit recorder name when editing a visit that doesn't have one.

SarahVray commented 1 year ago

Hello @johnvanbreda, linked to this issue, I noticed that the name of the recorder is not written the same way depending on whether it is encoded on butterfly-monitoring.net or on ButterflyCount app. If I am not mistaken, on butterfly-monitoring.net it is "Name, Firstname" (e.g. Vray, Sarah) and on the app "Firstname Name" (e.g. Sarah Vray). Do you think it would be possible to have the same "format" between the two encoding systems so that we avoid duplicates of recorders in the exports? Thanks.

xaviermestdagh commented 6 months ago

hi @johnvanbreda would it be however possible to delete these 5 records?

<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns="http://www.w3.org/TR/REC-html40">

Transect Sample ID | Date | Transect ID | Transect Name | Transect Code | Output sref. | Recorded by -- | -- | -- | -- | -- | -- | -- 12099100 | 09/06/2016 |   |   |   | 49.86844N 5.79648E | Helelrs, Marcel 12099125 | 05/05/2016 |   |   |   | 49.86432N 5.93989E | Helelrs, Marcel 12099128 | 09/06/2016 |   |   |   | 49.86844N 5.79648E | Helelrs, Marcel 12099131 | 09/05/2017 |   |   |   | 49.80419N 5.95903E | Helelrs, Marcel 12099133 | 08/05/2016 |   |   |   | 49.86432N 5.93989E | Helelrs, Marcel

andrewvanbreda commented 6 months ago

Hi @xaviermestdagh John asked me to help you with your request. Those have been deleted now. I hope that helps.

xaviermestdagh commented 6 months ago

Thanks @andrewvanbreda , but I just made a new export and these 5 lines are still there in the "...sample (visit)..." table.

andrewvanbreda commented 6 months ago

Hi @xaviermestdagh Is that done from the Downloads page on the website? It is likely the problem you are facing is that the data you are downloading is from Elasticsearch, that would mean the data you are seeing is a copy of what is in the actual database and probably has a delay before it is fully updated. If you could confirm how you do your data export, I can then ask someone how long we might expect it to be before ElasticSearch gets its data index updated.

xaviermestdagh commented 6 months ago

thanks @andrewvanbreda Using https://butterfly-monitoring.net/downloads, the records are still there at the time I write this message.

But if I export through https://butterfly-monitoring.net/annual-report?locationID=scheme:81&dataSet=scheme:81&year=2016 then I don't see them anymore.

I can try again tomorrow to see if it is finally deleted in the first download. Thanks!

andrewvanbreda commented 6 months ago

Hi @xaviermestdagh This sounds like the issue then. I have asked John how long we can expect the Index to take before it is updated.

andrewvanbreda commented 6 months ago

Hi @xaviermestdagh John said it is possible that the update task has got stuck. Anyway, have a go tomorrow, and let me know how you get on.

xaviermestdagh commented 6 months ago

@andrewvanbreda that's great these records are not anymore displayed in the export of this morning! Thanks!

andrewvanbreda commented 6 months ago

Hi @xaviermestdagh That's great. Thanks for letting me know :)