Problem with dates when surveys are entered through, or changed in, BioCollect

[ekol-ali]

For example:

In one recent survey, imported from the Intranet function, I needed to change the date. The date was changed to the correct 21 December 2020 (the previous and erroneous was 20 January 2021).

https://biocollect.biodiversitydata.se/sft/bioActivity/index/61a37560-facf-9be2-6005-95c5b36c0e4c

But when looking in the list of surveys carried out by this person, the date of that particular survey is one day wrong, 20 December.

https://biocollect.biodiversitydata.se/sft/person/edit/339a06bc-fb62-c034-1e71-4bedb7588e0d/?returnTo=/sft/project/index/89383d0f-9735-4fe7-8eb4-8b2e9e9b7b5c#activityList

See the very last survey of the list, which says 20 December 2020.

This error does not seem to occur in the files imported once from PostgreSQL, but only for protocols entered in BioCollect, or being edited in BioCollect.

Most importantly, it should be noted that also in the data exported to Excel using Intranet, the date becomes wrong!

[mathieuLU]

Quite strange. I was able to reproduce it, definitely a bug.

Before digging int details, I have contacted the australian team, to make sure this is not a bug they have as well. We never changed anything in that part of the code, so it's a very surprising bug.

[ekol-ali]

OK, strange! Strange also if they have it and didn’t notice it. If it is not solved until I make the new PostgreSQL tables, I’ll try to remember to change it “manually” in Excel. Hm, that will be almost impossible, because it is not ALL dates that are wrong, only those reported in BioCollect. We’ll have to live with it for while, it won’t affect anything in the year report work at least.

Fortsätt med ditt fina arbete Mathieu, det känns väldigt tryggt att ha din support nu. Förhoppningsvis skall du snart få tid att göra annat än SFT-support! Åke

Från: mathieuLU @.> Skickat: Thursday, 13 January 2022 16:47 Till: biodiversitydata-se/biocollect @.> Kopia: Åke Lindström @.>; Author @.> Ämne: Re: [biodiversitydata-se/biocollect] Problem with dates when surveys are entered through, or changed in, BioCollect (Issue #179)

Quite strange. I was able to reproduce it, definitely a bug.

Before digging int details, I have contacted the australian team, to make sure this is not a bug they have as well. We never changed anything in that part of the code, so it's a very surprising bug.

— Reply to this email directly, view it on GitHubhttps://github.com/biodiversitydata-se/biocollect/issues/179#issuecomment-1012259648, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AXBWL5UBYJWRK7EVBMBYPB3UV3XWDANCNFSM5K6DOOJA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you authored the thread.Message ID: @.***>

[ekol-ali]

Wild guess. Could it be that there is something with Australian time vs. Swedish time? I know they are 7-10 h ahead of us down there....

[mathieuLU]

ok so we have 2 problems in one.

1- When a survey is edited, the output date (surveyDate field) is changed, but the change is not applied to the records (eventDate field). => no impact on the extract. But foor instance the page "Alla Observationer" will have the wrong date (because these dates are coming from records) => the australian answered, we'll see what we can find there. Could be tricky.

2- A date edited can be displayed "the day before". Because of the way it's stored in the database. Let's use an example, yours : 61a37560-facf-9be2-6005-95c5b36c0e4c when the survey was created, the surveyDate (output) and eventDates (records) were set as : 2020-01-21T08:35:00Z when you edited it, you picked the 21st of december. In the database it looks like this :

db.output.find({activityId:"61a37560-facf-9be2-6005-95c5b36c0e4c"},{"data.surveyDate":1,"data.surveyStartTime":1}).pretty()

{
"_id" : ObjectId("61adf3a1efb513055451acbc"),
"data" : {
**"surveyDate" : "2020-12-20T23:00:00Z",
"surveyStartTime" : "08:35"**
}
}

so due to the timezones, the date is set to the day before, kl23:00 !

And then when the date is extracted, we get only the first part, regardless of the time...

to fix it, the best would be to keep the surveyStartTime attached to the date. To make sure we keep this information. I will look into it now and see what I can do

[ekol-ali]

[I did this while you were active - but I publish it nevertheless] Hm, I checked my own two latest censuses that I added in BioCollect. For them this problem does NOT exist! I also extracted all vinterpunktrutter data in Intranet, and my two routes drop out correctly, NOT following the "logic" of the bug. My two censuses:

https://biocollect.biodiversitydata.se/sft/bioActivity/index/b6bd73a1-d868-4b6c-93d9-8acda09b3801 https://biocollect.biodiversitydata.se/sft/bioActivity/index/c34cfecb-1687-4a4c-aea3-2c7b47176197

They were both "under review". Just for testing, I therefore made them "approved", but that didn't change anything. They were still OK (as they should be). Both on my "personal page" (undersökningar) and in a new Intranet extraction of all vinterpunktruttdata.

[mathieuLU]

I'm quite sure I understand why it works with these 2, and not for other previous examples. According to the database, you have created these 2 surveys the same day you made these observations. Meaning that you DID NOT touch the date field when you created the survey. (since it's set as default as the datetime of the day)

=> so the date is saved with the default setting, which was the datetime => it works fine. In the database :

    "dateCreated" : ISODate("2022-01-04T14:45:59.236Z"),
    "data" : {
        "surveyDate" : "2022-01-04T14:26:40+0000",
        "surveyStartTime" : "09:35"
    }

For all the other cases with bugs, it happens the moment the user is changing the date. And the moment the user is changing the date, then the internal time of the surveyDate is set to 00:00:00 (because the widget to select the date does not offer the possibility to edit the time) . So let's say that you reopen this survey, you change the date and set it back to the 4th of january (just to force it to change the time that is not shown). Then the date will be set to 2022-01-04T00:00:00, and when it will be saved, the magic of the timezone will transform it to : 2022-01-03T22:00:00

[mathieuLU]

The temporary fix is in place. It fixes the 2nd part of the problem (date saved to DAY - 1) in the extract files.

SO when you extract the files, the dates are fixed. I display a message, but I can hide it if it's too disturbing

[2022-01-14 12:30:11] WARN : Date tranformed from 2021-05-23T22:00:00Z to 20210524 [2022-01-14 12:30:14] WARN : Date tranformed from 2021-06-28T22:00:00Z to 20210629 [2022-01-14 12:30:14] WARN : Date tranformed from 2021-05-15T22:00:00Z to 20210516 [2022-01-14 12:30:14] WARN : Date tranformed from 2021-05-21T22:00:00Z to 20210522 [2022-01-14 12:30:15] WARN : Date tranformed from 2021-05-29T22:00:00Z to 20210530 [2022-01-14 12:30:16] WARN : Date tranformed from 2021-05-31T22:00:00Z to 20210601

Remains to be done :

• set to 12:00:00 when the date is changed through the JS snipet in BioCollect
• display day+1 for dates registered kl22:00 on page like that create_extract_excel.php
• fix the existing data in the database that have been set to day-1 (and set them to 12:0:00 ?)
• fix the update function in BC to make sure the records are changed accordingly

[mathieuLU]

ok i fixed that page as well, to display the date accordingly o the timezone => https://biocollect.biodiversitydata.se/sft/person/edit/339a06bc-fb62-c034-1e71-4bedb7588e0d/?returnTo=/sft/project/index/89383d0f-9735-4fe7-8eb4-8b2e9e9b7b5c#activityList

(you can check your example below. The date is still "2020-12-20T23:00:00Z" in the database, but now the page is taking in consideration the timezone to display correctly that date)

if you think about other places where survey dates are displayed, please let me know !

Remains to be done :

set to 12:00:00 when the date is changed through the JS snipet in BioCollect
fix the existing data in the database that have been set to day-1 (and set them to 12:0:00 ?)
fix the update function in BC to make sure the records are changed accordingly

[mathieuLU]

update

1- set to 12:00:00 when the date is changed through the JS snipet in BioCollect => "to modify the behaviour of the date picker component in the ecodata-client-plugin" (email from Temi/Chris)

2- fix the existing data in the database that have been set to day-1 (and set them to 12:0:00 ?) => still important to do it to fix the filtering page (because I think this page is truncating the date, to be verified)

3- fix the update function in BC to make sure the records are changed accordingly => most important one

4- fix that page as well https://biocollect.biodiversitydata.se/sft/bioActivity/allRecords date are displayed with long format without using the TZ

[mathieuLU]

number 4 is fixed

[mathieuLU]

Sometimes you're happy to be wrong :D

3- fix the update function in BC to make sure the records are changed accordingly => most important one

=> NO PROBLEM AT ALL. The record date (eventDate field) is perfectly edited, according to the output date (surveyDate field) I fooled myself with the fact that records are recreated after a modification (the old ones are left unchanged with the status "deleted", the new ones are edited)

Even the recorder name is changed accordingly.

i change the priority now.

[mathieuLU]

Small things that remain :

1- set to 12:00:00 when the date is changed through the JS snipet in BioCollect => "to modify the behaviour of the date picker component in the ecodata-client-plugin" (email from Temi/Chris)

2- fix the existing data in the database that have been set to day-1 (and set them to 12:00:00 ?) => still important to do it to fix the filtering page (because I think this page is truncating the date, to be verified)