What is the boolean field camera_timestamp for? #83

Closed niconoe closed 3 years ago

In GitLab by @peterdesmet on Jan 26, 2021, 2:32

Deployments has the field:

"name": "camera_timestamp",
"type": "boolean",
"description": "`true` if datetime settings in the camera were set up properly and in consequence all recorded multimedia files have proper timestamps.",
"example": "true",
"constraints": {
  "required": false
}

Is this regarding the timestamp in the exif of the multimedia files or the timestamp field in multimedia.csv. If the former, then I think that is not that important. If the latter, then I think this should be ensured by the camera data management software.

I think this field can be removed.

In GitLab by @kbubnicki on Jan 26, 2021, 14:40

I would say both. I agree that these wrong timestamps should be fixed prior to publication of given dataset but sometimes it is not possible i.e. no information is available to fix wrong timestamps but still recorded data is valuable. These can be still valid species observations (useful e.g. for species distribution modelling etc) but what is missing is the exact date of record (so not very useful for e.g. activity patterns analysis; however an approximate date can be still likely extracted from deployment metadata). I would keep this field as in my opinion this is a critical information for cases like described above.

In GitLab by @peterdesmet on Jan 26, 2021, 14:55

I would say both

I think we should limit that to the timestamps given in the csv, otherwise users don't know what this applies to.

Also, could we have this as a project level setting?

In GitLab by @kbubnicki on Jan 26, 2021, 15:32

Yeah, definitely it is about wrong timestamps given in the csv (then naturally you will the same wrong timestamps in the exif). I agree that we can make a description more clear in this respect.

As for the second point, I would answer: we can not as the majority of data (i.e. images/videos collected during the majority of deployments) can have correct timestamps and only some of them should be marked as having wrong timestamps. Does it make sense to you?

In GitLab by @peterdesmet on Jan 26, 2021, 15:44

only some of them should be marked as having wrong timestamps. Does it make sense to you?

But with the current implementation you can only indicate this at deployment level, so all or none of the assets (and I assume observations) of that deployment.

Do you have a real world example, that might help to see how the timestamps would look like.

In GitLab by @Tim-Hofmeester on Jan 27, 2021, 09:39

I was checking the issues and saw this one and have some thoughts I'd like to share.

I would say that this timestamp issue is an image specific issue as well as a potential deployment issue.

For images, one clear example are Bushnell cameras that sometimes out of nowhere reset the time and date to default settings. In that case, part of the deployment will have the correct time stamps and part of it won't, so that is something that should be handled at image level.

However, sometimes it happens that people make mistakes when entering the time and date. I have seen occasions where the whole deployment was a few hours off (in this case we only knew because we photographed a little sign with location, time and date when deploying the camera specifically to check these kind of things). Sometimes, the camera time is off by 12 hours (especially with Reconyx this can happen as it uses am/pm which is most confusing when setting the camera time between 12 and 13. We take a picture at noon and midnight to check camera functioning, so if you see the 'noon' picture is at night and the 'midnight' picture is during the day, you know the camera was off by 12 hours (but not necessarily if it was too early or too late). In these cases, of course it would be best to fix the time, but especially in the later case you might not be able to do that with certainty, and then this boolean field would be useful (I agree it might be a very rare thing).

In GitLab by @peterdesmet on Jan 28, 2021, 14:27

Another case of unprecise timestamps are timestamps without timezone. In the next release of Agouti, users will be able to indicate the UTC offset for all images of a deployment. Until that is done, all exported timestamps have no timezone and will fail camtrap-dp validation (see #81). I'm fine with this, since 1) it is a good requirement to expect timezones for correct data use, 2) users can see the difference between timestamps with and without timezones and 3) data providers can improve the quality by indicating the UTC offset in Agouti.

For the cases described above I'm wondering if we cannot take the same approach: export these without timezone or export them as a yyyy-mm-dd and have them fail validation.

Another case of unprecise timestamps are timestamps without timezone.

@peterdesmet I understand that your last comment is more related to #81 and this PR #114. So, what is the final conclusion for this particular issue? Can we conclude that we keep camera_timestamp and close this issue?

@kbubnicki Imo, we could indeed use a field that indicates that at deployment level the timestamps of the assets and the observations in the csv can be trusted or not trusted. Reasons for not trusting it are:

Timestamps probably don't have the right timezone and we can't/won't fix it
Camera was resetting time (https://github.com/tdwg/camtrap-dp/issues/83#issuecomment-796757946) and we can't/won't fix it

Since data are provided as best effort, I find it more logical to set it to TRUE if there is an issue that cannot be fixed, and NULL if it wasn't checked. I would therefore call it something like timestamp_drift or timestamp_issues = TRUE.

Agree, lets keep it at deployment level and change the name to timestamp_issues

tdwg / camtrap-dp

What is the boolean field camera_timestamp for? #83