Processing Camtrap DP "3 pilot studies"

peterdesmet commented 5 months ago

https://zenodo.org/records/10671148 (described in https://doi.org/10.1016/j.dib.2024.110544) is a Zenodo deposit published by the University of Amsterdam. It contains 3 Camtrap DP datasets (pilot1.zip, pilot2.zip, pilot3.zip). This issue is to discuss technical details of the datasets, to inform/guide processing by GBIF.

Validation errors

I have tested pilot2.zip with frictionless-py (frictionless validate datapackage.json) and found the following issues:

events resource is defined 3 times in datapackage.json. Solution: remove the duplicates. See also note on events further.
events resource is defined as a tabular data package, but lacking a schema (required for tabular data package). Solution: remove "profile": "tabular-data-resource"
Timestamps in dep.deploymentStart, dep.deploymentEnd, media.timestamp, obs.eventStart, obs.eventEnd use a space rather than T to separate date and time. The standards expects T, which is what Agouti exports, so this might be the result of further processing. Solution: use T.
obs.individualCount contains doubles (1.0) rather than integers (1), which is what Agouti exports, so this might be the result of further processing. Solution: remove .0
Not a validation error, but filePath is expected to contain the full path to the media file, not just the directory that contains it, i.e. it is not intended to be concatenated with fileName, which is just a convenience term for sorting or the provide original file names if they were changed (i.e. what Agouti does). In addition, backslashes should be avoided. So filePath:

# don't do this
media\AWD_1_13082021_pilot 46576a8c-019a-4dd8-852e-86380e0973fc
# do this
media/AWD_1_13082021_pilot 46576a8c-019a-4dd8-852e-86380e0973fc/10b0e4da-ca2d-4026-8574-bff8d15a3dcb.JPG

Note: while it can be useful to define a separate CSV with the events as a convenience file, it 1) will typically be ignored by software reading Camtrap DP and 2) can typically be derived automatically. So you may want to opt to not providing it:

camtrapdp::read_camtrapdp() automatically assign an eventID to the media when reading data (based on the info found in observations)
camtrapdp::events() will give you all events. This is the result for pilot2: events.csv.

If pilot1 and pilot3 went through the same processing, then they likely have the same issues.

jevansbio commented 5 months ago

The dataset has been tweaked, now passing all frictionless tests. The updated version is live on zenodo. https://zenodo.org/records/11440456

I still prefer to include the events file for convenience, as I feel it makes the data a little more 'off the shelf' if a user is not using the R package.

peterdesmet commented 5 months ago

Looks good! Here's how pilot2 looks when processed by https://inbo.github.io/camtrapdp/reference/write_dwc.html: dwc.zip

@MattBlissett the accessURI is now of the form:

pilot2/media/AWD_1_13082021_pilot 2c79df2e-87bb-4416-bf3f-50e4fbf28a88/4f13384c-2970-4b69-bf6f-1261b1507566.JPG

I'm not entirely sure what extra info is needed to support reading of that file from gbif.org?

gbif / pipelines

Processing Camtrap DP "3 pilot studies" #1070

Validation errors