Create `merge()` to merge datasets

jimcasaer commented 1 year ago

Note by @peterdesmet: see https://github.com/inbo/camtrapdp/issues/75#issuecomment-2142308577 for recent thoughts.

Quite often we want to merge the exports of several research projects (more and more for multi-site research), all using Camtrap-DP as export format, mostly all resulting from agouti. Though manually this can be done by using several times read_camtrap_dp and subsequently merging the different resulting csv-files, I was wondering if it would not be better to add a function to camtraptor that automatically

[ ] merges different data packages resulting from read_camtrap_dp
[ ] adds a column to the deployments.csv using the project_name as variable
[ ] adds a column to the deployments.csv using the project location / study site
[ ] the function name could be "join_exports" or something similar but this I leave up to Peter :-)

peterdesmet commented 1 year ago

I agree with the need of this functionality and that it is best supported by a function in Camtrap DP. The implementation details are to be discussed, but I'd prefer to tackle this after https://github.com/inbo/camtraptor/milestone/3

damianooldoni commented 1 year ago

A logic request, definitely. I agree with @peterdesmet: it's better to add this functionality after the big refactoring.

jimcasaer commented 1 year ago

Just to have an idea (and planning the next analyses) -- what would be the timing for this new feature ?

damianooldoni commented 1 year ago

Agreed with @peterdesmet to add this functionality to v1 and to give it a higher priority.

peterdesmet commented 1 year ago

Suggested in camtraptor July 2023 coding sprint

Call this function merge()
Limit it to two packages merge(package, package2) to avoid complex operations, testing and error handling. Users can add additional packages using pipe: package1 %>% merge(package2) %>% merge(package3)

PietrH commented 1 year ago

A package that is the result of a merge() https://github.com/inbo/camtrapdp/issues/75 should combine its metadata (not sure how)

From inbo/camtrapdp#74

peterdesmet commented 3 months ago

I have described the functionality in more detail at https://github.com/tdwg/camtrap-dp/issues/380. I think the function should be called merge_camtrapdp().

peterdesmet commented 1 month ago

Task list by @sannegovaert:

[x] Deployments: combination of deployments (but check that deploymentID remains unique)
[x] Media: combination of media (but check that mediaID remains unique)
[x] Observations: combination of media (but check that observationID remains unique)
[x] name: a new name is created (old values are not retained)
[x] id: a new ID is created. The original ids are stored in relatedIdentifiers
[x] created: reset to current timestamp (old values are not retained)
[x] title: a new title (old values are not retained)
[x] contributors: a combination is made, duplicates (on some or all fields) are removed and roles are combined (in Frictionless v2). The order is not retained.
[ ] description: a new one is generated, potentially still listing the previous descriptions
[x] version: is reset to 1.0 (old values are not retained)
[x] keywords: a combination is made, duplicates are removed
[x] image: is removed (old values are not retained)
[x] homepage: is removed (old values are not retained)
[x] sources: a combination is made, duplicates are removed
[x] licenses: a combination is made, duplicates are removed. If e.g. two different license with scope: media are listed, it won't be clear which one applies to which media.
[x] bibliographicCitation: is removed (old values are not retained)
[ ] project: ideally this becomes an array of projects and deployments have a projectID to link to the correct project info.
[x] coordinatePrecision: is reset to least precise precision (old values are not retained)
[x] spatial: is reset based on new deployments (old values are not retained)
[x] temporal: is reset based on new deployments (old vales are not retained)
[x] taxonomic: a combination is made, duplicates are removed.
[x] relatedIdentifiers: a combination is made, duplicates are removed
[ ] references: a combination is made, duplicates are removed

inbo / camtrapdp

Create `merge()` to merge datasets #75