Function for aggregating metadata outside the IMOS Animal Tracking WebApp

[fmaron]

Is your feature request related to a problem? Please describe.

Aggregate detections data and metadata outside the WebApp

Describe the solution you'd like

A function that allows the user to easily aggregate receiver metadata for different arrays according to the detections recorded.

Describe alternatives you've considered Aggregation function created by Yuri Niella. The function aggregates the receiver metadata for the different arrays, place them into the same folder and combines the metadata into one file.

Additional context

Yuri's comments:

One issue I had was that my tags get detected in multiple receiver arrays since the sharks travel a lot. I created a small function to aggregate receiver metadata for different arrays which you guys could incorporate into remora. The user only needs to download all receiver metadata for the different arrays of interest and put them all into the same folder. Then provide the folder's path to the recComb() function - please feel free to rename it if you do want to use it :) - and it will return a combined metadata file. I tested and it does works with the QC functions.

@ianjonsen comments:

I haven't implemented Yuri's code for aggregating receiver metadata as this requires some additional thought on how to merge into the pkg. Based on Yuri's comments, it seems like he was aggregating detections & metadata outside of the WebApp. As our primary goal is to support data via WebApp download. Unless I'm missing something, an aggregation function shouldn't be necessary as this is part of what the WebApp should be doing. When/if we move toward supporting data outside of AODN's WebApp then we can pull in Yuri's aggregation code in some form.

[fjaine]

I think the issue Yury experienced is primarily driven by the fact that he downloaded receiver deployment metadata from individual projects/arrays via the web-app instead of downloading all historical receiver metadata at once (hence why he needed to create this additional aggregation function) - downloading all available receiver deployment records would be my recommended practice (the file is pretty light and downloads quite fast).

Hopefully in the near future, if we can connect the package directly to the database API, we might be able to automatically extract all available receiver deployment metadata to date, which would resolve the issue that Yuri experienced. Thoughts @ianjonsen @fmaron @vinayudyawer @RossDwyer ?

[fjaine]

Labelling as high priority as other users may experience similar issues until we address this

[ianjonsen]

Has anyone tried running the QC on eg., a single species' detections using the complete IMOS receiver deployment metadata? I didn't envisage this as a possibility when coding the QC fn's so not certain it will work smoothly. If it does then we could update the QC vignette to recommend that users who are likely to have focal species' detections spread across many receiver arrays download the complete receiver deployment metadata file. @fmaron could you test runQC() using the full receiver deployment metadata?

[fjaine]

@ianjonsen that's the only way I've used the QC to date 😁 but let's have @fmaron double check and confirm that it still works fine.

[fjaine]

That was my thinking in terms of an easy fix for this issue - communicate better in the vignette that they can access full network's receiver deployment metadata rather than using multiple individual array datasets.

[fmaron]

@ianjonsen @fjaine I tested the QC function with all receivers' metadata and it worked. It took a little bit longer than it normally does, but everything seems fine. I'll send the plotQC image in an email.

[fjaine]

@ianjonsen can we close this issue or should we keep it open for implementing Yuri's aggregation function at a later stage?