search

ctmm-initiative / ctmmweb

Web app for analyzing animal tracking data, built upon ctmm R package
http://biology.umd.edu/movement.html
GNU General Public License v3.0
34 stars 22 forks source link

Error calibration when having data issues #84

Open NoonanM opened 5 years ago

NoonanM commented 5 years ago

A user is having some issues with the error calibration step. The data have two location classes, so as.telemetry returns a class column, and two classes in the UERE slot. When trying to calibrate the error on a dataset with some calibration data, as.telemetry is returning two classes for the calibration data as well, but the names of the classes are different (I have no idea why the names are coming back differently). That means that when you try to calibrate the errors, the classes don't match up and it fails. As a work around in the command line, you can simply rename the classes, but this isn't possible in the app.

I also tried manually assigning a UERE of 10 on the dataset with 2 location classes, but that also failed.

xhdong-umd commented 5 years ago

I looked at the data but don't know about the class column. In the app I just use the recommended code to calibrate data, and I'm not sure what's preferred the treatment for this case. @chfleming what should we do with this?

NoonanM commented 5 years ago

The same issue arises with trying to assign a fixed value to the UERE when there are different classes.

uere(DATA) <- 10 adds a class "all" to the data, but because class "all" doesn't exist in the dataset, it fails.

xhdong-umd commented 5 years ago

I think we need @chfleming to determine how to deal with this kind of cases...

chfleming commented 5 years ago

If this is Pati's Telonics data, there are other issues with the data we need to discuss to determine if the data are corrupted or if this is normal for some Telonics QFPs to have NA DOP value and satellite count.

I don't think blanket assignment should work when there are multiple location classes. The classes need to be deleted first.

With how as.telemetry is coded currently, when there are different missing columns / location classes in the tracking and calibration data, then the location class levels can come out differently if they are imported separately and these differences are not accounted by a location class column. What I do is first look for location class columns. Then I look for missing columns like speed or altitude that I know distinguish location classes (e.g., altitude doesn't exist for 2D fixes). If any further structure is not accounted for by the current location classes, then I introduce a minimal number of extra location classes like bifurcating class "X" -> "X speed" and "X missing". I will update these levels to "X [speed]" and "X NA[speed]" to be more informative/distinctive.

Unless there is a smarter way to do this on my end, which is really working around data quality issues, the only way to make this easier in the webapp is to have the ability to import the tracking and calibration data together and run them through as.telemetry together, to enforce the same location classes, and then designate them as tracking or calibration afterwards.

xhdong-umd commented 5 years ago

OK, do you think this process can be the universal approach for all the calibration data? i.e. We always load calibration data and tracking data, then process together? This means we cannot process calibration data itself.

import the tracking and calibration data together and run them through as.telemetry together, to enforce the same location classes, and then designate them as tracking or calibration afterwards.

chfleming commented 5 years ago

I don't think that would always work, because sometimes people modify file headers, etc. Users can also have different devices in the same study (with different file structures) and could get confused. I see two paths to catch all possibilities:

  1. When importing tracking and calibration data separately, check that the class column has the same levels. Throw an error if they don't match. Either the files aren't consistent or there are location classes missing from the calibration data.
  2. When importing simultaneously, check that calibration UERE DOFs are all >0. When some are zero that means they are missing in the calibration data. Throw a warning for that.