Nondeterminism

[rustychris]

Overall, we are getting fantastic results from yaps compared to our previous solutions! That said, I've noticed that I sometimes get different results for the same inputs, such as the two solutions below (note differing position at southern end of track). For some tracks the difference is much more severe. I don't know if this is a bug, a sign that my data is dirty, or an unavoidable aspect of the numerical approach. I think the difference comes in during the clock synchronization, and as far as I can tell if I save a sync_model and reload it, I get essentially the same track result out of runYaps.

During getSyncModel, I get messages like this relatively often:

Error in newton(par = c(TOP = 4997.52986435247, TOP = 7600.59601676365,  : 
  Newton failed to find minimum.
In addition: Warning message:
In stats::nlminb(inits, obj$fn, obj$gr) : NA/NaN function evaluation

On occasion runYaps fails, and generally I get a handful of these warnings after several calls to runYaps:

1: In stats::nlminb(inits, obj$fn, obj$gr) : NA/NaN function evaluation
2: In sqrt(diag(object$cov.fixed)) : NaNs produced
3: In sqrt(diag(object$cov.fixed)) : NaNs produced
4: In sqrt(diag(object$cov.fixed)) : NaNs produced
5: In sqrt(diag(object$cov.fixed)) : NaNs produced

In more severe cases, one iteration will give me a great track with small uncertainty, and a second iteration will give a very different result and very large uncertainty, as if the optimization step bailed out too early.

Should I expect this sort of non-determinism? Is there a good rule of thumb for detecting that the sync model should be recalculated?

thanks again for the excellent tool!

[baktoft]

Thanks for your question and feedback. I am aware of the issues you raise and am trying to find time to fix it. The numerical estimation method (i.e. TMB) is, as I understand it, deterministic. The variation in the final track can arise from two sources: the sync model and the initial parameters (generated randomly in getInp()) going into runYaps(). The sync model itself can vary based on initial values (also generated randomly in getInpSync()), but also on the actual input data if you use a keep_rate < 1 to subset your data. Hard to tell, but from the look of it, I believe the variation between the two plots arise from sync model variation. May I ask what system you are using? How many sync tags? Burst interval type and length of the fish transmitter?

The synchronization is extremely important and might take some time and trial/error to get right. I suggest to establish an as good as possible sync model covering the entire study period and use that for all track estimations. You can use plotSyncModelResids() to evaluate how good the sync model is and to identify e.g. if some hydro positions are inaccurate, if they were moved during the study etc. If you share/send me some of these, I can have a look. Also, always do a tow track combining gps and transmitter to have a ground truth to compare against.

The errors you mention, are from the TMB optimization process. Some of them are seemingly harmless (?) and a future version of yaps will silence them. A future version will also attempt to catch the cases in which yaps fails miserably, retry a number of times and ultimately return a fail.

Hope it makes sense and help your progress. Let me know how it turns out.

[rustychris]

Thanks for the tips. We are using JSAT tags, with receivers from Teknologic. The beacon interval on the sync tags is ~30 seconds, and the fish tags ping every 5 seconds. I think you're right that the variation is from the sync model. If I save and re-use a sync model, I get consistent results when re-running runYaps(). I was processing the data in 6 hour chunks, but based on your advice have shifted to much longer periods (a few hydrophones were recovered and redeployed during the experiment, so I am breaking up the dataset at those times). When the reprocessing is complete I will compare the old and new tracks, but so far this seems to be working more reliably.

thanks --

On Fri, Mar 13, 2020 at 1:01 AM Henrik Baktoft notifications@github.com wrote:

Thanks for your question and feedback. I am aware of the issues you raise and am trying to find time to fix it. The numerical estimation method (i.e. TMB) is, as I understand it, deterministic. The variation in the final track can arise from two sources: the sync model and the initial parameters (generated randomly in getInp()) going into runYaps(). The sync model itself can vary based on initial values (also generated randomly in getInpSync()), but also on the actual input data if you use a keep_rate < 1 to subset your data. Hard to tell, but from the look of it, I believe the variation between the two plots arise from sync model variation. May I ask what system you are using? How many sync tags? Burst interval type and length of the fish transmitter?

The synchronization is extremely important and might take some time and trial/error to get right. I suggest to establish an as good as possible sync model covering the entire study period and use that for all track estimations. You can use plotSyncModelResids() to evaluate how good the sync model is and to identify e.g. if some hydro positions are inaccurate, if they were moved during the study etc. If you share/send me some of these, I can have a look. Also, always do a tow track combining gps and transmitter to have a ground truth to compare against.

The errors you mention, are from the TMB optimization process. Some of them are seemingly harmless (?) and a future version of yaps will silence them. A future version will also attempt to catch the cases in which yaps fails miserably, retry a number of times and ultimately return a fail.

Hope it makes sense and help your progress. Let me know how it turns out.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/baktoft/yaps/issues/23#issuecomment-598598128, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABJ6ME4RMWI6ZM2GH6HAOLDRHHR7DANCNFSM4LGWRLUA .