search

strawlab / nextgen-camera-users

next generation camera software from the Straw Lab
1 stars 0 forks source link

Crash in tracking_core + reproj. distance values issues #7

Closed elhananby closed 3 years ago

elhananby commented 3 years ago

Hi Andrew, I've been having some weird crashes recently, always with this error message: thread 'main' panicked at 'called 'Result::unwrap()' on an 'Err' value: Error { kind: CovarianceNotPositiveSemiDefinite }', flydra2/src/tracking_core.rs:764:34 note: run with 'RUST_BACKTRACE=1' environment variable to display a backtrace Error: channel is empty and disconnected

Also, during tracking I see these types of messages a lot: mean reprojection 100x distance value 1216378 out of expected range I assume it is an issue with the 3d calibration? Generally my reproj. error is low enough I think. This also causes issues with later analysis, when I suddenly see points that are way out of the tracking volume.

I'm using the latest Braid release on Ubuntu 18.04.

EDIT: Oh, I also forgot to mention I still have the issue that I can use "Synchronize Cameras" only once - otherwise Braid crashes without any specific error message: [2020-11-16T14:54:38Z INFO flydra2_mainbrain] preparing to synchronize cameras [2020-11-16T14:54:38Z INFO flydra2_mainbrain] preparing for triggerbox to temporarily stop sending pulses [2020-11-16T14:54:38Z INFO flydra2_mainbrain] requesting triggerbox to stop sending pulses [2020-11-16T14:54:41Z INFO flydra2_mainbrain] requesting triggerbox to start sending pulses again [2020-11-16T14:54:41Z INFO flydra2::connected_camera_manager] cam Basler_23088882 synchronized. frame_offset: 1710 Doing the same for the rest of the cameras... [2020-11-16T14:54:41Z INFO flydra2_mainbrain::multicam_http_session_handler] for cam Basler_23047980, sending frame offset 1878 (flydra_py3) elhanan_la@nfc3006:~$

Thanks Elhanan

astraw commented 3 years ago

Can you run braid --version and make sure you are really running 0.8.0? In this release, I updated braid to use the OptimalKalmanForcedSymmetric covariance update method which should prevent the covariance getting not positive semi-definite.

We also see some unreasonably high reprojection errors here in a small fraction of frames (<.1%) and this on my list to investigate.

elhananby commented 3 years ago

Hi Andrew, running braid --version gives me braid 0.8.0. I initially tried to install it over the previous version, but also tried to completely remove and reinstall it.

astraw commented 3 years ago

OK, thanks for checking. I am perplexed how this error could arise and I would really like to fix it. Can you start recording and then upload here or send me directly the .braid directory in the state after the crash? I will have a look.

elhananby commented 3 years ago

Hi Andrew, I've tried reinstalling it completely again and so far haven't had any issues. If it happens again I will send you the .braid directory.

astraw commented 3 years ago

OK, sounds good.

elhananby commented 3 years ago

Hi Andrew, I didn't have the problem for a few days now so I think I'll close this issue for now. I think it may have been an issue with not uninstalling the previous version properly? I'll let you know if the issue pops-up again.

Thanks