synchronization failing with flydra2-mainbrain

[astraw]

Via email, I received:

I've collected some data with flydra2/camnode 2 (the most recent ones you sent me today), just using a single camera to test things out. A lot of the csv files are coming up empty, most importantly kalman_estimates (but also data2d_distorted, data_association, and experiment_info). So when I run:

python convert_kalmanized_csv_to_flydra_h5.py

It is breaking the do_ml_estimates() function called on line 321 (ValueError: No columns to parse from file).

[astraw]

I've collected some data with flydra2/camnode 2 (the most recent ones you sent me today), just using a single camera to test things out. A lot of the csv files are coming up empty, most importantly kalman_estimates (but also data2d_distorted, data_association, and experiment_info).

I guess your camera is not synchronized. The camera's trigger input must be connected to our trigger device, set to external trigger mode, and the appropriate synchronize command must have been given (e.g. the button in the mainbrain browser UI).

It is actually on my to-do list to raise a user-visible error if a user attempts to save data without the connected cameras being synchronized. Flydra just isn't designed to be able to do much without that. (Admittedly, "synchronizing" doesn't make much conceptual sense when there is only one camera. But then again, using only a single camera defeats the point of flydra in the first place.)

If you just want to save data from a single camera using flydra2-mainbrain, you could "synchronize" it as described above. Else, you can use the fview2/camnode2 browser UI directly and save .csv files directly from the single camera.

Also, no frame data or 2D points will be saved if no points are detected (unless you use the --save-empty argument to flydra2-mainbrain).

So when I run:

python convert_kalmanized_csv_to_flydra_h5.py

It is breaking the do_ml_estimates() function called on line 321 (ValueError: No columns to parse from file).

You can run convert_kalmanized_csv_to_flydra_h5.py with --2d-only to prevent it from trying to read the 3D data.

The script convert_kalmanized_csv_to_flydra_h5.py could certainly be improved in several ways in terms of ergonomics and speed.

I'm closing this for now, as I guess this solves the problem. Feel free to re-open or contact me directly.

[EricThomson]

Thanks this is all helpful stuff!

I guess your camera is not synchronized.

This is right: mainbrain2 was giving an error when I ran:

python scripts/simple-console.py --sync

I then made a (naive) assumption that maybe with mainbrain2 it might not be needed, which is something I probably should have mentioned. I get the same error with two cameras set up, and when I instead try to synchronize from the mainbrain2 browser.

mainbrain2 crashes and we get:

thread 'tokio-runtime-worker-7' panicked at 'no entry found for key', libcore/option.rs:1000:5 note: Run with RUST_BACKTRACE=1 for a backtrace. thread 'main' panicked at 'called Result::unwrap() on an Err value: Canceled', libcore/result.rs:1009:5 ERROR 2018-11-20T16:01:15Z: flydra2_mainbrain::multicam_http_session_handler: HttpSessionHandler::post() got error Error { kind: Execute, cause: "tokio::spawn failed (is a tokio runtime running this future?)" } [flydra_mainbrain-2] process has died

[astraw]

OK, I see that python scripts/simple-console.py --sync is throwing this error. I will investigate. How about clicking the "synchronize cameras" in the browser UI?

[EricThomson]

The above cut/paste is from when I clicked it in the browser UI. When I run the simple-console.py command in bash, I get:

thread 'tokio-runtime-worker-7' panicked at 'no entry found for key', libcore/option.rs:1000:5 note: Run with RUST_BACKTRACE=1 for a backtrace. thread 'main' panicked at 'called Result::unwrap() on an Err value: Canceled', libcore/result.rs:1009:5 ERROR 2018-11-20T21:28:35Z: image_tracker: error sending socket data: Os { code: 111, kind: ConnectionRefused, message: "Connection refused" } ERROR 2018-11-20T21:28:35Z: image_tracker: error sending socket data: Os { code: 111, kind: ConnectionRefused, message: "Connection refused" } Error (Error (fview2/src/fview2.rsfview2/src/fview2.rs::15991599): ): in frame_process_threadin frame_process_thread

Caused by: Caused by: Connection refusedConnection refused (os error (os error 111111))

Error (Error (fview2/src/fview2.rsfview2/src/fview2.rs::20712071): ): receive errorreceive error

[astraw]

OK, thanks. I will do my best to check this out tomorrow afternoon (German time). I'm not sure why you are having a problem - I tested all this quite recently.

[astraw]

OK, I just tested here both from the browser UI and the ROS python script and both result in a synchronization for me with out the errors. So let's try to debug what might be going wrong on your system.

Are the cameras really in external trigger mode? I.e. if you unplug the trigger cable, images stop being acquired? Do you have an oscilloscope and can check that immediately after the "synchronize" command that the trigger pulses stop coming out of the trigger box?

Set the RUST_LOG=info (or if perhaps even RUST_LOG=debug) environment variables and also set RUST_BACKTRACE=1. Then please try get this same error again and paste or upload the output, including a bit from prior to when things started to go wrong if possible.

[EricThomson]

I am launching with the attached launch file (with suffix changed from 'launch' to 'txt' so I could upload it).

When I unplug the USB to the trigger box, the camera nodes die and I stop getting images.

When I send the sync command, the trigger pulses are briefly interrupted (after the logger says flydra2_mainbrain: requesting triggerbox to stop sending pulses).

To reproduce it again, I change RUST_LOG to debug and RUST_BACKTRACE to 1, run the attached launch file, and then in a separate terminal, run python scripts/simple-console.py --sync. Then in the launch terminal, the following (with some of the previous lines during the time when everything is working well):

INFO 2018-11-21T18:13:12Z: fview2: listening at http://127.0.0.1:4012/ INFO 2018-11-21T18:13:12Z: fview2: running forever INFO 2018-11-21T18:13:12Z: fview2: starting async capture INFO 2018-11-21T18:13:12Z: fview2: listening at http://127.0.0.1:4011/ INFO 2018-11-21T18:13:12Z: fview2: running forever INFO 2018-11-21T18:13:12Z: fview2: starting async capture INFO 2018-11-21T18:13:12Z: flydra2_mainbrain::multicam_http_session_handler: opening session for cam Basler_22448452 to http://127.0.0.1:4012 INFO 2018-11-21T18:13:12Z: flydra2_mainbrain::multicam_http_session_handler: opening session for cam Basler_22448453 to http://127.0.0.1:4011 INFO 2018-11-21T18:13:32Z: flydra2_mainbrain: preparing to synchronize cameras INFO 2018-11-21T18:13:32Z: flydra2_mainbrain: preparing for triggerbox to temporarily stop sending pulses INFO 2018-11-21T18:13:32Z: flydra2_mainbrain: requesting triggerbox to stop sending pulses INFO 2018-11-21T18:13:35Z: flydra2_mainbrain: requesting triggerbox to start sending pulses again INFO 2018-11-21T18:13:35Z: flydra2_mainbrain::multicam_http_session_handler: for cam Basler_22448453, sending frame offset 1006 INFO 2018-11-21T18:13:35Z: flydra2_mainbrain::multicam_http_session_handler: for cam Basler_22448452, sending frame offset 1006 thread 'tokio-runtime-worker-4' panicked at 'no entry found for key', libcore/option.rs:1000:5 note: Run with RUST_BACKTRACE=1 for a backtrace. thread 'main' panicked at 'called Result::unwrap() on an Err value: Canceled', libcore/result.rs:1009:5 thread '' panicked at 'set_image: Error(SerdeRosmsg(Error(Io(Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }), State { next_error: None, backtrace: None })), State { next_error: None, backtrace: None })', libcore/result.rsthread ':1009' panicked at ':set_image: Error(SerdeRosmsg(Error(Io(Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }), State { next_error: None, backtrace: None })), State { next_error: None, backtrace: None })5', libcore/result.rsnote: Run with RUST_BACKTRACE=1 for a backtrace. :1009:5 note: Run with RUST_BACKTRACE=1 for a backtrace. ERROR 2018-11-21T18:13:35Z: image_tracker: ROS sender disconnected (image-tracker/src/lib.rs:1319) ERROR 2018-11-21T18:13:35Z: image_tracker: ROS sender disconnected (image-tracker/src/lib.rs:1319) Error (fview2/src/fview2.rs:1599): in frame_process_thread Caused by: other error: ROS sender disconnected

thread 'video_streaming' panicked at 'cannot access stderr during shutdown', libcore/option.rs:1000:5 Error (fview2/src/fview2.rs:1599): in frame_process_thread Caused by: other error: ROS sender disconnected

thread 'video_streaming' panicked at 'cannot access stderr during shutdown', libcore/option.rs:1000:5 [flydra_mainbrain-2] process has died [pid 26665, exit code 101, cmd /home/fishvr/ros/freemovr-kinetic/src/ros_flydra/nodes/mainbrain2 --http-api-server-addr 127.0.0.1:4010 --output /home/fishvr/FLYDRA --trigger_device /dev/trig1 --trigger_device_fps 50.0 --reconstructor /home/fishvr/ros/freemovr-kinetic/src/ros_flydra/calib/twocam_reconstructor.xml name:=flydra_mainbrain __log:=/home/fishvr/.ros/log/13a5557e-edb9-11e8-9c57-107b4448afd0/flydra_mainbrain-2.log]. log file: /home/fishvr/.ros/log/13a5557e-edb9-11e8-9c57-107b4448afd0/flydra_mainbrain-2*.log [flydra_camera_node1-3] process has died [pid 26691, exit code 1, cmd /home/fishvr/ros/freemovr-kinetic/src/ros_flydra/nodes/camnode2-pylon --camera-name=Basler-22448453 --http-server-addr=127.0.0.1:4011 --tracker-config=/home/fishvr/ros/freemovr-kinetic/src/ros_flydra/calib/calibration_tracking.yaml name:=flydra_camera_node1 __log:=/home/fishvr/.ros/log/13a5557e-edb9-11e8-9c57-107b4448afd0/flydra_camera_node1-3.log]. log file: /home/fishvr/.ros/log/13a5557e-edb9-11e8-9c57-107b4448afd0/flydra_camera_node1-3.log [flydra_camera_node2-4] process has died [pid 26724, exit code 1, cmd /home/fishvr/ros/freemovr-kinetic/src/ros_flydra/nodes/camnode2-pylon --camera-name=Basler-22448452 --http-server-addr=127.0.0.1:4012 --tracker-config=/home/fishvr/ros/freemovr-kinetic/src/ros_flydra/calib/calibration_tracking.yaml name:=flydra_camera_node2 log:=/home/fishvr/.ros/log/13a5557e-edb9-11e8-9c57-107b4448afd0/flydra_camera_node2-4.log]. log file: /home/fishvr/.ros/log/13a5557e-edb9-11e8-9c57-107b4448afd0/flydra_camera_node2-4.log ^C[rosout-1] killing on exit [master] killing on exit shutting down processing monitor... ... shutting down processing monitor complete

Versions: ros kinetic ubuntu 16.04 flydra2-mainbrain 0.20.8 camnode2-pylon 0.20.8

flydra2-twocameras.txt

[astraw]

Thanks, but somehow your environment variable settings are not taking effect and so I do not see what I'm looking for. Can you either run like this:

RUST_LOG=debug RUST_BACKTRACE=1 roslaunch ./my-launch-file.launch --screen

Or add these env vars to each node like this:

    <node name="flydra_mainbrain" pkg="flycave" type="mainbrain2" args="--save_empty --output /home/strawlab/DATA -r /home/strawlab/ros/strawlab-vr-kinetic-catkin/src/flycave/calibration/fishtrax2/2018-11-13/good.xml --http-api-server-addr 127.0.0.1:3999 -t /dev/trig1">
        <env name="RUST_LOG" value="debug"/>
        <env name="RUST_BACKTRACE" value="1"/>
    </node>

    <!-- This way of specifying rosparam is different than the old flydra1 way where the rosparams were `/Baslser_1234/gain_db`. Here they would be `/ros_node_name/gain_db`. -->
    <node name="flydra_camera_node1" pkg="ros_flydra" type="camnode2-pylon" args="--camera-name=Basler-22005677 --http-server-addr=127.0.0.1:4011 --tracker-config=/home/strawlab/ros/strawlab-vr-kinetic-catkin/src/flycave/launch/fishtrax2/tracker-config.yaml">
        <env name="RUST_LOG" value="debug"/>
        <env name="RUST_BACKTRACE" value="1"/>
        <env name="DISABLE_VERSION_CHECK" value="1"/>
        <rosparam>
        gain_db: 1.
        exposure_usec: 500
        external_trigger: On
        </rosparam>
    </node>

In this case, please also use --screen as an argument to the command-line roslaunch invocation - this way, ROS will show all of the log messages which might otherwise get redirected.

[EricThomson]

Ah thank you so much I was wondering why the env vars seemed to be having no effect. That's more like it. Attached output after running --sync command. log_output.txt

[astraw]

There are definitely more logs now, but I'm still not sure what is going on. One idea is that there is a problem with the camera names. In your calibration file /home/fishvr/ros/freemovr-kinetic/src/ros_flydra/calib/twocam_reconstructor.xml, are the cam_id values should be specified with an underscore (like <cam_id>Basler_22005677</cam_id>). Could that be the issue?

[astraw]

Now I have a new idea - the calibration file has less than two cameras in it. This was a case I never considered and I think the error might arise due to this. I am making a new build which will give a more explicit error if this is the case. In the meantime, you can perhaps create a "fake calibration" file with your camera names in it but data from elsewhere.

[astraw]

I uploaded a new release of flydra2-mainbrain you can install. If I am correct, it should give a more explicit error message. Is is located here.

[EricThomson]

Will install it and test. In the meantime, the xml file is attached (with extension changed to txt). It is indeed "fake" data from elsewhere: all of this troubleshooting is part of my prep to run the "real" calibration with my four cameras! :)

twocam_reconstructor.txt

[EricThomson]

No change on my end that I noticed with this most recent iteration. I'm attaching the output in the launch terminal as log2.txt. The calibration_tracking.yaml I'm using is:

do_update_background_model: true
polarity: DetectAbsDiff
alpha: 0.009999999776482582
n_sigma: 7
bright_non_gaussian_cutoff: 255
bright_non_gaussian_replacement: 5
bg_update_interval: 200
diff_threshold: 30
threshold_fraction: 0.5
use_cmp: true
max_num_points: 1
feature_window_size: 30
clear_fraction: 0.30000001192092896
despeckle_threshold: 5
valid_region: Everything

Also, note that using plain old camnode/mainbrain, sync works on my setup without a hiccup (and the data saves and displays).

logfile2.txt

[astraw]

Indeed this didn't seem to help and I will have to look further. In the meantime, I suggest running mainbrain2 without a calibration. To do this, remove the -r or --reconstructor argument when launching the node. It is possible this will bypass whatever problem you are currently hitting.

[EricThomson]

Running without the reconstructor worked, so I will just do this for now. Thanks a lot for your help!

[astraw]

OK, I think I have now fixed the fundamental issue you were facing -- failing to synchronize with 2 cameras -- in the newly released version 0.20.11. I uploaded a new release to the usual Dropbox download location.

The problem was that I never thought about running flydra with just 2 cameras and there was a corner case where if you had only two cameras connected, one part of the code built a structure expecting a maximum of two cameras. But another part of the code assumed always 3 or more cameras and hence failed. Now such a case is explicitly handled to avoid the error.

To calibrate using MultiCamSelfCal, you will need 3+ cameras.

[EricThomson]

Nice: the sync works now with two (or N) cameras. I have learned my lesson and am now just running with three cameras, even just when troubleshooting on my desktop. Also, the following general workflow of saving hd5 and plotting etc is working too, as you suspected:

python convert_kalmanized_csv_to_flydra_h5.py directory_name.mainbrain flydra_analysis_plot_timeseries_2d_3d filename.mainbrain.h5