search

strawlab / strand-braid

Live, low-latency 2D and 3D tracking from single or multiple high-speed cameras
https://strawlab.org/braid/
Other
46 stars 8 forks source link

flydra2::model_server disconnects with SendError Full #7

Closed elhananby closed 3 years ago

elhananby commented 3 years ago

Hi Andrew, I've been using the braid_retransmit_udp.py script to process incoming position data from Braid, but recently I've started getting this error after a while: INFO flydra2::model_server: Failed to send data to event stream, client probably disconnected. SendError { kind: Full } and the script exits with: Process finished with exit code 0

The main issue is that I'm not sure what leads to what - does the script exit unexpectedly, which causes the model_server to disconnect; or does the model_server disconnect due to the SendError, which causes the script to exit?

If I manually stop the script, I get a different error: INFO flydra2::model_server: Failed to send data to event stream, client probably disconnected. SendError { kind: Disconnected } which makes sense; I'm just not sure what does the other Full error mean (I tried to go over the model_server rust code, but I'm not proficient enough to fully understand I.)

Let me know if you have any insights.

Thanks, Elhanan

astraw commented 3 years ago

For your first case (Braid prints that the stream is full), this indicates that the python script is processing the data more slowly than it is being generated by Braid (hence the "Full" error).

(I'm not sure why it would then end without error - exit code 0. The braid_retransmit_udp.py script enters an infinite loop with no way to exit other than an error. So it is puzzling for me that it appears to do so.)

Can you make the script do nothing other than read the network data (to speed it up) and see if these Full errors still arise?

There is a "send_pose" program which you can run standalone and to test these things without needing to run full Braid. To run this: cd flydra2 then cargo run. You can view logged events by setting the environment variable RUST_LOG to "info". You could speed up the send_pose by changing the line let mut interval_stream = tokio::time::interval(std::time::Duration::from_millis(100)); to send at intervals shorter than 100 msec.

elhananby commented 3 years ago

Hi Andrew, thanks for the advice. I debugged my script using send_pose and found out I was accidentally flooding the Arduino with requests, which caused the main loop in braid_retransmit_udp.py to slow down significantly; shortly after I get the Full error. I'm closing this for now, thanks for the help.

astraw commented 3 years ago

Sounds good. Backpressure can be tricky.