search

avstack-lab / demo-platform

Platform for ICCPS demo
MIT License
0 stars 0 forks source link

Dropping frames #18

Open roshambo919 opened 1 year ago

roshambo919 commented 1 year ago

When running in verbose mode, it is clear to see that frames are being dropped at some point in the pipeline. Is it a ZMQ issue? Is it a data rate issue? Is it a data buffer issue? Is it a timing issue? Still unclear.

Running in replay mode, you'll see that the replayer sends out all the frames. However, when logging the inference at the detection level, not all frames are accounted for. So somewhere between the replayer - data broker - detection workers, frames of data are being dropped.

natezelter1 commented 1 year ago

I suspect this is an issue with a combo of object_detection.py, object_tracking.py, and or simple.py. This problem persists even after I've reduced transmitted frame size. When i run 'make flir' with everything else running, frames are clearly lost. But when i run that command with no other processes running (ie, nothing to consume that published data), no frames are lost according to print statements in verbose mode. This leads me to think it's a buffering issue with peripheral modules, not with ZMQ or sensor.py directly.

roshambo919 commented 1 year ago

Object detection should just be taking in a frame and spitting out a detection. Maybe the data broker is dropping data somehow? We should confirm which frames the data broker receives from the sensors, which frames it tries to send out, and which frames the detection receives as a first step.

If the data broker is getting too overloaded with data, maybe we can add more buffer memory or zmq workers. Or we could try to speed up object detection (with a smaller NN model, for example).

roshambo919 commented 1 year ago

I think things are getting dropped in data_broker.py. I added an additional verbose printout and it seems like data is sent from the replayer or the camera to the data broker, then the data broker looks to see if a worker is available, and if no worker is available, it drops and continues on. The remedy options are either:

  1. Send at lower data rate (not ideal)
  2. Add more object detectors (not ideal)
  3. Make object detectors run faster (my preferred)

Making these run faster will come down to changing the perception algorithms to smaller/faster networks.