Frequent trashed frames

[icolwell-as]

Seems to happen about every 5-10 seconds with the Mako G-319C. It puts a dent in the output frequency of the driver since it appears as though some frames are not coming through correctly.

06-04-2020 10:09:09.450 [T] 1900 Data.Buffer ==NULL
06-04-2020 10:09:09.450 [T] Trashed frame! 3156,1 0
06-04-2020 10:09:09.500 [T] 1900 Data.Buffer ==NULL
06-04-2020 10:09:09.500 [T] Trashed frame! 3157,1 0
06-04-2020 10:09:09.550 [T] 1900 Data.Buffer ==NULL
ERR: FrameObserver VmbFrameStatusIncomplete
06-04-2020 10:09:09.550 [T] Trashed frame! 3158,1 0
06-04-2020 10:09:14.127 [T] Trashed frame! 3249,1481 0
ERR: FrameObserver VmbFrameStatusIncomplete
06-04-2020 10:09:14.837 [T] Trashed frame! 3263,2137 0
ERR: FrameObserver VmbFrameStatusIncomplete
06-04-2020 10:09:15.024 [T] Trashed frame! 3267,1117 0
ERR: FrameObserver VmbFrameStatusIncomplete
06-04-2020 10:09:28.899 [T] 1900 Data.Buffer ==NULL
06-04-2020 10:09:28.900 [T] Trashed frame! 3545,1 0
06-04-2020 10:09:28.949 [T] 1900 Data.Buffer ==NULL
06-04-2020 10:09:28.950 [T] Trashed frame! 3546,1 0
06-04-2020 10:09:28.999 [T] 1900 Data.Buffer ==NULL
06-04-2020 10:09:29.000 [T] Trashed frame! 3547,1 0

[isaacvandor]

@icolwell-as Any insight into what's causing this? Facing a similar issue with incomplete status for a similar AVT driver

[icolwell-as]

@isaacvandor, unfortunately no, I never looked into this in detail. The only thing I was thinking was to upgrade the "libVimba" library to the latest version and hope it improves things, the one embedded in this driver is version 2.0.

Out of curiosity: What driver are you using? And what libVimba/VimbaSDK version are you using?

[PhilippSchmaelzle]

Hi @isaacvandor , @icolwell-as , I use AVT Mako G223C camera with 3.1 VimbaSDK and your driver framework.

Got a similar issue, probably the same where sequentially multiple frames (in my case always 10 frames in a row) are lost/trashed. I haven't fixed it completely but at least reduced the frequency of it's occuring in my case (from avg > 50/drops to < 1 drops/min)

With increasing the buffer size of the UDP rx queue within the system:

• check buffer size with sudo sysctl 'net.core.rmem_max' => on my machine it was 212992
• Increase buffer size with sudo sysctl -w 'net.core.rmem_max=26214400'

Plus UDP package size needs to be set to >8228 Bytes, but I guess thats known.

[isaacvandor]

Thanks @PhilippSchmaelzle, I'll give that a try! @icolwell-as: I've been primarily using version 3.1, but have tested up to 4.2 and the issue persists. The driver is not open sourced yet, so I can't say too much about it unfortunately. It's mostly a port of an old LCM driver to ROS.

[PhilippSchmaelzle]

@isaacvandor have you had a chance to test it or even solved the framedrop issue by now?

[isaacvandor]

@PhilippSchmaelzle Unfortunately no, I got caught up on some other projects. Hoping to get back around to this sometime next week. I'll let you know how it goes.

[isaacvandor]

@PhilippSchmaelzle Tested on my home machine this morning with no issues after I increased the buffer size. I will test on our vehicle at some point, but it seems like that just works.

[icolwell-as]

I had a chance to test this today. Thank you @PhilippSchmaelzle, your solution worked. For those wondering, the UDP package size can be increased from Ubuntu's default 1500 bytes by setting the "mtu" (maximum transfer unit) on the network interface. I was able to set mine to 9014 as specified in the camera docs (see here page 47: https://cdn.alliedvision.com/fileadmin/content/documents/products/cameras/Mako/techman/Mako_TechMan_en.pdf)

You can check the current mtu by running the following command:

ip a | grep mtu

I don't see any trashed frames or complaints from Vimba at all anymore! I can't seem to push the FPS beyond 20 Hz (even though Mako is advertised to go to 33), but that is likely due to other bottlenecks on the receiving side, and not the fault of the camera hardware.

Thanks again! I'll leave this issue open until I add a section in the README explaining this.