FrameStatus.Incomplete when acquiring frames from Raspberry Pi 4

[bjarnekvae]

Hi

I have a setup with a Raspberry Pi 4 connected to a Allied Vision Alvium 1800 U-507m via one of the USB 3.0 ports. I am making a python script for acquiring images from the camera and storing them on a external drive, also connected to the Raspberry Pi via a USB 3.0 port. The higher acquisition rate I get the better.

The problem is that I often get "FrameStatus.Incomplete" errors, even when running the Python script with nice -20 and ionice 3. For my application it is critical that frames are not lost.

So far I've tried grabbing frames asynchronously, synchronously and with software trigger, where the code is mostly copy-paste from this repository (implemented a queue in order to take the load of the callback function). I've tried different acquisition rates and even not storing to the external drive at all and only to memory card at low acquisition rates.

The method that has been the most promising so far is synchronous acquisition at 0.5 FPS and storing to SD-card, but even then I get "FrameStatus.Incomplete" errors at a rate of 1-2% of the received frames.

Also, when I asynchronously acquire frames at 10 FPS and store to the external SSD, I usually do not see any "FrameStatus.Incomplete" errors before the 1018th frame, after this the camera stream halts, this does not happen at e.g 5- and 20 FPS, but I do get much more errors.

I am using the latest version of Vimba (4.0.0) and VimbaPython (1.0.0)

[NiklasKroeger-AlliedVision]

Hi and sorry for the long wait.

I personally never connected one of our USB-Cameras to a Raspberry-Pi, but I seem to recall that there were design choices made with the PI, where the USB Ports were limited by being routed through a single PCIe Link. I am not sure if the memory card is also connected to this Link. But this means, that grabbing an image from the camera passes the data through that single PCIe Link multiple times (camera -> RAM -> storage) which might explain the slow speed you are experiencing. Especially with a higher resolution camera. Some more information on the Raspberry Pis USB system can be found here. Perhaps the command mentioned there to get the USB-Topology of your connected devices can also help find some issues: lsusb -t.

In order to pin-point the issue a little better it would be helpful, if you could try running for example the asynchronous grab C example or view the camera stream with the Vimba Viewer to see if the frames there are also incomplete (The number of frames in the status bar at the very bottom of the Vimba Viewer should increase continuously. Incomplete frames are not shown and therefore do not increment that number) .

One thing that might help is reducing the USB throughput by adjusting the DeviceLinkThroughputLimit feature. It can be found in the Vimba Viewer in the DeviceControl category.

If the problem persists in the Vimba Viewer or the C examples, perhaps our support team has some more ideas on how the fix this. You can reach them with this form.

[NiklasKroeger-AlliedVision]

Another thing that just came to mind: Are you sure, that the Raspberry Pi is able to provide enough power via USB to the connected devices? If you try to run a camera and an external hard-drive from the USB ports I could imagine that this comes close to the power-limits of the USB Ports on the Pi.

[bjarnekvae]

Thank you so much for the help, adjusting the DeviceLinkThroughputLimit to half of default solved the main issue of random frame incomplete errors. However now it seems to always give a "FrameStatus.Incomplete" error at the 1018th received frame and a total camera halt (async grab), independent on FPS, (like I explained in the last paragraph if in my first post). I've tried do disconnect the hard drive and store images to the SD-card at 3 FPS to rule out the potential power restriction, but the problem persists.

Unfortunately, I am unable to test with Vimba Viewer as I have installed the "Lite"-variant (CLI only) of Raspberri Pi OS.

[NiklasKroeger-AlliedVision]

Alright so at least the frames are now arriving. That's a start. But a complete camera freeze at frame 1018 seems very odd. Could this perhaps be some kind of buffer/memory problem where not enough space is available to get the next frame? You could try to reduce the recorded AOI of the camera to get less data from the camera and see, if you are able to receive more frames that way. This can be done by setting the Height and Width features to some smaller value before you start the camera stream:

# assuming you have an opened camera connection in cam
cam.Height.set(128) # you might have to play with the actual values a bit depending on the allowed value range and increment
cam.Width.set(128)

If this allows you to receive more frames, I would assume that there is some kind of problem with how your memory is handled and your system is not able to allocate space for the 1019th frame.

My feeling is however, that this is not an actual issue with the VimbaPython wrapper. To verify this you could try to replicate the issue of only being able to receive a limited number of images with the asynchronous grab example compiled from the provided C examples. If my assumption is correct and this is not a VimbaPython problem our technical support team has more experience in debugging these issues and might be of better help here.

I've tried do disconnect the hard drive and store images to the SD-card at 3 FPS to rule out the potential power restriction, but the problem persists.

Yeah the power question was a bit of a shot in the dark...

[bjarnekvae]

Thank you for helpful pointers. So when adjusting the camera width from 2464 to 2456 seems to work, just asynchronously acquired 10.000 frames with only one frame lost. This is acceptable :)

The solution, for me, does not make sense though..

[NiklasKroeger-AlliedVision]

Glad I could be of help, but this does not really feel like a satisfying resolution to this problem. I will make sure we have an up-to-date entry in our internal bug-tracking for this so we can try to figure out what is going on. If you run into the same issue again feel free to reopen this issue, or create a new one!