Grab issues starting around the one hour mark

[stbnps]

Describe what you want to implement and what the issue & the steps to reproduce it are:

We are receiving frames from several cameras, and, around the one hour mark, we see GrabSucceeded starts returning false a lot of times:

This screenshot shows the number of grab failures for each camera, you can see that the lines stay mostly flat for a while, and then they we start seeing errors continously.

The error code returned by is GetErrorCode 3774873620, and we've tried tweaking the transmission parameters, but this keeps happening. Also, isn't it a little weird that the error begins happening after one hour of continuous operation with almost no issues?

Thanks!

Is your camera operational in Basler pylon viewer on your platform

Yes

Hardware setup & camera model(s) used

CPU architecture: x86 64 Operating System: Ubuntu 22.04 RAM: 128 GB

Type in type/model/name of interface cards: NetXtreme-E BCM57504 Switches or hubs used: Netgear M4300 Cable types/lengths: Cat 6A

Runtime information:

python: 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0]
platform: linux/x86_64/5.15.0-119-generic
pypylon: 3.0.1 / 7.4.0.38864

[stbnps]

When we restart the camera after each grab failure, using StopGrabbing(), Close(), and initializing the camera again, we see way less grab issues:

[thiesmoeller]

some questions:

• Have you enabled the flowcontrol on your switch ( Rx and TX must be on ) Default on this netgear model is "off" When you have multiple cameras flow control really helps to moderate the load on switch memory and the network card.

• Do all cameras run HW synchronized? or do you start them roughly at the same time only by SW?

[stbnps]

@thiesmoeller We have enabled flow control, but we still see the issue.

We run the cameras independently. We call RetrieveResult / Convert / GetArray roughly at the same time for every couple of cameras. We run the cameras in pairs, one camera is pointing forward, and the other sideways, so we only need the front camera to be roughly synchronized to the side camera.

[thiesmoeller]

From your first plot it seems that you have issues with lost frames all the time, it just exponentially get worse after one hour.

In a properly balanced system setup there will be no grab issues at all.

a) have you configured the network related parameters of your multiple cameras. Especially when they are not synchronized you have to look at the worst case peak bandwidth that can occur, if all cameras send their data in the same moment.

The features GevSCPD ( spacing between network packets ) and GevSCPSPacketSize ( size of each packet ) have the biggest impact.

b) I would assume that there are recovered errors even before you see lost frames.

you could create logging info per camera like this:

import pylon.pypylon as py
def get_stats(cam : py.InstantCamera):
    stat_entries = [
        "Statistic_Failed_Buffer_Count",
        "Statistic_Failed_Buffer_Count",
        "Statistic_Buffer_Underrun_Count",
        "Statistic_Total_Packet_Count",
        "Statistic_Failed_Packet_Count",
        "Statistic_Resend_Request_Count",
        "Statistic_Resend_Packet_Count",
        "Statistic_Total_Buffer_Count"
    ]

    return { e.replace("_","").replace("Statistic",""):getattr(cam.StreamGrabberNodeMap, e).Value for e in stat_entries }

[stbnps]

After optimizing the camera code, the camera parameters, and the network configuration, this issue seems to be gone. Thanks!