Grab times out with GigE camera on Ubuntu 18.04 and Jetson Nano

[milancurcic]

I've been struggling trying to grab from a GigE camera on Ubuntu 18.04 and Jetson Nano. I'm able to connect to the camera and get and set parameters, however any attempt to grab (e.g. cam.GrabOne()) or to cam.RetrieveResult while cam.IsGrabbing fails with a timeout. I'm able to grab images no problem from Pylon Viewer.

My config:

• Jetson Nano (ARM)
• Ubuntu 18.04
• Pylon 6.2.0.21487
• pypylon 1.8.0

Here's what I know:

• I can connect and grab from Pylon Viewer
• I can connect but can't grab from SDK
• I tried a variety of transport layer and camera settings; I also tried loading all the factory settings on the camera, with no change in behavior.
• I'm able to set MTU to 9000 (not sure if needed), but I was unable to set ring buffer size due to it being an unsupported feature on the Jetson's network adapter driver.
• I can reproduce the issue on an x86 Laptop with Ubuntu 20.04 and pylon-6.3.0, so I don't think this is specific to the Jetson Nano or the Pylon version.
• The problem doesn't appear if I use a USB camera.
• Unfortunately this is the only GigE camera I have on hands so I can't try another one.

Do you have any ideas? Am I missing any specific SDK call that's needed for grabbing from GigE cameras?

Minimal reproducer:

from pypylon import pylon

tlf = pylon.TlFactory.GetInstance()
tl = tlf.CreateTl('BaslerGigE')
cam_info = tl.CreateDeviceInfo()
cam_info.SetIpAddress('192.168.1.100')
cam = pylon.InstantCamera(tlf.CreateDevice(cam_info))
cam.Open()
res = cam.GrabOne(1000)

Error message:

Traceback (most recent call last):
  File "test_gige_grab.py", line 33, in <module>
    res = cam.GrabOne(1000)
  File "/home/milan/imager/venv/lib/python3.8/site-packages/pypylon/pylon.py", line 3604, in GrabOne
    return _pylon.InstantCamera_GrabOne(self, *args)
_genicam.TimeoutException: Grab timed out. : TimeoutException thrown (file 'InstantCameraImpl.h', line 1362)

Thank you!

[milancurcic]

FWIW, the ethernet adapter is Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15).

[thiesmoeller]

Please post a minimal grab-sample, where you set all network related parameters. So e.g. what is the value of GevSCPSPacketSize in your program?

[milancurcic]

Thanks, Thies.

I use all the parameters from a PylonViewer session saved into UserSet1. Though I found that camera by default opens with UserSet1 settings, I also load them explicitly:

from pypylon import pylon

tlf = pylon.TlFactory.GetInstance()
tl = tlf.CreateTl('BaslerGigE')
cam_info = tl.CreateDeviceInfo()
cam_info.SetIpAddress('192.168.1.100') #FIXME don't hardcode IP address
cam = pylon.InstantCamera(tlf.CreateDevice(cam_info))
cam.Open()

cam.UserSetSelector.SetValue('UserSet1')
cam.UserSetLoad.Execute()

cam.AcquisitionFrameRateEnable = True
cam.AcquisitionFrameRateAbs = 2

print('Width', cam.Width())
print('Height', cam.Height())
print('PixelFormat', cam.PixelFormat())
print('AcquisitionMode', cam.AcquisitionMode())
print('AcquisitionFrameRateAbs', cam.AcquisitionFrameRateAbs())
print('ExposureTimeAbs', cam.ExposureTimeAbs())
print('GainRaw', cam.GainRaw())
print('GainRaw', cam.GainRaw())
print('GevSCPSPacketSize', cam.GevSCPSPacketSize())
print('GevSCPD', cam.GevSCPD())
print('GevSCPFTD', cam.GevSCFTD())
print('GevSCBWR', cam.GevSCBWR())
print('GevSCBWRA', cam.GevSCBWRA())

res = cam.GrabOne(1000)

results in:

Width 2592
Height 2048
PixelFormat Mono8
AcquisitionMode Continuous
AcquisitionFrameRateEnable True
AcquisitionFrameRateAbs 2.0
ExposureTimeAbs 5000.0
GainRaw 136
GainRaw 136
GevSCPSPacketSize 1500
GevSCPD 0
GevSCPFTD 0
GevSCBWR 10
GevSCBWRA 10
Traceback (most recent call last):
  File "test_gige_grab.py", line 41, in <module>
    res = cam.GrabOne(1000)
  File "/home/milan/bellamare-imager/venv/lib/python3.8/site-packages/pypylon/pylon.py", line 3604, in GrabOne
    return _pylon.InstantCamera_GrabOne(self, *args)
_genicam.TimeoutException: Grab timed out. : TimeoutException thrown (file 'InstantCameraImpl.h', line 1362)

I have two more pieces of info that may be helpful:

1. I found another GigE camera to test with; I get exactly the same behavior (grabs with PylonViewer; fails to grab with SDK) with that camera as well; so I think this rules out the camera as being the culprit.
2. I can see the traffic from camera -> jetson after issuing cam.StartGrabbing(), with the rate consistent with the image size times frame rate. However, any attempt to cam.RetrieveResult() fails with a timeout, even for very large timeout values like cam.RetrieveResult(5000). I attach a screenshot of iftop:

Thanks a lot for looking into this, I really appreciate it.

[thiesmoeller]

Check the values of GevSCPSPacketSize and GevSCPD in pylonviewer when you can grab images.

They are both on their default values. This won't work as your camera is pretty fast in sending packets....

[milancurcic]

Confirming that I can grab from PylonViewer with the default Transport Layer values (see attached).

I met with two other engineers on the project who have much more experience than me with Basler and Pylon. We played with various transport layers parameters following their instruction, as well as using Pylon Viewer's Bandwidth optimizer. There are Transport Layer settings for which the grabbing in Pylon Viewer stops working (e.g. non-zero Errors next to Image count), but for all settings that worked in Pylon Viewer, the grabbing, or more specifically retrieving result while grabbing, does not work from pypylon.

We also tried connecting the camera directly to the Jetson instead of going through a switch, that didn't help either.

Is it possible that Pylon Viewer makes some SDK calls that are needed here and that I'm missing when going via pypylon?

Thanks!

[thiesmoeller]

Just looking at what is different in your setups: Do you also open the camera via ip address in pylon viewer?

[milancurcic]

Yes, connecting via IP address in Pylon Viewer.

Being paranoid and out of ideas, we even tried setting subnet mask and serial number in addition to the IP address in the SDK. It took it, but didn't make a difference.

We also tried putting the camera and the NIC it connects to on a completely different IP address than the other adapter (e.g. 10.0.1.x instead of 192.168.1.x), and that didn't make a difference either.

[SMA2016a]

@milancurcic

Use the pylon generic device info instead of native one.

try this code:

ip_address = "192.168.3.55" cam_info = pylon.DeviceInfo() cam_info.SetPropertyValue('IpAddress', ip_address) cam = pylon.InstantCamera(pylon.TlFactory.GetInstance().CreateFirstDevice(cam_info)) print("Using device ", cam.GetDeviceInfo().GetModelName())

cam.Open() res = cam.GrabOne(1000)

[milancurcic]

Thanks @SMA2016a for the suggestions and sorry about the delay in getting back. We pivoted back to a USB3 camera in order to meet the deadline to deliver to the customer, but I'll be able to revisit the GigE issue later next week.

[SMA2016a]

@milancurcic OK. Then we close it at this stage.