Resend doesn't work for Lucid Phoenix camera (on RaspberryPi)

[abrock]

Describe the bug A clear and concise description of what the bug is.

I use Lucid Phoenix cameras (3.2MP, PHX032S-CS-FFC). I connect them directly to USB-GigE-adapters and it works most of the time when I use my laptop as host. However, when I use a RaspberryPi4 (8GB RAM) running a 64bit OS (Linux raspberrypi 5.4.51-v8+ #1333 SMP PREEMPT aarch64 GNU/Linux), upgraded to Debian bullseye I get many buffers with status "timeout". I use the exact same cables and USB-GigE-Interfaces as with the laptop. I use hardware-trigger and I set frame_retention to 1sec, when I trigger the camera I either get the frame immediately or I get a timeout after 1sec. After every frame I read the values ReceivedPacketResendCount and UnavailablePacketResendCount from the camera and when the result was a timeout both numbers go up, when the result was success they stay the same. How much depends on the packet_timeout value, e.g. I tried 100ms and got ReceivedPacketResendCount=10, UnavailablePacketResendCount=7. I also tried 10ms, that only made the numbers go way up (e.g 570/570), not change the number of timeout buffers.

I captured 4 tries using wireshark.

To Reproduce Steps to reproduce the behavior:

• Connect a Lucid Phoenix 3.2MP to a Raspberry Pi 4, have some software wait for images
• Externally trigger the camera
• Get lots of timeout buffers, resend doesn't seem to work.

Expected behavior A clear and concise description of what you expected to happen.

Very few timeout buffers / working resend.

Camera description:

• Manufacturer: Lucid Vision
• Model: PHX032S-CS-FFC
• Interface: Gigabit ethernet

Platform description:

• Aravis version: latest git
• OS: Debian Bullseye (Linux raspberrypi 5.4.51-v8+ #1333 SMP PREEMPT aarch64 GNU/Linux)
• Hardware RaspberryPi4, 8GB RAM

[abrock]

wireshark.zip

[abrock]

There seems to be issues with UDP on RaspberryPi 4: https://www.raspberrypi.org/forums/viewtopic.php?t=264106

I'll test that later, but first I tried a different camera (Baumer VLXT-123C.I) and found that packet resend works nicely on the Pi. This is a recording of one successfully transmitted frame with 1us exposure time and the lens cap on:

baumer.zip

[abrock]

I configured my laptop to randomly drop 2% of all incoming packages like this:

sudo iptables -A INPUT -m statistic --mode random --probability 0.02 -j DROP

And I ran the Cpp_Acqusition demo program coming with the Arena SDK, it connects to the camera and reads 25 frames. This is the wireshark capture:

arena-cpp-acquistion-loss-2.zip

[abrock]

Here is the capture of a single buffer using Aravis under the same conditions:

aravis-16-resends-loss2.zip

The camera reported receiving 16 resend-requests, none of them "unavailable". The count matches the requests I count in the wireshark capture.

[EmmanuelP]

Thanks for the network capture.

It looks like for some reason the Lucid camera resend packets to a wrong port. May be it is a configuration issue.

[EmmanuelP]

Could you recapture the network traffic using aravis, and attach a complete trace, with the initialization part ?

[abrock]

Yes, here:

aravis-with-init.zip

I disabled my own logging and added arv_debug_enable("all:3"); at the start of the program, but in previous tests I've seen that the camera says "I got 100 resend-requests and 50 were invalid".

[abrock]

btw, with the 2% incoming packet loss the demo program by Lucid is unlikely to successfully initialize the camera, takes a lot of tries before one is successful. Aravis initializes the camera every single time, no problem :-)

[EmmanuelP]

The camera sends the requested missing packets to the port that was previously set in the SCP0 register, while the regular stream packet are sent to the port set by aravis.

It looks like an issue in the Lucid camera. I don't know why it works with the other SDK. The only obvious difference is aravis is using WRITEMEM while the other SDK uses WRITEREG. Also Arena seems to use the same port number for SCP0 and SCSP0, which could be a coincidence, but that is unlikely.

[EmmanuelP]

It would help if you could attach the wireshark capture with baumer software including the initialization part.

[abrock]

I used my own software (which uses Aravis) to capture 1 frame from the Baumer camera:

baumer-with-init.zip

I also used Baumer camera explorer to capture several frames from the Baumer camera:

baumer-camera-explorer.zip

I tried to use Baumer camera explorer to capture frames from the Lucid camera but it refused.

[abrock]

I have a workaround:

I found that arv_gv_stream_constructed (arvstream.c) and auto_packet_size (arvgvdevice.c) change the "GevSCPHostPort" feature and I was calling both. I changed my program to never call auto_packet_size, this allowed me to read GevSCPHostPort before and after calling arv_camera_create_stream. Knowing both ports allows me to automatically setup packet forwarding like this:

sudo iptables -t nat -A PREROUTING -p udp -m udp --dport <port read before arv_camera_create_stream> -j REDIRECT --to-ports <port read after arv_camera_create_stream.>

With this forwarding enabled the resend feature seems to works as expected.

The first time the cameras are initialized after booting they report zero for GevSCPHostPort before arv_camera_create_stream, in this case I simply restart the program and it works as expected.

I attached aravis log and wireshark capture for both the first and the second run.

workaround-log.zip

I also asked the Lucid support about this.