[HOW-TO] debug dequeue timer

[tennisparty]

Hello,

I have a fairly complex app running flask, picamera2 and autofocus on boot.

I am often getting this message

[0:05:43.417007419] [1582]  WARN V4L2 v4l2_videodevice.cpp:2007 /dev/video0[18:cap]: Dequeue timer of 1000000.00us has expired!
[0:05:44.308115388] [1582] ERROR RPI pipeline_base.cpp:1374 Camera frontend has timed out!
[0:05:44.308464398] [1582] ERROR RPI pipeline_base.cpp:1375 Please check that your camera sensor connector is attached securely.
[0:05:44.308506534] [1582] ERROR RPI pipeline_base.cpp:1376 Alternatively, try another cable and/or sensor.

I have checked and it definitely isn't a cable issue (happening on multiple pi's and cameras). What does this message mean exactly? I am trying to work out the best way to debug the issue because I can't make it repeatable

Thanks, Sam

[davidplowman]

Hi, thanks for the report.

Obviously that shouldn't be happening. It might be worth double-checking what kind of Pi you are using, what version of the OS (Bullseye or Bookworm), that the software is as up-to-date as it can be, and what kind of camera this is. Also, just to rule things out, how long is the cable, and can you confirm there are no obvious sources of interference nearby? Finally, are you using a Raspberry Pi power supply (it's well known that many third party "cheap" power supplies don't supply the current/voltage they claim).

Out of interest, does the camera keep running, or is it then locked up?

One thing you can try is to locate the rpi_apps.yaml file. Usually I think this is in /usr/share/libcamera/pipeline/rpi/vc4 for most Pis (replace vc4 by pisp for a Pi 5). In that file you can add the line:

    "camera_timeout_value_ms":    5000000,

to push the timeout up to 5s. If there's nothing much wrong but only some marginal timings, then this will help. Though I have my doubts.

@naushir Anything else you can think of to investigate?

[naushir]

Some users have reported this very issue when using using a 3rd party sensor with IR emitters on the sensor board. During daytime operation things are fine, but at night time, when the IR emitters are turned on, the power draw causes the camera to stop streaming and throw this error. Could it be the same problem here?

[tennisparty]

Hi both,

Thankyou.

Some info below:

• I am using a raspberry pi 3a+ running Bookworm lite.
• The software is up-to-date.
• I am using an official raspberry pi camera module 3 (both NoIR, wide and normal).
• The cable is the standard length supplied with the camera module.
• I am using a Raspberry Pi power supply and have also tried a wittypi4 off a 12v battery.
• The camera does stop running and is then locked up.
• I seem to be able to re-produce the issue if I shutdown, completely remove power and then boot the system and software.

I will try adding the line to the rpi_apps.yaml file and feedback.

Thanks again, Sam

[tennisparty]

Hello,

A quick update. Increasing the camera_timeout_value to 5s seems to fix the issue. I am not receiving the dequeue error.

I have tried to explore this a little further. If I reset the camera timeout value so it is 'default' I do not receive the camera dequeue message when I do not run autofocus whilst boot. The dequeue message is printed and the camera locks up if I run the autofocus described here Autofocus - issue 915 in a thread alongside a mjpeg stream/ webserver, the issue is erratic and doesn't happen on every boot.

[tennisparty]

Sorry, just to supplement that last post, I was running another pi without the autofocus on boot (with the camera timeout on default) and got once instance of the dequeue timer running out in a 24hr period. The pi was resetting every 5 mins

[davidplowman]

Hi again, I'm afraid I've got a bit confused as to exactly what isn't working now. Would you be able to summarise that again?

If you're able to create a short self-contained example that demonstrates the problem then we could try to reproduce it here. Thanks!

[Squeack]

I have a very similar issue. Using a wide-angle v3 camera on a Pi Zero 2W, and with the rpi_apps.yaml already set timeout set to 10s, I still get a very similar error to that above (an extra zero in the timeout and different line numbers for pipeline_base.cpp, so probably a libcamera different version).

[331:02:32.983494193] [1293]  INFO Camera camera_manager.cpp:297 libcamera v0.0.5+83-bde9b04f
[331:02:33.173252533] [1295]  INFO RPI vc4.cpp:437 Registered camera /base/soc/i2c0mux/i2c@1/imx708@1a to Unicam device /dev/media2 and ISP device /dev/media0
[331:02:33.173427897] [1295]  INFO RPI pipeline_base.cpp:1101 Using configuration file '/usr/share/libcamera/pipeline/rpi/vc4/rpi_apps.yaml'
Stream configuration adjusted
[331:02:33.183831827] [1293]  INFO Camera camera.cpp:1033 configuring streams: (0) 640x480-YUV420 (1) 2304x1296-SBGGR10_CSI2P
[331:02:33.184642710] [1295]  INFO RPI vc4.cpp:565 Sensor: /base/soc/i2c0mux/i2c@1/imx708@1a - Selected sensor format: 2304x1296-SBGGR10_1X10 - Selected unicam format: 2304x1296-pBAA
[331:43:48.332848098] [1295]  WARN V4L2 v4l2_videodevice.cpp:2007 /dev/video0[13:cap]: Dequeue timer of 10000000.00us has expired!
[331:43:50.413275084] [1295] ERROR RPI pipeline_base.cpp:1333 Camera frontend has timed out!
[331:43:52.697810696] [1295] ERROR RPI pipeline_base.cpp:1334 Please check that your camera sensor connector is attached securely.
[331:43:54.127755142] [1295] ERROR RPI pipeline_base.cpp:1335 Alternatively, try another cable and/or sensor.
ERROR: Device timeout detected, attempting a restart!!!

With the timeout increased from 1s to 10s I get far fewer timeouts than I used to, but they still happen, and seemingly at random times. The camera application might run for days, or just minutes before it happens. They are not restricted to very dark environments as another issue covers.

I have a second, nearly identical setup, but with a v2 NOIR camera and no timeout set in rpi_apps.yaml which seems to run much more reliably.

Possibly useful details

$ libcamera-hello --list-cameras
Available cameras
0 : imx708_wide [4608x2592] (/base/soc/i2c0mux/i2c@1/imx708@1a)
    Modes: 'SBGGR10_CSI2P' : 1536x864 [30.00 fps - (0, 0)/0x0 crop]
                             2304x1296 [30.00 fps - (0, 0)/0x0 crop]
                             4608x2592 [30.00 fps - (0, 0)/0x0 crop]

$ grep timeout /usr/share/libcamera/pipeline/rpi/vc4/rpi_apps.yaml
                # Set the timeout for camera response, in ms
                "camera_timeout_value_ms": 10000,

$ cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 11 (bullseye)"
NAME="Raspbian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

$ dpkg-query -l | grep libcam
ii  libcamera-apps-lite                  1.2.1-1                          armhf        libcamera-apps-lite
ii  libcamera0:armhf                     0~git20230720+bde9b04f-1         armhf        complex camera support library
ii  python3-libcamera                    0~git20230720+bde9b04f-1         armhf        complex camera support library (Python 3 bindings)

Last few lines of /proc/cpuinfo
Hardware        : BCM2835
Revision        : 902120
Serial          : 000000000649a90f
Model           : Raspberry Pi Zero 2 W Rev 1.0

Official RPi PSU, normal (15cm?) camera ribbon cable

Application code running is the streaming utility at https://github.com/Squeack/libcamera-broadcaster, which does not explicitly set auto-focus on or off, so it is whatever the default is. The current view is from a window over the garden, so autofocus is unlikely to trigger anyway.

[davidplowman]

@lurch Do you want to see if you can reproduce this? I think I'm seeing that we need a Pi Zero 2 W, Bullseye lite, a v3 camera, and then install and run https://github.com/Squeack/libcamera-broadcaster. Does that sound right?

[Squeack]

Or I can run any debug-augmented code you might want to try? The broadcaster code does not have any service/systemctl wrapper around it yet. I start it through a line in /etc/rc.local

runuser -u pi /usr/bin/node /home/pi/libcamera-broadcaster/ws+http-server.js &

Then it dies, and I restart it as a nohup command, but without the runuser, which is what gave me the errors above, in nohup.out.

[davidplowman]

Thanks for the offer, but I'm not quite sure what I'd actually want to debug, except for "everything" (which would probably change all the timings and make everything fail differently, or even work). So we probably just need to try it first and see if we get the error.

Presumably we can just run the server directly from the command line, connect a remote machine to view the camera, and that should be it. Anything else we would need to do to set it up?

[Squeack]

Just copy the files from github and follow the installation instructions in readme.md should be enough. The video stream should then be visible from any modern browser going to http://server:2050/camera.html If it makes a difference, I am currently running node v12.22.12 and npm v7.5.2, but that should be unimportant.

[Squeack]

One difference from the github code is that I changed the last line of ws+https-server.js to

new Broadcaster({width: 640, height: 480, framerate: 20, hdr:1, port:8080, mode:'1536:864'});

Adding the mode parameter makes it use the full FOV instead of clipping to the centre region. The mode is camera model specific so needs to be set to different values for v2 and v3 cameras, according to the values given by libcamera-hello --list-cameras.

[oliviacarino]

I too have been dealing with the camera timeout issue. I'm posting from mobile, but can add more information if needed.

I have tried the following Pi's: Pi 3 A+ Pi 4 B

I have tried the following cameras: Raspberry Pi Camera Module V3 Raspberry Pi Camera Module V3 NoIR

I have tried the following OS systems and their latest, stable version: Bullseye Bookworm

Running libcamera-hello --list-cameras, both cameras on both Pis are able to be detected. However, any other libcamera command I use causes the error (libcamera-hello, libcamera-jpeg, etc.)

I tried a combination of all of the above with each other and continued to get the same error that everyone here has mentioned. I tried setting my camera timeout to be 5s and even 10s. Sadly, this did not work. I have installed fresh images of each OS multiple times on the Pis, ran apt update and apt upgrade, this did not fix the issues either. Both cameras had the same error on both Pis and both OS (please note, I tried each OS and cameras in different combinations on each Pi).

I even tried turning on Glamor for thr Pi3, changing the gpu-memory on the Pi3 to 256, and both things did not help :/ so I ended up buying a Pi4B thinking it was a memory issue, but alas, it was not.

Reseating each camera multiple times didn't work either.

I also tried running libcamera with debug switched on and noticed in the error logs that the camera module component cannot be opened. (I can post that exact message later today or tomorrow if needed).

Any help or tips on how else to fix this would be much appreciated. I feel like I'm going crazy lol

[Squeack]

Hi Olivia, Have you tried disconnecting and carefully reconnecting the camera cable, then checking it is straight and fully inserted at both ends? Moving to a Pi 4 implies that you have changed it at least once, but a slightly angled cable can cause problems so it is something to check.

[oliviacarino]

@Squeack I've reseated the cable on both sides of the camera module multiple times and the timeout error continues to persist :/

[oliviacarino]

@davidplowman should I open a new issue for my previous comment? It's related to this issue's original content and I kind of just jumped in on the convo.

I too have been dealing with the camera timeout issue. I'm posting from mobile, but can add more information if needed.

I have tried the following Pi's: Pi 3 A+ Pi 4 B

I have tried the following cameras: Raspberry Pi Camera Module V3 Raspberry Pi Camera Module V3 NoIR

I have tried the following OS systems and their latest, stable version: Bullseye Bookworm

Running libcamera-hello --list-cameras, both cameras on both Pis are able to be detected. However, any other libcamera command I use causes the error (libcamera-hello, libcamera-jpeg, etc.)

I tried a combination of all of the above with each other and continued to get the same error that everyone here has mentioned. I tried setting my camera timeout to be 5s and even 10s. Sadly, this did not work. I have installed fresh images of each OS multiple times on the Pis, ran apt update and apt upgrade, this did not fix the issues either. Both cameras had the same error on both Pis and both OS (please note, I tried each OS and cameras in different combinations on each Pi).

I even tried turning on Glamor for thr Pi3, changing the gpu-memory on the Pi3 to 256, and both things did not help :/ so I ended up buying a Pi4B thinking it was a memory issue, but alas, it was not.

Reseating each camera multiple times didn't work either.

I also tried running libcamera with debug switched on and noticed in the error logs that the camera module component cannot be opened. (I can post that exact message later today or tomorrow if needed).

Any help or tips on how else to fix this would be much appreciated. I feel like I'm going crazy lol

[davidplowman]

No problem, I'm fine to leave everything here for now. There seem to be enough reports that we need to have a go at reproducing this, so I'm hoping we'll get round to it during next week. Then we'll see where that takes us!

[lurch]

I've now managed to get @Squeack 's software running on a Pi Zero 2 W with Bullseye Lite and a Camera Module 3 (it gives impressively low-latency streaming, at least compared to the last time I did anything like this), so I guess now I just need to wait and see if I get the timeout error... :slightly_smiling_face:

[oliviacarino]

If it helps, I ran the camera-bug-report tool and uploaded the error log .txt files to this issue

I can upload them here if that's easier too :)

[lurch]

@Squeack I've now been running your example application (with the one-line tweak you suggested) continuously for a full week, and haven't seen a single error. My terminal window still says:

pipi@bullseye32-lite:~/libcamera-broadcaster $ node ws+http-server.js
Creating new broadcaster
(node:7149) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
[0:56:33.186845626] [7156]  INFO Camera camera_manager.cpp:297 libcamera v0.0.5+83-bde9b04f
[0:56:33.335562173] [7157]  INFO RPI vc4.cpp:437 Registered camera /base/soc/i2c0mux/i2c@1/imx708@1a to Unicam device /dev/media3 and ISP device /dev/media0
[0:56:33.335692111] [7157]  INFO RPI pipeline_base.cpp:1101 Using configuration file '/usr/share/libcamera/pipeline/rpi/vc4/rpi_apps.yaml'
Stream configuration adjusted
[0:56:33.345661780] [7156]  INFO Camera camera.cpp:1033 configuring streams: (0) 640x480-YUV420 (1) 2304x1296-SBGGR10_CSI2P
[0:56:33.346340166] [7157]  INFO RPI vc4.cpp:565 Sensor: /base/soc/i2c0mux/i2c@1/imx708@1a - Selected sensor format: 2304x1296-SBGGR10_1X10 - Selected unicam format: 2304x1296-pBAA

and my web-browser window is still successfully streaming video. I started with a clean Bullseye-Lite image (2023-12-05-raspios-bullseye-armhf-lite.img.xz) on a PiZero 2W with a Camera Module 3, did a full apt-upgrade, and apt-installled git, nodejs & npm (with the latter installed with --no-install-recommends to avoid pulling in a bunch of GUI-related packages).

So unless you can come up with a more easily-reproducible problem, I'm not sure what else we can do here.

[tennisparty]

Hi again, I'm afraid I've got a bit confused as to exactly what isn't working now. Would you be able to summarise that again?

If you're able to create a short self-contained example that demonstrates the problem then we could try to reproduce it here. Thanks!

Hi David, sorry for not replying sooner. Increasing the timeout to 10 seconds has stopped all the errors in my code. I'm on quite a short deadline so have accepted 10 seconds as the solution and am implementing that on each install.

I'm happy to send you a copy of the whole code for debugging but probably won't have time to cut it down to the core picam2 elements myself. I'm creating a custom camera to sell so would have to send the code privately, sorry if that's a hassle,

Thanks, Sam

[Squeack]

@lurch Thank you for taking a look. Increasing the timeout through 1s, 5s, 10s has made the error less common, so I might just have to increase it further and just accept the occasional rare pause in the video.

[mikeshpatelnz]

Hi, thanks for the report.

Obviously that shouldn't be happening. It might be worth double-checking what kind of Pi you are using, what version of the OS (Bullseye or Bookworm), that the software is as up-to-date as it can be, and what kind of camera this is. Also, just to rule things out, how long is the cable, and can you confirm there are no obvious sources of interference nearby? Finally, are you using a Raspberry Pi power supply (it's well known that many third party "cheap" power supplies don't supply the current/voltage they claim).

Out of interest, does the camera keep running, or is it then locked up?

One thing you can try is to locate the rpi_apps.yaml file. Usually I think this is in /usr/share/libcamera/pipeline/rpi/vc4 for most Pis (replace vc4 by pisp for a Pi 5). In that file you can add the line:

    "camera_timeout_value_ms":    5000000,

to push the timeout up to 5s. If there's nothing much wrong but only some marginal timings, then this will help. Though I have my doubts.

@naushir Anything else you can think of to investigate?

Hi!

Sorry to sound stupid but can someone give me some instructions on how to action this?

I have a raspi4 connected to cam 3. I have this setup on voron 3d printer using klipper.

I know how to loginto SSH using Putty but can't execute that command and change the timeout timer in .yml

Do I need to just plug the SD into PC and open some file or? Sorry never used this kind of commands before.

thanks for your help.

[davidplowman]

If you can log in with ssh (via Putty for example), you should be able to run something like

sudo nano /usr/share/libcamera/pipeline/rpi/vc4/rpi_apps.yaml

assuming that's where the file is for you. You'll need root privileges to edit it (hence sudo), and nano is a simple text editor.

You might be able to plug the SD card into a PC and do it there, but the PC would have to recognise the filesystem. I think a PC running Linux probably would, but a Windows PC probably wouldn't (it would probably recognise the "boot" partition, but not the "root" partition which is where most files are).

[wilber82]

Hi, thanks for the report.

Obviously that shouldn't be happening. It might be worth double-checking what kind of Pi you are using, what version of the OS (Bullseye or Bookworm), that the software is as up-to-date as it can be, and what kind of camera this is. Also, just to rule things out, how long is the cable, and can you confirm there are no obvious sources of interference nearby? Finally, are you using a Raspberry Pi power supply (it's well known that many third party "cheap" power supplies don't supply the current/voltage they claim).

Out of interest, does the camera keep running, or is it then locked up?

One thing you can try is to locate the rpi_apps.yaml file. Usually I think this is in /usr/share/libcamera/pipeline/rpi/vc4 for most Pis (replace vc4 by pisp for a Pi 5). In that file you can add the line:

    "camera_timeout_value_ms":    5000000,

to push the timeout up to 5s. If there's nothing much wrong but only some marginal timings, then this will help. Though I have my doubts.

@naushir Anything else you can think of to investigate?

I don't see rpi_apps.yaml in /usr/share/libcamera/pipeline/rpi/pisp on Raspberry Pi 5. There is a file called example.yaml out there. Should I rename it to rpi_apps.yaml and add the parameter?

[naushir]

We don't use a settings file on Pi5, so it's not available by default. To set one up, you should copy example.yaml to something like pi5.yaml. Then in your python script, set

os.environ['LIBCAMERA_RPI_CONFIG_FILE'] = '/<path>/<to>/pi5.yaml'

before starting Picamera2.

[wilber82]

Just share my experience.

I had the same issue on Raspberry Pi Camera Module 3 NoIR model on a Raspberry Pi 5 on the cam1 and cam0.

Then I got a Raspberry Pi Camer Module 3 NoIR Wide model. It throw “_error:XDG_RUNTIMEDIR not set in the environment” error on the same Pi 5 when I used a 20cm cable. But it worked fine after I changed to another 50cm cable.

[mitya12342]

I am having similar timeout issue after connecting rpi 3b+ to rev1.3 camera with 2m cable. If I increase the timeout, it just hangs. Should i file another issue because for some reason it runs completely fine with legacy mode and raspistill/raspivid?

[davidplowman]

Hi, I think you would have to reproduce the problem with a ~30cm length cable. They're just not designed to be much longer than that, and there's no reason why different software would behave the same. If it has worked in the past with a 2m cable, that is in large part probably luck and it's quite likely there's nothing we can do to make it work reliably.

[mitya12342]

Yea, that is the thing, it works with original 15cm cable in both legacy and libcamera modes, but with 2m cable only legacy can get images so i thought it might be some software issue. libcamera-hello can recognize the camera but no image received. I dont have other cables to reproduce with like 30cm so I will stick with legacy stuff for now. I am happy to provide any details if that will help with development, including remote access to pi.

[naushir]

The legacy camera software has a very aggressive timeout/retry mechanism. It may seem to be working, but under the hood it is glitching and stopping/restarting over and over again. This aggressive retry code is not possible to recreate in Linux / V4L2 as-is, so this is why things seem worse.

[davidplowman]

Mostly I think people find the cables are pretty reliable up to about 50cm. You can get some quite nice shielded gold coloured cables for the narrow camera connector like you have on a Pi Zero 2W, but if you really need 2m then that's obviously not enough. Even when it appears to work I would regard 2m cables as high suspect - they'll be prone to all manner of interference, random corruptions even if you're not seeing a lock-up.

[nickoppen]

It sounds like this problem may occur for a number of reasons some of which may be outside the control of the programmer.

Is there a way of detecting the timeout from with the program (c++ in my case) so that it can make some sensible decision and keep running or at least degrade gracefully? I'm currently using a latch to block the main thread until the callbacks are done and if the camera times out and even one frame is missed then the main thread remains blocked.

I'm using a pi5 and the HQ camera (imx477)

[davidplowman]

The next version of Picamera2 (already on the next branch in this repo) has some extra timeout handling. When you do a straightforward "capture", you can put in a simple timeout value (rather than just block/don't block). Realistically, when that happens, the only choice you have is to carry on waiting (if you are doing the capture asynchronously), or simply to clear everything out, possibly shut the camera down, then cross your fingers and hope for better luck if start over. We'll be releasing that soon-ish (I'd guess 2 or 3 weeks).

In the C++ code you can obviously do all the same stuff - though I don't think anything has really been done for you there.