libcamerify causes motion daemon to create zombie processes

[AntiSol]

Hello,

I've been trying to track down the cause of this issue: https://github.com/motioneye-project/motioneye/issues/2900, and it seems that the problem is being caused by libcamerify.

If you read through my comments on that ticket, you'll see that I was able to track down the source of the zombie processes to the motion daemon. From there, I found this ticket (https://github.com/Motion-Project/motion/issues/1522) on the motion project which seems to indicate that the issue only happens when using libcamerify to run motion.

Unfortunately I only have v3 camera modules at hand, so I don't think I have any way to usefully confirm that the issue only happens when using libcamerify, and I'm not sure where I'd go from there even if I could prove that. I'd appreciate any thoughts, or advice as to what I can do to help track down the cause further?

Thanks for your time :)

[naushir]

Unfotunately I don't know anything about how the libcameraify script works. You will possibly get more help asking this question directly on the libcamera mailing list.

[kbingham]

Yes, please take this to the list, for completeness and relevance if someone searches for this - I've logged the following on https://github.com/Motion-Project/motion/issues/1522 - but as this issue is closed, it's certainly not the right place to continue the discussion either.

I don't think setting the FPS is handled by libcameraify (the libcamera-v4l2 wrapper layer) yet. Not many people use libcamerify - and the more correct route is to use libcamera API's directly, not go through libcamerify.

That said, if you want to investigate this - take a look at https://patchwork.libcamera.org/patch/15392/ - It looks like that's a patch that implements what you need, but seems like it never got merged.

If you test it, and it works - please report that back to the libcamera-devel list and we can see what's needed to get it integrated, or what additional work is required on top of that.

[AntiSol]

thanks for the responses, I sent a message to the mailing list

[naushir]

Discussing on the mailing list...

[AntiSol]

this should be reopened, as there were no meaningful responses on the mailing list

[kbingham]

did you try the patch I suggested?

[AntiSol]

no, you never responded to the questions I asked about the patch on the mailing list.

[kbingham]

no, you never responded to the questions I asked about the patch on the mailing list.

I thought other people were responding. That's the purpose of the mailing list - to get a wider audience. I can only do so much.

From: https://lists.libcamera.org/pipermail/libcamera-devel/2024-April/041424.html

I see the patch kbingham points me to, but I'm not sure how I'd compile and install that on raspbian without causing issues/conflicts with the packaged version (e.g if I uninstall libcamera0 with apt, it's going to uninstall everything that depends on it, too, right?). Some guidance/resources on how to achieve that would be super helpful.

We have documentation on how to build and install libcamera at https://libcamera.org/getting-started.html

[AntiSol]

nobody responded in any meaningful way, One person responded to say that they weren't experiencing the same problem. Certainly nobody actually looked into the issue. Meanwhile I got tons of spam on unrelated subjects, making continued subscription to the mailing list untenable for me.

That getting started guide you linked to also doesn't address the questions I asked, like whether following those instructions is going to nerf my system by uninstalling all the stuff that replies on libcamera.

I'm also unconvinced as to how (not) being able to set the fps would be related to child processes being zombified when v4l2-compat.so is loaded via LD_PRELOAD? From what I've been able to discern the issue is likely related to not properly handling the SIGCHLD signal when children terminate, and I'm not seeing any actual issues related to fps, and this patch doesn't seem to deal with child processes as far as I can tell, so I'm dubious that that error message you spotted is related to the issue I'm describing. If you'd like to expand on why you think this is related then I'll certainly look into that :)

[kbingham]

You don't need to stay subscribed to the list to be able to send/receive on your thread.

I thought the patch would help because one of the error messages you reported showed a failure while trying to handle frame rates, at which point I could imagine that it would then close down/crash if it didn't get what it wanted, so that needs to be checked (by someone motivated, who can replicate the issue ...)

The handling for frame rate configuration through libcamerify is unimplemented in mainline libcamera (hence there's that patch) .. .but it needs someone to work on it / look into it.

Dec 27 14:57:59 pi3 motion[966]: [0:motion] [NTC] [ALL] conf_load: Processing thread 0 - config file /etc/motioneye/motion.conf
Dec 27 14:57:59 pi3 motion[966]: [0:motion] [NTC] [ALL] config_camera: Processing camera config file camera-1.conf
Dec 27 14:57:59 pi3 motion[966]: [0:motion] [NTC] [ALL] motion_startup: Logging to syslog
Dec 27 14:57:59 pi3 motion[966]: [0:motion] [NTC] [ALL] motion_startup: Motion 4.6.0 Started
Dec 27 14:57:59 pi3 motion[966]: [0:motion] [NTC] [ALL] motion_startup: Using default log type (ALL)
Dec 27 14:57:59 pi3 motion[966]: [0:motion] [NTC] [ALL] motion_startup: Using log type (ALL) log level (WRN)
Dec 27 14:57:59 pi3 motion[966]: [1:ml1:Camera1] [ERR] [VID] v4l2_fps_set: Error setting fps. Return code -1
Dec 27 14:58:00 pi3 libcamerify[560]:     INFO: Reinigung hat begonnen
Dec 27 14:58:00 pi3 libcamerify[560]:     INFO: wsswitch wurde gestartet
Dec 27 14:58:00 pi3 libcamerify[560]:     INFO: Aufgaben wurden gestartet
Dec 27 14:58:00 pi3 libcamerify[560]:     INFO: Der mjpg-Client-Garbage-Collector wurde gestartet
Dec 27 14:58:00 pi3 libcamerify[560]:     INFO: Der Server wurde gestartet
...
Dec 27 14:58:33 pi3 motion[567]: [1:ml1] [NTC] [ALL] mlp_capture: Video signal lost - Adding grey image
...

"Dec 27 14:57:59 pi3 motion[966]: [1:ml1:Camera1] [ERR] [VID] v4l2_fps_set: Error setting fps. Return code -1"

sounds like it fails to set fps and gives up ... that giving up - probably ends up in the zombie processes maybe ? I don't know I haven't attempted to reproduce the issue or investigate. That's where you're best positioned as you have the issue.

[pinchartl]

You don't need to stay subscribed to the list to be able to send/receive on your thread.

You need to be subscribed to send to the list, put you can disable e-mail delivery so you won't receive anything sent on the list.

[AntiSol]

You don't need to stay subscribed to the list to be able to send/receive on your thread.

Oh cool! That's really handy! How do I continue to receive messages on my thread after I've unsubscribed? Maybe I will continue to get them, and it's just that there haven't been any other responses at all since I unsubscribed.

sounds like it fails to set fps and gives up ... that giving up - probably ends up in the zombie processes maybe ? I don't know I haven't attempted to reproduce the issue or investigate.

I would expect if that was the case, it wouldn't work at all, right? i.e libcamerify and motion would just crash when trying to use the camera, right? Or I'd see video at some frame rate other than what I expect? That's not what happens, as mentioned it runs just fine, but processes spawned by motion become zombies when run via libcamerify. I'd be happy to elaborate in more specific detail exactly what happens if the issue reports I've linked to are unclear in some aspect, please do ask if you have any questions at all and I'll do my best to clarify.

Perhaps taking the time to read the full description/history of the problem and look into the issue that has been reported rather than making assumptions and guesses based on an almost-certainly-unrelated log entry would allow you to be more sure about what's going on here.

I really don't know that I am "best positioned" to investigate as I'm neither familiar with libcamera's codebase nor a C(++) programmer. It seems to me that one of libcamera's developers is probably better positioned to make a basic attempt to replicate the issue I've described, which others seem to also be having (for example in the motioneye issue I linked to in the initial report), and which is happening for me very reliably on two different machines. I wouldn't know where to begin at actually tracking down the issue, which has nothing at all to do with framerates as far as I can tell, if that patch doesn't solve it.

Should I raise a separate issue to get a response to my question about trying to compile while also having the packaged version of libcamera installed?

[pinchartl]

You don't need to stay subscribed to the list to be able to send/receive on your thread.

Oh cool! That's really handy! How do I continue to receive messages on my thread after I've unsubscribed? Maybe I will continue to get them, and it's just that there haven't been any other responses at all since I unsubscribed.

Using "reply-to-all" puts you on CC, so you then get CC'ed on all new messages in the mail thread (unless someone drops you manually from the CC list). I would however recommend staying subscribed, and disabling mail delivery. Otherwise your replies to the list will be blocked. We currently don't allow non-subscribers to post, as an anti-spam measure.

[AntiSol]

IIUC I also won't get replies in that case if someone uses the 'reply list' feature.

I don't really see much point in re-subscribing due to the lack of response there. It seems like there's really not much point in me being able to send to the list if I'm not receiving anything from it.

But thanks, I appreciate the advice. If somebody does post something useful on the mailing list thread please feel free to let me know here :)

[AntiSol]

Oh yeah I just remembered: there was also one response on the mailing list from somebody who raised the issue on the mailing list a few months ago and also had no response.

[naushir]

@AntiSol if you want to attempt to build libcamera with the suggested patches you can follow the instructions in the README.rst file in this repo. To ensure it does not conflict with any other libcamera installation, you can do sudo apt purge libcamera* first. I would strongly suggest trying this on a new sd-card so that your main installation will not get trashed.

As mentioned by @kbingham, the fps patch sounds like it would be worth testing out, so I suggest you give that a try.

I'll be closing down https://github.com/raspberrypi/libcamera/issues/132 as it's not related to this repo at all.

[AntiSol]

@naushir thanks for the response, but I remain unconvinced that this patch has anything at all to do with the issue - could you explain which part of this patch is related to signal handling and child processes? As mentioned I read through the patch and I don't see anything to do with signals or children. But it's possible I've missed some detail.

On what basis do you contend that this is "not related to this repo at all"?

[naushir]

On what basis do you contend that this is "not related to this repo at all"?

This repo is Raspberry Pi's downstream fork of libcamera, and any possible changes to libcamerify that may fix this issue must go into the libcamera upstream tree before being put here. Also the Motion application is not a project that is maintained by us or the libcamera maintainers.

[AntiSol]

@naushir oh I see, I'm on the wrong repo! Fair enough! Can you point me at the correct one? Thanks!

[naushir]

Issues with libcamera are discussed on the mailing list. There is also the bug tracker at bugs.libcamera.org, but I will defer to @kbingham to advise if the latter should be used for such issues.

[pinchartl]

A point that seems important to clarify: While libcamera strives to support its users as much as possible, our amount of free time, like everybody else's, is limited. This is true for all open-source projects: reporting an issue on a mailing list, in a bug tracker or in a forum, doesn't guarantee it will be fixed or even investigated right away. The reports that are the most likely to be followed up are the ones that include a proposed patch, or at least a detailed investigation of the problem with targetted questions.

[naushir]

Fully agree with that @pinchartl and want to add that it might be worth getting the maintainers of the Motion application helping with this. Although they may already be, I have not looked through their GitHub issues...

[AntiSol]

@pinchartl This is all well and good, and very true and fair enough, but when I ordered my raspberry pi camera module 3, it didn't say "uses a new and buggier camera library that's not compatible with the software you've been using for decades" in the item description. If it had said that, I'd have bought a camera 2 module, which doesn't suffer from this issue.

Now I'm very sure that's not the fault of the libcamera developers, it's the fault of raspberry pi, who should be putting resources into supporting the open source projects they're relying on and forcing people to use (without putting the aforementioned facts in the item descriptions on their website).

The fact that @kbingham informs me that he does not even work for raspberry pi makes the (lack of) response I've had and zero attempt to replicate the issue from raspi even sadder than it already was. I know where I won't be getting my next single board computer from.

[AntiSol]

@naushir the motion developers have been much much more helpful, e.g I've seen evidence that one of them took 10mins to attempt to replicate the problem! :O

[AntiSol]

sounds like it fails to set fps and gives up ... that giving up - probably ends up in the zombie processes maybe ? I don't know I haven't attempted to reproduce the issue or investigate.

I would expect if that was the case, it wouldn't work at all, right? i.e libcamerify and motion would just crash when trying to use the camera, right? Or I'd see video at some frame rate other than what I expect? That's not what happens, as mentioned it runs just fine, but processes spawned by motion become zombies when run via libcamerify. I'd be happy to elaborate in more specific detail exactly what happens if the issue reports I've linked to are unclear in some aspect, please do ask if you have any questions at all and I'll do my best to clarify.

Just a note for the record on this discussion / line of questioning that nobody could be bothered considering or responding to: I was correct, and the problem was totally unrelated to this patch that @kbingham insisted I waste my time trying, and he had no idea what he was talking about because he hadn't bothered to look.

[naushir]

Funny that the issue was not in libcamera or libcamerify as you insisted though.

For your continued aggressive behaviour, you are now going to be banned from our repos.