signag / raspi-cam-srv

Web Server for Raspi Camera Access
MIT License
36 stars 4 forks source link

Restarting service doesn't saved reload pan/crop, but 'Load Saved Configuration' does #7

Closed Zopiac closed 3 months ago

Zopiac commented 3 months ago

This is primarily an issue due to the fact that idle streams (watching then leaving the stream) keeps the CPU loaded indefinitely, so I systemctl restart raspiCamSrv on a cron schedule for power/thermal reasons, but anyway:

Restarting the server service resets the pan and crop I have set (lens used is a bit too wide for the given use case), but going to Settings > Load Saved Configuration -- which I assumed would effectively load the same settings as restarting it entirely -- does preserve any pan/crop set on last Store Configuration. Consistency here would be nice, although so too would better cleanup of unused threads when there are no connected clients.

signag commented 3 months ago

This is fixed in V2.3.3 (see Releasenotes)

signag commented 3 months ago

would better cleanup of unused threads when there are no connected clients.

Threads used for streaming (separate for each camera) are automatically stopped after 10 seconds if there is no client streaming.
You can see the respective status indicators being turned off if you refresh the screen in a dialog without live view.
The web client, in general, does not poll; therefore, the status indicators are only updated with a client action or screen refresh.

Zopiac commented 3 months ago

This is fixed in V2.3.3 (see Releasenotes)

Brilliant, works nicely!

Threads used for streaming (separate for each camera) are automatically stopped after 10 seconds if there is no client streaming.

I understand this in theory, but loading a stream and closing it keeps around 22 threads running and my Raspberry Pi 5's cores each loaded at about 30% utilisation. My cron job checks to see if the server has many threads open and if it isn't active on the network, restarts it. Otherwise the system sits at around 70C indefinitely -- and I mean days at end, with the active cooler installed.

signag commented 3 months ago

Are you sure that streaming with raspiCamSrv is the root cause of the issues you are describing?

I do performance monitoring on regular terms on these systems and have not found significant problems, so far.

Here is the output from my monitoring of a 5 hour test run with V2.3.3:
I have indicated when the server (as a service) has been started and stopped and also the times when streaming began and ended.

image

The test has been run on a Pi5 with Debian 12.5 Bookworm.
2 cameras were installed and streaming: a model 3 and a HC camera.
The first client had the raspiCamSrv WebCam dialog open with 2 streams and an additional browser window for each of the two camera streams.
After some time, I had connected with an additional client from a mobile phone, opening just the two streams of the WebCam dialog. Streaming was interrupted from time to time.
So, there were 6 streams in total.
This can be clearly seen on the WiFi transmission rates.

Looking at the CPU temperature, I see an increase from ~47°C in idle state to ~54°C while the server is active.
Also, CPU temperature does not raise with additional clients streaming. It is constant throughout the entire test.

CPU utilization raises from 0% to 8% while the server is active. Also here, I do not see significant change depending on streaming.

Memory consumption increases from 4.5% in idle state to 17% during server activity, absolutely constant throughout the test.

CPU frequency remains constatnt at 1.6 GHz.

signag commented 3 months ago

Just to complete the analysis by counting threads:

Streaming Flask Camera System Encoder raspiCamSrv Total
None 3 3
Cam 1 3 17 1 1 22
Cam 1+2 3 17 + 9 2 2 33
Stopped 3 17 + 9 0 0 29

With 1 camera, you see the 22 threads which you also have mentioned.

When streaming stops after 10 sec if no client is connected, only the encoder and raspiCamSrv threads vanish.
I currently do not stop the camera.
This means that there would be 20 or 29 threads left in case of 1 or 2 cameras, respectively.

If the remaining camera threads would significantly impact system performance, the camera could be stopped if no client is using it. However, up to now, I did not see the necessity.

Zopiac commented 3 months ago

I can't be certain of a whole lot, and simply deal with what I see, so I'm willing to be wrong and chase a different bugbear.

I'm running a Raspberry Pi 5 with Arducam Global Shutter and High Quality cameras attached (started with just the GS) and no other services, really. Basic Raspbian lite installation with just enough installed to get mjpeg streaming running. Usage is light -- only a few hours a day, typically. While I'm not 100% certain that raspiCamSrv is the root cause, restarting it/resetting the threads it has created is certainly working as a solution. For all I know it's a bug with the GS camera module, but I'm not sure how else to test this right now.

At any rate, at first I was noticing thermal runaway when I just had a (beefy) heatsink on it without active cooling, so I installed an active cooler. It was still running fairly hot so I started logging temp and fan speed, before and after telling the service to restart when not in use:

RPi5rpcsNocron

RPi5rpcsWithcron

Usage between these tests wasn't identical, but similar. The service is always "running", but a stream is only opened here and there and closed by nightfall (when the CPU drops due to ambient conditions changing). I had changed fan curve and added the HQ cam between the two tests, but started a new one right now by just disabling the restart routine cron schedule.

Specifically the cron job, every ten minutes, counts the number of threads for each raspiCamSrv instance, and then if threadcount is "high" (>4) and no network activity is seen on its associated port, issues a systemctl restart command for it.

At any rate it's a band-aid solution that was only inadequate because of the pan/zoom I had applied not re-applying on restart, which of course has been fixed. But if there is a further problem it would be nice to determine what it may be, of course.

Zopiac commented 3 months ago

Here's today's log:

RPi5rpcsNocronNew

It is clear when the stream was opened and closed, but it remained heavy on the system afterwards.

Edit: Actually, I think it was only running for about two hours (when the fan would occasionally spike to 3300RPM), and the dip ~9 hours in may have been from the sun going down. At the end I finally restarted the RaspiCamSrv service and the temperature quickly plummeted.

signag commented 3 months ago

This is really strange and completely different compared to what I see. The idle temperature of mid 40 is the same as in my system. However, I do not see these high temperatures.
Also, I can not imagine that this is an effect of the GS camera.
Which stream size do you use?

signag commented 3 months ago

I tried to get rid of these camera threads by stopping the camera(s), closing them and even removing any references to the camera system after streaming has stopped.
Unfortunately, this did not help.

Some of the threads vanished but 25 of 29 survived.
Only when the server is stopped also these threads go away.

So, actually I can't help here.
Probably, I need to contact the Picamera2 team, but my concern would be that it goes deeper to libcamera.

It might be worth looking at the specifics of your system with Raspbian lite and your specific cameras.

Zopiac commented 3 months ago

All right, well thanks for taking a look at the situation! And thank you for resolving the actual title of the topic. Perhaps I'll try a fresh install, lite or not, at some point.

signag commented 2 months ago

Version V2.4.1 has a solution for stopping and closing cameras in times of inactivity (see Release Notes).

Now, when streaming or other background processes are not active, the number of threads goes down to 3 (1 for Bullseye) and CPU utilization should be at a minimum.

If .mp4 videos had been recorded, some threads may have survived (see picamera2 Issue #1023 but they usually do not generate CPU load.