[BUG] Threads not terminating after camera is closed

[signag]

Hello,

Bug Description

I am working on a Flask-based web server for controlling Raspberry Pi cameras with picamera2 (raspi-cam-srv) Among others, this includes motion detection with recording of .mp4 videos.

Users had recognized that the number of threads of the server process is increasing with time.

To analyze the situation, I have written a small test program which monitors the number of threads after specific process steps for different scenarios.

For results, see Console Output, below.

Obviously, already the import of Picamera2 starts 2 threads (only observed on Pi5).

Within the loop, the encoder is started and stopped 5 times, while the camera keeps running.
Each encoder start seems to start 9 threads. However, these threads do not in all cases terminate after the encoder has been stopped.
After the first run, 9 threads are still alive after the camera had been closed and the reference to the camera has been removed.

In the example, all threads seem to have terminated after the second cycle (only the 3 threads from import survive).
However, I have also seen cases where these threads lived for hours.

With other encoders, the behavior was as expected: No additional threads survived after the camera was closed.

To Reproduce

The attached test program can be used to reproduce the described bahavior.

TestPicamera2Threads.zip

Expected Behaviour

I would expect that no additional threads survive after the encoder has been stopped and the camera has been closed.

Console Output, Screenshots

Below is the function for recording .mp4 videos:

def doCapture_mp4(ser, count, pause):
    """Capture MP4"""
    print(f'\nCapturing mp4')
    picam2 = None
    print(f'picam2: None               #Threads: {countThreads(PROCESSNAME)[2]}')
    picam2 = Picamera2(0)
    print(f'picam2: Instantiated       #Threads: {countThreads(PROCESSNAME)[2]}')
    config = picam2.create_video_configuration()
    picam2.configure(config)
    print(f'picam2: configured         #Threads: {countThreads(PROCESSNAME)[2]}')
    picam2.start(show_preview=False)
    print(f'picam2: started            #Threads: {countThreads(PROCESSNAME)[2]}')
    time.sleep(2)
    for n in range(0, count):
        fn = f'test_{ser}_{n}.mp4'
        print(fn)
        encoder = H264Encoder()
        encoder.output = FfmpegOutput(fn, audio=False)
        picam2.start_encoder(encoder)
        print(f'picam2: encoding mp4       #Threads: {countThreads(PROCESSNAME)[2]}')
        time.sleep(pause)
        picam2.stop_encoder(encoder)
        time.sleep(2)
        print(f'picam2: encoder stopped    #Threads: {countThreads(PROCESSNAME)[2]}')
    picam2.stop()
    print(f'picam2: Stopped            #Threads: {countThreads(PROCESSNAME)[2]}')
    picam2.close()
    print(f'picam2: closed             #Threads: {countThreads(PROCESSNAME)[2]}')
    del picam2
    gc.collect()
    print(f"picam2: del'd              #Threads: {countThreads(PROCESSNAME)[2]}")

For recording .mp4 videos with H264Encoder and FfmpegOutput, the test gives the following output:

Start                      #Threads: 1
Picamera2 imported         #Threads: 3
Encoders imported          #Threads: 3
Select capture mode: 1=jpg, 2=mjpg(JpegEncoder), 3=mjpg(MJPEGEncoder), 4=mp4 0:terminate
4

Capturing mp4
picam2: None               #Threads: 3
picam2: Instantiated       #Threads: 9
picam2: configured         #Threads: 9
picam2: started            #Threads: 11
test_1_0.mp4
picam2: encoding mp4       #Threads: 20
picam2: encoder stopped    #Threads: 20
test_1_1.mp4
picam2: encoding mp4       #Threads: 29
picam2: encoder stopped    #Threads: 29
test_1_2.mp4
picam2: encoding mp4       #Threads: 38
picam2: encoder stopped    #Threads: 38
test_1_3.mp4
picam2: encoding mp4       #Threads: 47
picam2: encoder stopped    #Threads: 47
test_1_4.mp4
picam2: encoding mp4       #Threads: 56
picam2: encoder stopped    #Threads: 56
picam2: Stopped            #Threads: 55
picam2: closed             #Threads: 48
picam2: del'd              #Threads: 12
Press <Enter> to continue or <Ctrl+C> to terminate

Capturing mp4
picam2: None               #Threads: 12
picam2: Instantiated       #Threads: 18
picam2: configured         #Threads: 18
picam2: started            #Threads: 20
test_2_0.mp4
picam2: encoding mp4       #Threads: 29
picam2: encoder stopped    #Threads: 20
test_2_1.mp4
picam2: encoding mp4       #Threads: 29
picam2: encoder stopped    #Threads: 20
test_2_2.mp4
picam2: encoding mp4       #Threads: 29
picam2: encoder stopped    #Threads: 20
test_2_3.mp4
picam2: encoding mp4       #Threads: 29
picam2: encoder stopped    #Threads: 20
test_2_4.mp4
picam2: encoding mp4       #Threads: 29
picam2: encoder stopped    #Threads: 20
picam2: Stopped            #Threads: 19
picam2: closed             #Threads: 12
picam2: del'd              #Threads: 3
Press <Enter> to continue or <Ctrl+C> to terminate

Hardware

RPi 5
Attached cameras:

• (0): module 2 camera ("Model": "imx708")
• (1): module HQ camera ("Model": "imx477")

OS: Bookworm
Debian version: 12.5

Additional Context

[davidplowman]

Thanks for the report and the code to reproduce it.

It seems to be specific to the FfmpegOutput, where the ffmpeg process isn't cleaning up completely. It seems that adding a gc.collect() when the FfmpegOutput stops mostly solves the problem, and the thread count never goes above 20, so I'll make that change. (I'm not really clear why this is necessary, though I note that the refcount on the subprocess object is always 2, so I wonder if there's a circular reference going on? Who knows...)

In the meantime, I'd recommend adding

encoder = None
gc.collect()

to your code just after picam2.stop_encoder(encoder). You might even want to leave this in once you've got the proposed fix as (for me at least!) it limits the number of threads to 13.

[signag]

Thanks for the quick response.

I am following your recommendation.

However, I am not sure whether it is the ffmpeg process which does not clean up.
ffmpeg runs in an own process with typically 2 threads which all vanish after encoding was completed. (I am showing now ffmpeg process information along with main process data in a Camera Info screen)

I also need to correct:
2 threads are started with import of Picamera2 in case of Bookworm, even on a Pi Zero 2W.
For Bullseye, no additional thread is started with import.

[davidplowman]

In truth, I wouldn't like to claim I fully understand exactly what is happening, and I don't know why Bookworm would be different to Bullseye. But if there's still a problem with these threads, please do report back (preferably with some code I can run - that was very helpful!) and we can investigate some more.