Frequency scan crashes (.. when running auto_rx within another Python subprocess)

Hi, I am currently running an instance of radiosonde_auto_rx on my Raspberry Pi 4 connected to a Nooelec NESDR SMArTee v2 dongle. I've been running it continuously for the past weeks and have been experiencing a weird issue lately:

I rebooted my Pi yesterday (17/08/2024) and it worked fine during the day and the last radiosonde (U3260091) was detected at 17:28. After that no further flights were recorded, even though multiple radiosondes should have been in range over the night. When coming back in the morning, the scan plot on the web interface seemed to have crashed, showing no spectrum scan and a "peak" at 0MHz (see below). This is really weird to me because the station is configured to only scan between 402 and 406 MHz. Restarting the systemd service that controls auto_rx fixed the issue, but it has since cropped up again and again.

This behaviour might be related to #685. I have two relevant logfiles that were created during such a failure:

In both cases the frequency scan stops for seemingly no reason. After the first crash, I enabled the debugging option in the station config, here are the last few lines of this logfile:

As you can see the scanner just stops without throwing an error message. The last

DEBUG:Scanner - dft_detect exited in 6.9 seconds with return code 1.

should be followed by

DEBUG:Scanner (RTLSDR 0) - Using detection command: timeout -k 30 10 rtl_fm -M raw -F9 -p 0 -d 0 -s 48000 -f 404020000 - 2>/dev/null | ./dft_detect -t 5 --iq --bw 20 --dc - 48000 16 2>/dev/null

to start radiosonde detection on the second frequency (402.5MHz). For some reason this does not happen.

One quirk of my setup is that I do not start auto_rx directly, but call it from another python script:

 def start(self, frequency=None, radiosonde_type=None):
        """
        Starts auto_rx.py in scanning mode, if no frequency is provided. \n
        Frequency in MHz.
        """

        # create argument chain to start process
        process_arguments = []
        process_arguments.append(self.decoder_directory + "venv/bin/python3")
        process_arguments.append(self.decoder_directory + "auto_rx.py")
        # specify config file
        process_arguments.append("-c")
        process_arguments.append(self.generated_config_path)
        # specify frequency
        if frequency != None and frequency != "SCANNING":
            process_arguments.append("-f")
            process_arguments.append(str(frequency))
        if radiosonde_type:
            process_arguments.append("-m")
            process_arguments.append(radiosonde_type)

        # start process
        self.decoder_process = subprocess.Popen(
            process_arguments,
            cwd=self.decoder_directory,
            stdin=None,
            stdout=subprocess.PIPE,
            stderr=None,
        )
        logging.debug("Created decoder process with: {}".format(process_arguments))
        return

I'm doing it this way because there are some other python threads I'm experimenting with, that need to start at the same time as auto_rx. I'm currently trying to reproduce the problem when running auto_rx directly as a systemd service. Unfortunately this might take some time as I haven't figured out how to intentionally crash the scanner.

During testing, these were some of the circumstances were I observed a crash:

messing with the web control during receiving (start/stopping decoder or scanner)
transition from radiosonde decoding to scanning
refreshing the web interface a lot
plotting a lot of historical data

Again, I am unable to reliably reproduce the issue and it might be totally unrelated to what I was doing to the web interface at that time.

Maybe someone has experienced similar issues before and is able to help me here.

Greetings, Moritz

PS: I have tried blacklisting 0MHz in the station config, but unsurprisingly that did not solve any issues 😅

projecthorus / radiosonde_auto_rx

Frequency scan crashes (.. when running auto_rx within another Python subprocess) #896