search

projecthorus / radiosonde_auto_rx

Automatically Track Radiosonde Launches using RTLSDR
GNU General Public License v3.0
476 stars 123 forks source link

Add skip option #72

Open pit711 opened 6 years ago

pit711 commented 6 years ago

If the software detects a probe, the decoder remains attached to this probe. But as there are many probes in the air it would be good if the decoder could jump to the next probe after a few seconds.

darksidelemm commented 6 years ago

This is a bit of a difficult one to implement, and kind of goes against how auto_rx currently operates (which is to lock on to a sonde and track it until signal is list). It would also result in gaps in the flight paths.

To track multiple sondes with the current codebase, you can add more RTLSDRs.

pit711 commented 6 years ago

We have multiple probes at the same time on air here. The auto_rx scanner tool is fantastic to catch every probe, because it really scans and does peak detection. other tools are only static frequency tables, and you miss a lot of probes with non fixed frequency's (DFM)

a friend did a quick hack to skip to the next probe, so i can now catch all. but the "hack" is very dirty :) https://pastebin.com/raw/YGBhAvMX

For logging i use https://radiosondy.info/ Its a nice place for probe hunters, would be cool if i could use your scanner here.

darksidelemm commented 6 years ago

I can see the value in this option. We'll need to think a bit about how to implement it in a clean manner, determine listen times, re-scan rates, etc. It will likely need to be a separate 'mode', as it's significantly different to how auto_rx operates at the moment.

It would be a lot cleaner doing this with 2 RTLSDRs - in that case one can be constantly scanning, and then the other RTLSDR can round-robin through the currently detected radiosondes. Doing it with a single RTLSDR will add long gaps while scanning and detection takes place.

darksidelemm commented 6 years ago

Ideas on this (so I don't forget):

The loop would:

The same options that are currently disabled in multi-SDR mode (OziMux output, Rotator control, any uploaders with fixed sonde IDs) would need to be disabled in this mode.

darksidelemm commented 6 years ago

Have raised https://github.com/projecthorus/radiosonde_auto_rx/issues/77 to deal with the APRS uploader hackyness, which is probably why it doesn't work with radiosondy.info at the moment.

flux242 commented 3 years ago

Halloha, I have written a simple bash script to receive multiple sondes in parallel using only 1 rtl-sdr. No, this is not frequency hopping, decoders really work at the same time with different frequencies. Then I patched auto_rx so that it receives the decoded info as json strings over UDP. Here's an example of receiving 3 sondes in parallel https://imgur.com/d7uGjuj

darksidelemm commented 3 years ago

Would you be able to share your script? What changes did you have to make to auto_rx? The UDP input should handle output from multiple instances of any of the RS decoders without issues.

flux242 commented 3 years ago

I'm writing an article about it and I let you know when it's ready. I patched a version that didn't have UPD sensor type. So basically I made sure that auto-rx doesn't touch my rtl-sdr dongle but would still run and added a new UDP sonde type like you did but my cmd was just 'nc -luk 5678'. Then set rx_timeout = 18000000000000 so that it wouldn't exit

darksidelemm commented 3 years ago

Probably better to try it with the latest version of auto_rx then? It would be useful if this could be done without major modifications to auto_rx.

flux242 commented 3 years ago

the change that is needed is to start auto_rx without any radio hardware, so that it would simply sit and listen on a UDP port for decoders json strings. The newest trunk code would still access/stop my dongle if I start it with -m UDP. If I define sdr_quantity = 0 then it wouldn't start at all.

Second thing which could be useful is that beside desoders json string I could provide power measurements in some form. It could also be a json string. For my sonde hunting app I'm adding a 'response_type' tag to every json that is sent back to the client. So each decoder json string is preprocessed with jq --unbuffered -rcM '. + {"response_type":"sonde"}'. In case of the power scanner output I'm adding "response_type":"log_power" into a json string with power measurements. So if the auto-rx is in the UDP state and simply listens it can also show the spectrum

flux242 commented 3 years ago

do you know btw how multiple receiving is implemented in dxlAPRS? Are they using frequency hopping?

darksidelemm commented 3 years ago

Solving the first problem (Allowing auto_rx to run without SDRs) is probably fixable with some kind of override when in UDP mode. Feeding in spectrum data is a bit more difficult, as that runs in a very different thread.

I believe dxlAPRS has some kind of channeliser system, but it seems to have fairly fixed frequency steps. Of course there's always the limitation of the SDRs bandwidth too.

I'd still prefer some kind of network-based server (think: spyserver, but not closed source), that allows a client to connect in and request an IQ stream at a particular frequency. This would allow the greatest flexiblity.

flux242 commented 3 years ago

not sure what channaliser is, but if it what I think then my script works similarly. Tuner is always on the fixed frequency but the IQ stream is multiplexed into number of decoding threads and into the power scanner. Power scanner then tells decoder threads to shift the IQ signal to specific frequency or to stop if no activity detected anymore.

flux242 commented 3 years ago

http://flux242.blogspot.com/2020/08/how-to-receive-and-decode-multiple.html just finished the article and haven't done proof-read yet

rs1729 commented 3 years ago

Does this duplicate the baseband rtl_sdr stream for each signal?

Similar to https://github.com/rs1729/RS/tree/test/demod/multi I made a preliminary TCP version that sends the down-converted IF IQ-data (no demod/decode): https://github.com/rs1729/RS/tree/test/demod/iq_svcl The decoders (or dft_detect) could demodulate and decode the IF data, or a separate demodulator could be used.

flux242 commented 3 years ago

https://github.com/rs1729/RS/tree/test/demod/iq_svcl

looks interesting. This would allow dynamic creation/disposing of decoding processes unlike my current solution where it is statically preallocated

darksidelemm commented 3 years ago

Dynamic creation/disposal of channels via connections via a network interface is exactly what I'm looking for to move auto_rx to a parallel decoding system :-)

flux242 commented 3 years ago

I could try the following design:

  1. client connects over tcp
  2. client sends tuner parameters (tuning freq, gain, sampling rate)
  3. client receives power measurements in a form say JSON strings with some interval
  4. client sends command to activate/deactivate specific channel (frequency) within the baseband
  5. client receives power measurements and decoder JSON strings at the same time

will this work for you?

flux242 commented 3 years ago

or client connects to the control block using TCP and receives JSONs over UDP broadcasts.

rs1729 commented 3 years ago

@darksidelemm I hope it could work this way. It is without a buffer or anything, there are too many buffers involved anyway. For realtime sdr data I tried timing the delay if there are too many clients, but if you don't have more clients than cores/cpu-threads, it should be ok. I could not test it on a raspi3 with 4 clients (plus occasional FFT scans) though. There is also the --ip <address> option for the client, if server/client are not on the same localhost on the local network.

@flux242 Maybe it is enough if the clients outputs the down-converted IQ-data, then auto_rx can choose the demodulator and decoder.

flux242 commented 3 years ago

@rs1729 well, auto_rx can of course handle clients itself. The power scans should not be stored in a file then but be sent over TCP the same way like IF is sent

rs1729 commented 3 years ago

If auto_rx runs the server, it could read the power scan file as it would when created by rtl_power (it is not the same format right now). For more than one sdr device the filenames should be different. Probably it could also be send over TCP. For auto_rx it would be easiest if server and client are on the same machine, auto_rx could evaluate the FFT, and for each interesting peak it would start a client at that frequency for detection/identification, and if there is a radiosonde signal, it decodes the IF stream.

You could separate server and client, but then more information about the scan results etc has to be communicated. Or all the scanning, detection and demodulation could be done in one program, and only the JSON output would be passed to the auto_rx parser.

EDIT: Ok, right, if the client calls "--scan" (or rather "--fft"), probably it is better that the client receives the FFT, so it can be processed or saved at the clients location.

flux242 commented 3 years ago

here is a new script version that uses iq_server/iq_client concept https://flux242.blogspot.com/2020/08/how-to-receive-and-decode-multiple_10.html

flux242 commented 3 years ago

here I've created a pastebin with the changes I've made to the auto-rx latest trunk to run it without any hardware in the UDP mode https://pastebin.com/CqWvNunc

basically the trick is to let the device_idx start with TCP in the station.cfg (no changes in the code required) Then I had to remove the utilities check ( as I'm only using auto_rx part and nothing else) Then I'm reading freq from telemetry Then udplistener listens on the default interface and not just localhost. This allows to get decoder strings from a different computer on the local network. And it would be nice to have UDP port configurable for the udplistener

darksidelemm commented 3 years ago

I'm really running low on time to work on these kind of changes.

If you can work the changes into a pull request (targeted at the testing branch) that doesn't result in degradation of functionality to other aspects of the software, i'd be happy to merge them in. The changes look fairly benign.

darksidelemm commented 3 years ago

Some more notes on the IQ server/client concept, from a recently email I sent:

So auto_rx needs two things:- It needs to be able to get a snapshot of spectrum of the entire band of interest. At the moment this is done using rtl_power, which handles things like stepping the receiver in steps to cover a frequency range wider than the receivers bandwidth, and it also does averaging. I get the output from this in CSV format and run it through a peak detection algorithm in Python. That peak detection alg works fine.

Next, I need to be able to get a narrow-band IQ channel (48 or 96 kHz sample rate & bandwidth, signed 16-bit samples). This is what drives pretty much all the decoder, and the dft_detect utility accept nowadays. The first step auto_rx does is to pipe the IQ into dft_detect, which does the signal classification. Once we know the sonde type, I then start up the decoder. All of this is done using python subprocesses, and piping signals around via stdout/stdin. Run auto_rx with the -v option and you'll see all the commands used. There are still a few decoders that require FM demodulated audio as input. At the moment i'm using rtl_fm's FM mode for this, but it's also fairly easy to take some IQ and pass it through CSDR to do FM demodulation - so realistically the source only needs to provide IQ.

So, the sources of data at the moment are rtl_power (FFT data over a wide bandwidth), and rtl_fm (FM audio and narrowband IQ). Each of these take up an entire RTLSDR while running, which is obviously a waste of resources. The ideal solution here is a SDR server of some kind, which allows multiple clients to connect in via a TCP socket and:- Request a snapshot of spectrum data (e.g. FFT output).- Request a stream of narrowband IQ at a specified frequency (within the receiver passband), and bandwidth.

The first requirement could be realistically just be done by requesting the full SDR bandwidth, and just taking a FFT, so I wouldn't say it's a hard requirement that the server does the FFTing.

This software here is essentially doing the server part of the above: https://github.com/dernasherbrezon/sdr-server .. however it's RTLSDR only. As you mention, something like SoapySDR would allow use of a wider range of SDRs.

So the above works for a single SDR - but we also have the issue that quite often the frequency ranges we are interested in exceed the bandwidth of a single SDR (e.g. a RTLSDR - max 2.4 MHz reliable bandwidth, but the radiosonde band in europe is 6 MHz wide). So what do we do here?  There are a few approaches I have thought of, which Ideally the server would handle.

The client part of this could be implemented as some binary which we can use in a similar way to rtl_power and rtl_fm right now - but instead of connecting to a SDR directly, it connects to a server which is started up separately. There are a few projects which get part of the way to what I'm looking for:

mhaberler commented 3 years ago

RTL-Airband can receive many AM voice channels from multiple rtlsdr's simultaneously, and FM in a limited way (runs out of cycles)

the fft is done on the GPU (at least for raspberries)

flux242 commented 3 years ago

Some more notes on the IQ server/client concept, from a recently email I sent:

So auto_rx needs two things:- It needs to be able to get a snapshot of spectrum of the entire band of interest. At the moment this is done using rtl_power, which handles things like stepping the receiver in steps to cover a frequency range wider than the receivers bandwidth, and it also does averaging. I get the output from this in CSV format and run it through a peak detection algorithm in Python. That peak detection alg works fine.

Next, I need to be able to get a narrow-band IQ channel (48 or 96 kHz sample rate & bandwidth, signed 16-bit samples). This is what drives pretty much all the decoder, and the dft_detect utility accept nowadays. The first step auto_rx does is to pipe the IQ into dft_detect, which does the signal classification. Once we know the sonde type, I then start up the decoder. All of this is done using python subprocesses, and piping signals around via stdout/stdin. Run auto_rx with the -v option and you'll see all the commands used. There are still a few decoders that require FM demodulated audio as input. At the moment i'm using rtl_fm's FM mode for this, but it's also fairly easy to take some IQ and pass it through CSDR to do FM demodulation - so realistically the source only needs to provide IQ.

So, the sources of data at the moment are rtl_power (FFT data over a wide bandwidth), and rtl_fm (FM audio and narrowband IQ). Each of these take up an entire RTLSDR while running, which is obviously a waste of resources. The ideal solution here is a SDR server of some kind, which allows multiple clients to connect in via a TCP socket and:- Request a snapshot of spectrum data (e.g. FFT output).- Request a stream of narrowband IQ at a specified frequency (within the receiver passband), and bandwidth.

The first requirement could be realistically just be done by requesting the full SDR bandwidth, and just taking a FFT, so I wouldn't say it's a hard requirement that the server does the FFTing.

This software here is essentially doing the server part of the above: https://github.com/dernasherbrezon/sdr-server .. however it's RTLSDR only. As you mention, something like SoapySDR would allow use of a wider range of SDRs.

So the above works for a single SDR - but we also have the issue that quite often the frequency ranges we are interested in exceed the bandwidth of a single SDR (e.g. a RTLSDR - max 2.4 MHz reliable bandwidth, but the radiosonde band in europe is 6 MHz wide). So what do we do here?  There are a few approaches I have thought of, which Ideally the server would handle.

  • If using a single SDR:

    • When a spectrum data snapshot is requested, hop the SDR as required to cover the requested range. This will necessarily result in any other clients connected having a brief drop in their IQ data streams, but I think we can live with this.
    • When the first narrowband-IQ client connects, pick some nominal SDR centre frequency to maximise available of the SDR in the rest of the desired frequency range. If a second client connects, the SDR centre frequency may need to be re-tuned. It may not be possible to re-tune the SDR far enough, and hence the server should return an error to the client.

  • If using multiple SDRs: 

    • Space the centre frequencies apart far enough such that we cover the band we are after.
    • When a client requests a spectrum data snapshot, make use of each SDR to avoid sample drops.
    • When a client connects and requests a narrowband-IQ stream, use the appropriate SDR for the frequency they request.

The client part of this could be implemented as some binary which we can use in a similar way to rtl_power and rtl_fm right now - but instead of connecting to a SDR directly, it connects to a server which is started up separately. There are a few projects which get part of the way to what I'm looking for:

  • The SDR# Spyserver does the server part well, and handles multiple clients with minimal CPU. However, it's closed source, and only supports a limited number of SDRs (Airspy and RTLSDR). I really don't want to make auto_rx depend on a closed source project!
  • The OpenWebRX project is kind of doing what we need, but its doing it by a master python server spawning subprocesses which consist of a bunch of CSDR instances piped together. This is certainly a possibly solution!
  • Zilog's iq_server looks to be getting pretty close to what is required too! Neither of the above really deal with the issue of being able to observe a frequency range wider than the SDR bandwidth.  Hopefully this gives you some idea of what I'm looking for...

The idea to retune hardware to be able to receive wider than the baseband can be implemented even using iq_server. There's a problem though- if 2 sondes are 2.5mhz apart and baseband is only 2.4mhz , to which one should it be tuned? Should there be a time window to jump from one to another? Is it really a viable solution ? How well does it scale? No, it isn't a solution. If you need wider tuning range than there are 2 options: use proper hardware that covers you frequency range or start multiple iq servers. The receivemultisonde.sh can be adjusted a bit to run in Multihardware mode.

rs1729 commented 3 years ago 
* Zilog's iq_server looks to be getting pretty close to what is required too!
  Neither of the above really deal with the issue of being able to observe a frequency range wider than the SDR bandwidth. 
  Hopefully this gives you some idea of what I'm looking for...

The iq_svcl server does not control the sdr, it just reads the data that is provided. If the center frequency of the input is changed, another band could be scanned. Though when decoding, the clients would have to change their frequencies as well. If the decoding is interrupted, it is normally not a big issue, only for RS41 it could be worth saving the calibration data. Anyway, I agree with @flux242 that after an initial wide scan to determine the best center frequency, the sdr should stick to that center frequency as long as there are (enough) signals in that band. 2 MHz of rtl-sdr bandwidth is still not bad, and if more is needed, I would prefer to add another rtl-sdr or use an airspy, before frequency hopping is considered.