SoapyRemote Source Not Working

sally-yachts commented 2 years ago

Since learning that SoapyRemote is a native part of the SoapySDR stack, I've started trying to test using an Airspy receiver connected to an RPi 4, with the intention of streaming the data over the network to a separate machine running the latest Docker image for trunk-recorder. I have had success running a similar architecture using multiple NooElec SDR dongles via rtl_tcp; the goal here is basically trying to replicate that setup with a receiver that supports more bandwidth. SoapyRemote can also allow multiple clients to share the same SDR which would make it easy to compartmentalize systems using different instances of trunk-recorder.

Note that this config is confirmed working when the Airspy is locally connected using osmosdr and driver=airspy. Docker is running on Ubuntu Server 20.04, SoapyRemote is running on an RPi 4 pisdr image which i believe is built on Raspbian. Both sides are up to date on patches.

Current source block from config:

   "sources": [{
        "center": 854784375,
        "rate": 10000000,
        "gain": 10,
        "debugRecorders": 0,
        "digitalRecorders": 1,
        "modulation": "qpsk",
        "driver": "osmosdr",
        "device": "driver=remote,remote=1.1.1.1,remote:driver=airspy,soapy=0"
}],

Relevant logs:

[2021-07-17 21:16:03.994511] (info)   LNA Gain: 0,
[INFO] SoapyRemote::setupRxStream(remoteFormat=CS16, localFormat=CF32, scaleFactor=32767, mtu=1500, window=44040192),
[INFO] Using format CS16.,
SOURCES,
[2021-07-17 21:18:58.609652] (info)   Driver: osmosdr,
[2021-07-17 21:18:58.609657] (info)   Center: 854.784375 MHz,
[2021-07-17 21:18:58.609669] (info)   Rate: 10000000,
[2021-07-17 21:18:58.609680] (info)   Error: 0,
[2021-07-17 21:18:58.609687] (info)   PPM Error: 0,
[2021-07-17 21:18:58.609694] (info)   Auto gain control: false,
[2021-07-17 21:18:58.609698] (info)   Gain: 10,
[2021-07-17 21:18:58.609706] (info)   IF Gain: 0,
[2021-07-17 21:18:58.609712] (info)   BB Gain: 0,
[2021-07-17 21:18:58.609720] (info)   LNA Gain: 0,
[2021-07-17 21:18:58.609726] (info)   PGA Gain: 0,
[2021-07-17 21:18:58.609732] (info)   TIA Gain: 0,
[2021-07-17 21:18:58.609738] (info)   MIX Gain: 0,
[2021-07-17 21:18:58.609743] (info)   VGA1 Gain: 0,
[2021-07-17 21:18:58.609749] (info)   VGA2 Gain: 0,
[2021-07-17 21:18:58.609756] (info)   Idle Silence: 0,
[2021-07-17 21:18:58.609761] (info)   Digital Recorders: 1,
[2021-07-17 21:18:58.609767] (info)   Debug Recorder: false,
[2021-07-17 21:18:58.609774] (info)   SigMF Recorders: 0,
[2021-07-17 21:18:58.609778] (info)   Analog Recorders: 0,
[2021-07-17 21:18:58.609785] (info)   Source Device: driver=remote,remote=10.15.1.92,remote:driver=airspy,soapy=0,
gr-osmosdr 0.2.0.0 (0.2.0) gnuradio 3.8.1.0,
built-in source types: file osmosdr fcd rtl rtl_tcp uhd miri hackrf bladerf rfspace airspy airspyhf soapy redpitaya freesrp ,
[INFO] Client side stream bound to 2.2.2.2:43927,
[INFO] Client side status bound to 2.2.2.2:45715,
[INFO] Server side stream bound to 1.1.1.1:32923,
[INFO] Server side stream connected to 3.3.3.3:43927,
[INFO] Server side status connected to 3.3.3.3:45715,
[INFO] Configured sender endpoint: dgram=1452 bytes, 357 elements @ 4 bytes, window=43008 KiB,
[INFO] Client side stream connected to 1.1.1.1:32923,
[INFO] Configured receiver endpoint: dgram=1452 bytes, 357 elements @ 4 bytes, window=43008 KiB,
[WARNING] Set thread priority 0.5 failed: Operation not permitted,
[2021-07-17 21:18:58.710157] (info)   SOURCE TYPE OSMOSDR (osmosdr),
[2021-07-17 21:18:58.710197] (info)   Setting sample rate to: 10000000,
[2021-07-17 21:18:58.710929] (info)   Actual sample rate: 10000000,
[2021-07-17 21:18:58.710962] (info)   Tuning to 854.784375 MHz,
[2021-07-17 21:18:58.712146] (info)   Gain Stage: LNA supported values: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 ,
[2021-07-17 21:18:58.712412] (info)   Gain Stage: MIX supported values: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 ,
[2021-07-17 21:18:58.712671] (info)   Gain Stage: VGA supported values: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 ,
[2021-07-17 21:18:58.712692] (info)   Max Frequency: 859.734375 MHz,
[2021-07-17 21:18:58.713668] (info)   Gain set to: 10,
[2021-07-17 21:18:58.712711] (info)   Min Frequency: 849.834375 MHz,
[2021-07-17 21:18:58.714177] (info)   Auto gain control is OFF,
[2021-07-17 21:18:58.714210] (info)   Setting antenna to [RX]

SoapyRemote Logs:

######################################################
## Soapy Server -- Use any Soapy SDR remotely
######################################################

Server version: 0.6.0-gc09b2f10
Server UUID: [removed]
Launching the server... tcp://0.0.0.0:55132
Server bound to 0.0.0.0:55132
Launching discovery server... 
Connecting to DNS-SD daemon... 
[INFO] Avahi version:  avahi 0.7
[INFO] Avahi hostname: soapyremote-01
[INFO] Avahi domain:   local
[INFO] Avahi FQDN:     soapyremote.local
[INFO] avahi_entry_group_add_service(soapyremote._soapy._tcp)
Press Ctrl+C to stop the server
SoapyServerListener::accept(3.3.3.3:46350)
SoapyServerListener::accept(3.3.3.3:46352)
SoapyServerListener::accept(3.3.3.3:46354)
SoapyServerListener::close()
SoapyServerListener::close()
SoapyServerListener::accept(3.3.3.3:46356)
SoapyServerListener::handlerLoop() FAIL: SoapyRPCUnpacker::recv(header) FAIL: 
SoapyServerListener::handlerLoop() FAIL: SoapyRPCUnpacker::recv(header) FAIL: 
SoapyServerListener::close()
SoapyServerListener::close()

(the last four lines happen only when stopping the container, no errors are seen while running/connected)

Sanitized IP address definitions:

1.1.1.1 - SoapyRemote Server
2.2.2.2 - Docker container's internal IP address
3.3.3.3 - Docker server's actual IP address

Weirdly enough the port numbers reported by the server and the docker image don't seem to match, but they appear to be establishing a connection. I'm guessing this has to do with NATting being done by docker. If the connection wasn't successful then there would be a different error that would prevent startup of trunk-recorder that looks like this:

[ERROR] SoapyRemote::find(tcp://1.1.1.1:55132) -- transact FAIL: SoapyRPCUnpacker::recv() TIMEOUT

Observations:

Have not done a packet trace yet since the connection appears to be successful. It is on the list for possible next steps.
I notice the sample formats appear to be different between client and server. Trying to force the format using the remote:format=xxxx setting does not appear to have any effect.
SoapyRemote Wiki is available here
SoapyRemote sample format definitions are available here

leee commented 2 years ago

I don't have much to offer here, but maybe might be useful for factoring out various parts of the chain:

Do you have success when you run a rtl-sdr remote, a la "device": "driver=remote, remote 1.1.1.1, remote:driver=rtlsdr, soapy=0"?

sally-yachts commented 2 years ago

Do you have success when you run a rtl-sdr remote, a la "device": "driver=remote, remote 1.1.1.1, remote:driver=rtlsdr, soapy=0"?

That's a brilliant idea. Completely slipped my mind that I would be able to test in this way.

From a quick test using a single RTL-SDR it seems to be working just fine, so it looks like it might have something to do with the airspy driver/config. Let me see if I can verify something like gqrx via soapyremote works.

sally-yachts commented 2 years ago

Confirmed that I can connect to the Airspy remotely from another PC running GQRX with the same device string and it appears to work fine.

leee commented 2 years ago

Another thing I'm curious about: what sample rate were you running at on GQRX - what happens if you try the airspy at a pretty low sample rate to start for trunk-recorder?

If you're running the radio full tilt at 10 MSps times 12 bit depth/sample times 2 components, that's 240 Mb/s, excluding any overhead. Even with sufficient link state and bandwidth capacity, it's possible you're pushing past the limits of your switching capacity in terms of packets/second.

Of course, this is all moot if you confirm that you were running the radio remote with GQRX at the full monty.

sally-yachts commented 2 years ago

Yeah I've tried with both 2.5MSps as well as 10MSps, no dice either way with trunk-recorder. I was able to receive at both sample rates via the test GQRX box so it should be functional. Perhaps there's an issue setting the airspy gain remotely; I've tried both the regular "gain" setting as well as splitting it into the ifGain/lnaGain/mixGain settings.

I haven't used debug recorders before, would those possibly help or are they only active when recording a voice transmission? I've never used them before.

leee commented 2 years ago

To clarify: when you test a remote airspy configuration, trunk-recorder appears to run, but just isn't receiving anything? Or are there messages in the logs you've pasted that I've missed?

If I were thinking gain might be a factor, I might try the following test that assumes the following:

trunk-recorder is successfully connected to the remote airspy
but gain isn't set properly, and that's why whatever systems you've programmed in aren't being received and recorded

And test that hypothesis by configuring trunk-recorder for a conventional system, and either directly connect a RF siggen to the airspy, or just key up a radio right next to it, and see if you get any activity/recordings.

sally-yachts commented 2 years ago

To clarify: when you test a remote airspy configuration, trunk-recorder appears to run, but just isn't receiving anything?

That's correct. The full log output of a test session is in this pastebin - trunk-recorder will run but just keep hunting for a control channel.

test that hypothesis by configuring trunk-recorder for a conventional system, and either directly connect a RF siggen to the airspy, or just key up a radio right next to it, and see if you get any activity/recordings.

Thanks, I'll try and mock something up to see what happens.

robotastic commented 2 years ago

OK! @sally-yachts I think I know the problem... but not the solution. Are you using the Trunk-Recorder image from DockerHub? I noticed in the log that you provided that Soapy was listed but not Soapy-Remote:

[2021-07-17 21:18:58.609785] (info)   Source Device: driver=remote,remote=10.15.1.92,remote:driver=airspy,soapy=0,
gr-osmosdr 0.2.0.0 (0.2.0) gnuradio 3.8.1.0,
built-in source types: file osmosdr fcd rtl rtl_tcp uhd miri hackrf bladerf rfspace airspy airspyhf soapy redpitaya freesrp ,

That reminded me that I was having a lot of trouble getting the docker image to build because of Soapy-Remote and may have removed it: https://github.com/robotastic/trunk-recorder/issues/450#issuecomment-832369120

So - that could be part of the problem...

The fix could be finding a way to get soap-remote to play nicely with being in a docker container.

sally-yachts commented 2 years ago

I am indeed using the image from Docker Hub, but that makes things even more interesting considering it works if I use the exact same setup but change the config to use RTL_SDR dongles via SoapyRemote instead of the single Airspy. Very odd. I know the current examples for launching via Docker include passing the avahi socket and enabling privileged mode and I can't see where the SoapyRemote package is removed in the Dockerfile, was that package perhaps eventually reintroduced with the docs tweak to use the avahi/privileged mode workaround?

robotastic / trunk-recorder

SoapyRemote Source Not Working #495