search

pothosware / SoapySDR

Vendor and platform neutral SDR support library.
https://github.com/pothosware/SoapySDR/wiki
Boost Software License 1.0
1.12k stars 178 forks source link

A homebrewed SoapySDRUtil --rate syntax leads to serious crashes #233

Closed metssigadus closed 5 years ago

metssigadus commented 5 years ago

I am afraid I have discovered some serious problems related to SoapySDRUtil --rate syntax (or should I say Soapy in general...). My environment this time: OS = Ubuntu 18.04.2LTS; SDRPlay lib v 2.13.1, Soapy stock modules used to the max extent, except SDRPlay locally compiled from git sources.

P1 - In the rate test, Megabits per second are abbreviated as MBps not Mbps?

7.99748 Msps 31.9899 MBps 7.998 Msps 31.992 MBps

P2 - Describing the situation. Trying to test the limits on concurrent rate tests (productivity-wise or naming sceme wise). Currently two SDRs are connected to the host with summary bandwith of 10+10 Msps: SoapySDRUtil --find ######################################################

Soapy SDR -- the SDR abstraction library

######################################################

Found device 0 driver = remote label = SDRplay Dev0 RSP1A 180700DC93 remote = tcp://10.10.1.142:55132 remote:driver = sdrplay serial = 180700DC93

Found device 1 driver = remote label = SDRplay Dev1 RSP1A 180700D393 remote = tcp://10.10.1.142:55132 remote:driver = sdrplay serial = 180700D393

Found device 2 driver = sdrplay label = SDRplay Dev0 RSP1A 180700DC93 serial = 180700DC93

Found device 3 driver = sdrplay label = SDRplay Dev1 RSP1A 180700D393 serial = 180700D393

P3. Trying to investigate what is the proper syntax for (concurrent) rate tests. In separate windows, the following commands are given: SoapySDRUtil --rate=10e6 --direction=RX --args="driver=sdrplay,serial=180700D393" SoapySDRUtil --rate=10e6 --direction=RX --args="driver=sdrplay,serial=180700DC93" The first rate test starts normally, the second one stays on "0 Msps 0 MBps" forever The second device is in full working order - a single rate test against it works.

lsusb -t /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M | Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M | Port 1: Dev 3, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M | Port 1: Dev 3, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M | Port 2: Dev 6, If 0, Class=Vendor Specific Class, Driver=, 480M |__ Port 3: Dev 7, If 0, Class=Vendor Specific Class, Driver=, 480M

lsusb Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 007: ID 1df7:3000
Bus 001 Device 006: ID 1df7:3000
Bus 001 Device 003: ID 04cf:0022 Myson Century, Inc. OCZ Alchemy Series Elixir II Keyboard Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

dmesg [ 157.490743] usb 1-1.2: USB disconnect, device number 4 [ 157.714754] usb 1-1.2: new high-speed USB device number 6 using ehci-pci [ 157.823740] usb 1-1.2: New USB device found, idVendor=1df7, idProduct=3000 [ 157.823743] usb 1-1.2: New USB device strings: Mfr=0, Product=0, SerialNumber=1 [ 157.823746] usb 1-1.2: SerialNumber: 180700D393 [ 157.824477] usb 1-1.3: USB disconnect, device number 5 [ 158.046762] usb 1-1.3: new high-speed USB device number 7 using ehci-pci [ 158.155736] usb 1-1.3: New USB device found, idVendor=1df7, idProduct=3000 [ 158.155740] usb 1-1.3: New USB device strings: Mfr=0, Product=0, SerialNumber=1 [ 158.155742] usb 1-1.3: SerialNumber: 180700DC93 [ 284.862736] usb 1-1.3: usbfs: usb_submit_urb returned -28 [ 296.862760] usb 1-1.3: usbfs: usb_submit_urb returned -28

Q4 - Two assumptions - that the rate tester takes the first "free" device and that a busy device cannot be attached once more. Expecting the second rate tester will take the next free device. Using the commands (in separate windows): SoapySDRUtil --rate=10e6 --direction=RX SoapySDRUtil --rate=10e6 --direction=RX

the following happens:

Terminal A: SoapySDRUtil --rate=10e6 --direction=RX ######################################################

Soapy SDR -- the SDR abstraction library

######################################################

[INFO] SoapyRemote::setupRxStream(remoteFormat=CS16, localFormat=CS16, scaleFactor=32767, mtu=1500, window=44040192) [INFO] Client side stream bound to 10.10.1.142:54862 [INFO] Client side status bound to 10.10.1.142:35079 [INFO] Using format CS16.

[INFO] Configured sender endpoint: dgram=1452 bytes, 357 elements @ 4 bytes, window=43008 KiB [INFO] Client side stream connected to 10.10.1.142:41730 [INFO] Configured receiver endpoint: dgram=1452 bytes, 357 elements @ 4 bytes, window=43008 KiB Stream format: CS16 Num channels: 1 Element size: 4 bytes Begin RX rate test at 10 Msps Starting stream loop, press Ctrl+C to exit... 9.87491 Msps 39.4996 MBps 9.94211 Msps 39.7684 MBps /**[INFO] Using format CS16.

[INFO] Configured sender endpoint: dgram=1452 bytes, 357 elements @ 4 bytes, window=43008 KiB 9.96424 Msps 39.857 MBps 9.96747 Msps 39.8699 MBps** \^C[ERROR] SoapyLogAcceptor::handlerLoop() FAIL: SoapyRPCUnpacker::recv(header) FAIL: Error in rate test: SoapyRPCUnpacker::recv(header) FAIL: recv() [104: Connection reset by peer]

Terminal B SoapySDRUtil --rate=10e6 --direction=RX ######################################################

Soapy SDR -- the SDR abstraction library

######################################################

[INFO] SoapyRemote::setupRxStream(remoteFormat=CS16, localFormat=CS16, scaleFactor=32767, mtu=1500, window=44040192) [INFO] Client side stream bound to 10.10.1.142:33492 [INFO] Client side status bound to 10.10.1.142:37312 [INFO] Using format CS16.

[INFO] Configured sender endpoint: dgram=1452 bytes, 357 elements @ 4 bytes, window=43008 KiB [INFO] Client side stream connected to 10.10.1.142:60387 [INFO] Configured receiver endpoint: dgram=1452 bytes, 357 elements @ 4 bytes, window=43008 KiB Stream format: CS16 Num channels: 1 Element size: 4 bytes Begin RX rate test at 10 Msps Starting stream loop, press Ctrl+C to exit... ^C[ERROR] ~SoapyRemoteDevice() FAIL: SoapyRPCUnpacker::recv(header) FAIL: [ERROR] SoapyLogAcceptor::handlerLoop() FAIL: SoapyRPCUnpacker::recv(header) FAIL:

P5 - after a provident cold reboot, leave the host to which SDRs are connected and continue testing on a SoapyRemote connected host. SoapySDRUtil --find is correctly identifying both devices at the remote end. The commands (in separate windows): SoapySDRUtil --rate=10e6 --direction=RX --args="driver=remote,serial=180700D393" SoapySDRUtil --rate=10e6 --direction=RX --args="driver=remote,serial=180700DC93"

The result is rather similar to the previous testcase - the second ratetester listening on port 36266, is able to enforce the first stream to a port change. Both ratetesters start emitting weird S-letters.

Terminal A:

SoapySDRUtil --rate=10e6 --direction=RX --args="driver=remote,serial=180700D393" ######################################################

Soapy SDR -- the SDR abstraction library

######################################################

[INFO] SoapyRemote::setupRxStream(remoteFormat=CS16, localFormat=CS16, scaleFactor=32767, mtu=1500, window=44040192) [INFO] Client side stream bound to 10.10.1.145:38122 [INFO] Client side status bound to 10.10.1.145:49513 [INFO] Using format CS16.

[INFO] Configured sender endpoint: dgram=1452 bytes, 357 elements @ 4 bytes, window=43008 KiB [INFO] Client side stream connected to 10.10.1.142:38965 [INFO] Configured receiver endpoint: dgram=1452 bytes, 357 elements @ 4 bytes, window=43008 KiB Stream format: CS16 Num channels: 1 Element size: 4 bytes Begin RX rate test at 10 Msps Starting stream loop, press Ctrl+C to exit... 9.87187 Msps 39.4875 MBps 9.94127 Msps 39.7651 MBps |**[INFO] Using format CS16.

[INFO] Configured sender endpoint: dgram=1452 bytes, 357 elements @ 4 bytes, window=43008 KiB 9.96825 Msps 39.873 MBps 9.97372 Msps 39.8949 MBps** /SSSSSSSSSSSSSSS-SSSSSSSSS/SSSSSSSSSSSSSSSSSSS-SSSSSSSSSSSSSSSSSSSSSS\9.83665 Msps 39.3466 MBps -\SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS|SSSSSSSSSSSSSSSSSSSS/SSSSSSSSSSSSSSSSSSSSSS-SSSSSSSSSSSSSSSSSSSSSSS\9.68443 Msps 38.7377 MBps -\^CError in rate test: SoapyRPCUnpacker::recv() TIMEOUT: [ERROR] ~SoapyRemoteDevice() FAIL: SoapyRPCUnpacker::recv() TIMEOUT:

Terminal B SoapySDRUtil --rate=10e6 --direction=RX --args="driver=remote,serial=180700DC93" ######################################################

Soapy SDR -- the SDR abstraction library

######################################################

[INFO] SoapyRemote::setupRxStream(remoteFormat=CS16, localFormat=CS16, scaleFactor=32767, mtu=1500, window=44040192) [INFO] Client side stream bound to 10.10.1.145:60050 [INFO] Client side status bound to 10.10.1.145:41466 [INFO] Using format CS16.

[INFO] Configured sender endpoint: dgram=1452 bytes, 357 elements @ 4 bytes, window=43008 KiB [INFO] Client side stream connected to 10.10.1.142:36266 [INFO] Configured receiver endpoint: dgram=1452 bytes, 357 elements @ 4 bytes, window=43008 KiB Stream format: CS16 Num channels: 1 Element size: 4 bytes Begin RX rate test at 10 Msps Starting stream loop, press Ctrl+C to exit... 0 Msps 0 MBps |SSSSSSSSSSSSSSS/SSSSSSSSSSS0 Msps 0 MBps SSSSS/SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS-0 Msps 0 MBps -SS\SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS|SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS/SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS-SSSSSSSSSSSSSSSSSSSSSSSSS/^C

P6 - repeating the remote experiment without identifying the serial numbers, after a cold reboot on both machines (because Soapy functionality got affected by the previous test). The idea is that the first available device in the pool device will be connected. Commands are launched in separate terminals: SoapySDRUtil --rate=10e6 --direction=RX SoapySDRUtil --rate=10e6 --direction=RX The result as expected: port hijacking, S-chars, a duplicate item in inventory, Soapy layer rendered unusable (needs restart) A remark - before apt update/upgrade the OS, there were no S-chars but the messages known from #https://github.com/pothosware/SoapySDR/issues/232 : "WARNING: sync read failed. -1".

Terminal A: SoapySDRUtil --rate=10e6 --direction=RX ######################################################

Soapy SDR -- the SDR abstraction library

######################################################

[INFO] SoapyRemote::setupRxStream(remoteFormat=CS16, localFormat=CS16, scaleFactor=32767, mtu=1500, window=44040192) [INFO] Client side stream bound to 10.10.1.145:49062 [INFO] Client side status bound to 10.10.1.145:35842 [INFO] Using format CS16.

[INFO] Configured sender endpoint: dgram=1452 bytes, 357 elements @ 4 bytes, window=43008 KiB [INFO] Client side stream connected to 10.10.1.142:52269 [INFO] Configured receiver endpoint: dgram=1452 bytes, 357 elements @ 4 bytes, window=43008 KiB Stream format: CS16 Num channels: 1 Element size: 4 bytes Begin RX rate test at 10 Msps Starting stream loop, press Ctrl+C to exit... 9.87345 Msps 39.4938 MBps 9.94138 Msps 39.7655 MBps |[INFO] Using format CS16.

[INFO] Configured sender endpoint: dgram=1452 bytes, 357 elements @ 4 bytes, window=43008 KiB 9.96698 Msps 39.8679 MBps \SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS|SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS/SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS-SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS\SSSSSSSSSSSSSSSSSS|-^C ^C Error in rate test: SoapyRPCUnpacker::recv() TIMEOUT: select() [4: Interrupted system call] [ERROR] ~SoapyRemoteDevice() FAIL: SoapyRPCUnpacker::recv() TIMEOUT: select() [4: Interrupted system call]

Terminal B: SoapySDRUtil --rate=10e6 --direction=RX ######################################################

Soapy SDR -- the SDR abstraction library

######################################################

[INFO] SoapyRemote::setupRxStream(remoteFormat=CS16, localFormat=CS16, scaleFactor=32767, mtu=1500, window=44040192) [INFO] Client side stream bound to 10.10.1.145:33620 [INFO] Client side status bound to 10.10.1.145:43118 [INFO] Using format CS16.

[INFO] Configured sender endpoint: dgram=1452 bytes, 357 elements @ 4 bytes, window=43008 KiB [INFO] Client side stream connected to 10.10.1.142:38002 [INFO] Configured receiver endpoint: dgram=1452 bytes, 357 elements @ 4 bytes, window=43008 KiB Stream format: CS16 Num channels: 1 Element size: 4 bytes Begin RX rate test at 10 Msps Starting stream loop, press Ctrl+C to exit... /-S\SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS[..]SSSSSSSSSSSSSSSSS/SSSS[..]SS-SSSS[..]SSSSSSSSSSS\SSSS0 Msps 0 MBps ^C

Sorry but for this summer I have exhausted my free time to fight with Soapy and its dependances. If somebody with 25 yrs of Linux administration experience and an IT/infosec analyst experience is not able to get the fresh version of SDRPlay+Soapy working on a stock OS, then I imagine, mostly nobody is. Something is very wrong, the repeatability of the installation is close to nil. And this "something" has to do with the last year or so.

guruofquality commented 5 years ago

P1 - In the rate test, Megabits per second are abbreviated as MBps not Mbps?

correct. Big B is bytes, little b is bits.

P2 - Describing the situation. Trying to test the limits on concurrent rate tests (productivity-wise or naming sceme wise).

There is one very unique thing about SDRPlay (or at least its underlying driver). You cannot have more than one mirics lib open in a given process. Once SDRPlay, one process. That means one server, one device. Or if you arent trying to go through the server, only one device per application.

So the caveat usually is, then someone wants to use two SDR play in the same application. SoapyRemote is usually the only way. I think the recommendation is to spawn a two servers. So I think this is what you are running in to.

As far as the args, this can help:

metssigadus commented 5 years ago

Oh, nice, we are slowly approaching the root cause.

Well, let's try to re-phrase the current knowledge for these low-educated technicians from emerging countries whose English is not from Oxford ... rather slowly and using less ambiguous wording.

My own ugly problem is, I had last summer a PoC installation on my desk which was able concurrently serve three instances of OpenWebRX for for CB, 2m and 70cm bands respectively (the first one served by a RTL-SDR stick, two ham bands by SDRPlays). However the vendor lib version was 2.11.1 then. Just my HW was slow and needed some upgrade. If I had the knowledge I currently have, I should have made a backup. Which I did not, in a hope that OSS will compile at every weather. :(

guruofquality commented 5 years ago

CLAIM1: as of today, SoapySDRPlay module is able to support a single thread only - Y/N?

Yes, only one thread for the rx stream

CLAIM2: SoapySDRPlay as a module will only be needed at the transmitting end of a pair of SoapyRemotes - Y/N?

Yes

CLAIM3A: it takes two SoapyRemoteServer instances on different ports to transfer I/Q signals from two SDRPlay devices to the remotes - Y/N?

Right

(CLAIM3B: or still the SoapyRemoteServer can transmit signals of two SDRPlay devices on the condition these will not be consumed locally - Y/N?)

Not sure what consumed locally means. But you cant have the single SDRPlay device open in both the server process and open by some other SDR app on the machine to which the SDRPlay is plugged into.

CLAIM4: multiple instances of third party applications like e.g. rx_sdr can be launched at the remote end to consume SoapyRemote delivered streams, and the above-mentioned driver limit does not apply there - Y/N?

Right, the driver limit only applies to the machine to which the SDRPlays are attached.

FYI, this came up before, this was the discussion: https://github.com/pothosware/SoapySDRPlay/issues/46

Lets just take this over to https://github.com/pothosware/SoapySDRPlay/issues if there is nothing I can really "fix" on this project.

@SDRplay any comments?

metssigadus commented 5 years ago

What can be done: a: let @SDRplay create a clear and distinctively worded comment regarding the single-threadiness "feature". What Pothosware can do, make these authoritative comments visible as a big warning to everybody trying to make use of multiple SDRPlays under Pothosware; b: Make a clear note at a suitable place in Wiki that the Pothosware support level for SDRPlay is below the average level (no pre-compiled modules available for Linux distros); c: if possible, in SoapySDRUtil, implement some self-protecting checks against the commands surely leading to the crash and core dump (regarding the issue mentioned above).

Regarding two SoapyServers - I cannot comment yet, need time to test. If successful, me will produce a test case.

SDRplay commented 5 years ago

There can only be one instance of the API in any single user space and each API instance can only deal with one IQ stream. Just run multiple servers - I know of a number of people that run multiple websdr servers on the same hardware.

SDRplay commented 5 years ago

FYI whilst what I said is true about the single use nature of the API, you can create a multi instance application. Here's an example of what the university of Sheffield did in the UK with multiple RSP2s in Matlab... https://www.sdrplay.com/wp-content/uploads/2019/05/Time-and-phase-synchronization-with-two-RSP2s.pdf and here's the repo... https://github.com/VasAthanasios/MultiThread_SDRplayRSP2_Matlab_Toolbox

Hope that helps.

metssigadus commented 5 years ago

Thnx. It is much clearer now. Actually I should have been tested it before having an opinion in loud, but currently the end result looks like this for me: One radio host with:

One web host with:

The local network MUST be:

This blueprint looks doable (provided no other abyssal limitations met). Yes, @guruofquality , you are free to close the issue or move it to more appropriate modules.