search

pothosware / SoapyRemote

Use any Soapy SDR remotely
https://github.com/pothosware/SoapyRemote/wiki
Boost Software License 1.0
122 stars 22 forks source link

OSX / Raspberry Pi issues #9

Closed cjcliffe closed 9 years ago

cjcliffe commented 9 years ago

Just trying out SoapyRTLSDR and SoapyRemote on a Raspberry Pi here, few items to note:

pi@raspberrypi ~/SoapyRemote/build $ SoapySDRServer --bind
######################################################
## Soapy Server -- Use any Soapy SDR remotely
######################################################

upid://raspberrypi?pid=5198&hid=8323329
Launching the server... tcp://[::]:55132
Server socket bind FAIL:

Running it with a specific bind parameter seems to work:

pi@raspberrypi ~/SoapyRemote/build $ SoapySDRServer --bind="0.0.0.0:55132"
######################################################
## Soapy Server -- Use any Soapy SDR remotely
######################################################

upid://raspberrypi?pid=5202&hid=8323329
Launching the server... tcp://0.0.0.0:55132
Server bound to 0.0.0.0:55132
Press Ctrl+C to stop the server
guruofquality commented 9 years ago

I'm aware of the bind issue, probably its on platforms which are missing ipv6 support in the kernel. You can also use --bind="0.0.0.0" to get the default port. I think it needs to test and fallback to ipv4 automatically... seems more graceful that way.

Regarding the flag, I think it need a test and fallback as well. I dont want a newer gcc to bomb out either just in case they finally take -std-c++0x away.

cjcliffe commented 9 years ago

Aye, it looks IPV6 related for sure; not a huge issue in that regard. Having a fallback for c++0x would be a good idea but I'm guessing it'll be around for awhile to support the many projects that haven't updated yet.

On another note might as well keep this issue going with the following:

pi@raspberrypi ~/SoapyRemote/build $ SoapySDRServer --bind="0.0.0.0:55132"
######################################################
## Soapy Server -- Use any Soapy SDR remotely
######################################################

upid://raspberrypi?pid=5241&hid=8323329
Launching the server... tcp://0.0.0.0:55132
Server bound to 0.0.0.0:55132
Press Ctrl+C to stop the server
SoapyServerListener::accept(192.168.1.101:63177)
Segmentation fault

Seems to crash right away once I connect from desktop; I'll do some more debugging on the Pi to figure out what's going on.

guruofquality commented 9 years ago

Might be the same problem, not sure since he said the module wasn't all filled out: https://groups.google.com/forum/#!topic/pothos-users/w_ABDy-xOU0

But arm to/from amd64 should be OK in principal. So I guess a backtrace will be very telling.

guruofquality commented 9 years ago

Hopefully the first issues have been fixed now in master, fixed flags, strerror, and ipv6 bind

guruofquality commented 9 years ago

This may be the fix for the crash: https://github.com/pothosware/SoapyRemote/commit/2fbb9acf89b5202ed42c68134e8ebebcc0f7daa8

cjcliffe commented 9 years ago

Excellent, was just about to do some more testing on the Pi

cjcliffe commented 9 years ago

Just a typo found so far:

        set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std-c++0x")

should be

        set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++0x")
guruofquality commented 9 years ago

doh! and i copied that around quite a bit -- fixed now

cjcliffe commented 9 years ago

Another quick typo, looks like you didn't mean to return here, compiles when removed:

SoapyRemote/common/SoapyRPCSocket.cpp:214:12: error: cannot initialize return object of type 'const char *' with an rvalue of type 'int'
    return strerror_r(errno, buff, sizeof(buff));
           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

(when compiled on OSX)

guruofquality commented 9 years ago

Unfortunately there are two versions of strerror_r, and one of them returns a char pointer, which I have to return, the other doesn't. The ifdefs were supposed to be the way to figure it out: http://linux.die.net/man/3/strerror_r

It looks like (_POSIX_C_SOURCE >= 200112L || _XOPEN_SOURCE >= 600) && ! _GNU_SOURCE should become ((_POSIX_C_SOURCE >= 200112L || _XOPEN_SOURCE >= 600) && ! _GNU_SOURCE) || __APPLE__

Thats a mess :-)

cjcliffe commented 9 years ago

Seems to be working properly with SoapySDRUtil; probes, finds, makes remotely; but when I actually try to start a stream:

cjmacbook:build ccliffe$ SoapySDRServer --bind
######################################################
## Soapy Server -- Use any Soapy SDR remotely
######################################################

upid://cjmacbook.local?pid=23325&hid=0
Launching the server... tcp://[::]:55132
Server bound to [::]:55132
Press Ctrl+C to stop the server
SoapyServerListener::accept([::ffff:127.0.0.1]:53537)
SoapyServerListener::accept([::ffff:127.0.0.1]:53538)
Found Rafael Micro R820T tuner
SoapyServerListener::accept([::ffff:127.0.0.1]:53539)
SoapyServerListener::close()
SoapyServerListener::close()
SoapyServerListener::accept([::ffff:127.0.0.1]:53540)
Found Rafael Micro R820T tuner
[R82XX] PLL not locked!
Floating point exception: 8

This isn't specific to the Pi; that's output from my OSX console connecting to localhost.

On the CubicSDR side:

[DEBUG] Found RTL-SDR Device using device index parameter 'rtl' = 0
[DEBUG] RTL-SDR opening device 0
[DEBUG] RTL-SDR Using buffer length 16384
[DEBUG] RTL-SDR Using 6 buffers
[DEBUG] RTL-SDR direct sampling mode 0
[DEBUG] RTL-SDR I/Q swap: No
[DEBUG] RTL-SDR offset_tune mode: No
[DEBUG] RTL-SDR PPM: 0
[DEBUG] RTL-SDR Tuner type: Rafael Micro R820T
[DEBUG] Setting sample rate: 2400000
[DEBUG] Setting center freq: 106072873
[DEBUG] Setting RTL-SDR AGC: Automatic
[INFO] SoapyRemote::setupRxStream(remoteFormat=CF32, localFormat=CF32, scaleFactor=32768, mtu=1500, window=16384)
[INFO] Client side stream bound to 127.0.0.1:60019
[INFO] Client side status bound to 127.0.0.1:60019
[DEBUG] Generating RTL-SDR lookup tables
[INFO] Using format CF32.
[INFO] Server side stream bound to [::ffff:127.0.0.1]:63781
[INFO] Server side stream connected to [::ffff:127.0.0.1]:60019
[INFO] Server side status connected to [::ffff:127.0.0.1]:63736

Connect seems fine, but as soon as I call:

    SoapySDR::Stream *stream = device->setupStream(SOAPY_SDR_RX,"CF32");

Immediately throws a runtime error:

[ERROR] SoapyLogAcceptor::handlerLoop() 
libc++abi.dylib: terminating with uncaught exception of type std::runtime_error: SoapyRPCUnpacker::recv(header) FAIL: Undefined error: 0

Will do some debugging to try and narrow it down.

cjcliffe commented 9 years ago

Also ((_POSIX_C_SOURCE >= 200112L || _XOPEN_SOURCE >= 600) && ! _GNU_SOURCE) || __APPLE__ does the trick on OSX for the strerror_r issue.

cjcliffe commented 9 years ago

gdb reveals:

Found Rafael Micro R820T tuner
[R82XX] PLL not locked!

Program received signal EXC_ARITHMETIC, Arithmetic exception.
[Switching to process 23811 thread 0xe07]
0x0000000100030790 in SoapyStreamEndpoint::SoapyStreamEndpoint (this=0x1002012b0, streamSock=@0x100200214, statusSock=@0x100200218, isRecv=false, numChans=0, elemSize=8, mtu=1500, window=16384) at /Users/ccliffe/Documents/GitHub/SoapyRemote/common/SoapyStreamEndpoint.cpp:41
41      _buffSize(((_xferSize-HEADER_SIZE)/numChans)/elemSize),

looks like numChans is causing divide-by-zero?

guruofquality commented 9 years ago

I think i see...

1) The error is forwarded over the rpc and then not caught, the log acceptor should at least catch here 2) The channels vector is empty, which we support, but its also used to set numChan, so that needs to bumped to a minimum of 1

guruofquality commented 9 years ago

Well, i pushed some fixes, I hope that does the trick

cjcliffe commented 9 years ago

Just pulled the update; seems to be working now -- unfortunately the number of elements returned per call is a consistent 178; far lower than the 40448 I am requesting which causes problems on the CubicSDR side -- should I be doing some additional buffering on readStream input when it's a remote connection or is there a buffer missing in SoapyRemote module on the client side?

[DEBUG] Setting RTL-SDR AGC: Automatic
[INFO] SoapyRemote::setupRxStream(remoteFormat=CF32, localFormat=CF32, scaleFactor=32768, mtu=1500, window=16384)
[INFO] Client side stream bound to 192.168.1.101:55699
[INFO] Client side status bound to 192.168.1.101:55699
[DEBUG] Generating RTL-SDR lookup tables
Loaded font 'Bitstream Vera Sans Mono' from 'vera_sans_mono12.fnt', parsed 167 characters.
[INFO] Using format CF32.
[INFO] Server side stream bound to 192.168.1.107:38872
[INFO] Server side stream connected to 192.168.1.101:55699
[INFO] Server side status connected to 192.168.1.101:61467
[WARNING] StreamEndpoint resize socket buffer: set 44040192 bytes, got 163840 bytes
[INFO] Configured sender endpoint: dgram=1452 bytes, 178 elements @ 8 bytes, window=160 KiB
[INFO] Client side stream connected to 192.168.1.107:38872
[INFO] Configured receiver endpoint: dgram=1452 bytes, 178 elements @ 8 bytes, window=16 KiB
calculated optimal element count of 40448
guruofquality commented 9 years ago

So, you are seeing fragmentation for the ethernet MTU. The SoapyRemote client side read/writeStream doesnt attempt to loop and continue on the same buffer, it just returns after the first datagram. I can make the read/writeStream implement the loop, but its worth noting that not all modules have this loop and repeat until the buffer is exhausted (or timeout) behaviour. Although convenient, if you want to be robust against lazy implementations, you might need CubicSDR to have a loop anyway. Thoughts?

FYI, if you set the "remote:format=CS16" stream arg, then we will get shorts over the ethernet and get double the number of samples per packet. SoapyRemote can convert CS16 to CF32 on the client side. So it might also be worthwhile handling CS8 conversions and CS8 as an RTL output.

cjcliffe commented 9 years ago

That works; I just wasn't sure if readStream on the client side was handling that buffer loop or I should expect to -- I think it's perfectly reasonable to do this on the CubicSDR side and I will make the adjustments.

I'll add the CS8 format to SoapyRTLSDR too; that makes sense to have available for this.

cjcliffe commented 9 years ago

SoapySDRServer is working now on localhost with OSX and remotely on Raspberry Pi. CS8 is also now implemented for SoapyRTLSDR, but the remote driver doesn't seem to acknowledge the "remote:format=CS8" as I still see:

[INFO] SoapyRemote::setupRxStream(remoteFormat=CF32, localFormat=CF32, scaleFactor=32768, mtu=1500, window=16384)

I also seem to be dropping a lot of packets and the audio demodulation is stuttering regularly as a result -- even with localhost at 250KHz bandwidth it's skipping; Pi is much much worse but Pi is wired to the network at 100Mbit and I'm running on 802.11a Wifi so at least I'd expect some loss there.

guruofquality commented 9 years ago

So I was working on the CS8 as we speak. I just checked it in. Needs testing though.

There is something regarding the socket buffering on OSX that needs to be figured out. From my experience, increasing the SO_RCVBUF beyond a few kb causes a crash. The current window default for OSX is actually way smaller (window=16384 in the snippet above). For linux, I set this to 10s of megabytes.

Can you mess around with this? either pass in "remote:window=something_big" to the stream args or change the default in SoapyRemoteDefs.hpp I think the bigger buffer is critical, just didnt want to crash OSX as the default option.

cjcliffe commented 9 years ago

Hmm, I tried the CS16 remote format and it seems to be returning I/I or Q/Q instead of I/Q so it's mirrored on the spectrum.

CS8 seems to be working with RTLSDR once I fixed the conversion; but both CS16 and CS8 remote format seem to make the stuttering worse (constant).

I've taken the window size as high as 4MB so far to localhost and there doesn't seem to be any improvement.

cjcliffe commented 9 years ago

Hmm, actually I don't think the window is working:

[1m[ERROR] StreamEndpoint resize socket buffer FAIL: No buffer space available
[INFO] Configured sender endpoint: dgram=1452 bytes, 714 elements @ 2 bytes, window=4000 KiB

~2MB seems to be about the limit for me:

[INFO] Configured sender endpoint: dgram=1452 bytes, 178 elements @ 8 bytes, window=2000 KiB
[INFO] Client side stream connected to 127.0.0.1:59736
[INFO] Configured receiver endpoint: dgram=1452 bytes, 178 elements @ 8 bytes, window=2000 KiB
calculated optimal element count of 40448

No change in performance though; and CS16 is still funky and CS8 stutters like crazy.

guruofquality commented 9 years ago

I just found a ridiculous typo that explains the conversion issues... how does it look now? (I tested cs16 w/ the bladerf, so I don't even)

cjcliffe commented 9 years ago

CS8 seems to be working slightly better and the stuttering is now on-par with CF32; CS16 still seems to be mirrored and duplicating I/I or Q/Q though.

guruofquality commented 9 years ago

More typos from when I added the CS8, works over here now...

Is there an OSX sysctl equivalent setting to get the buffer larger than 2M?

I'm curious about the role the flow control is playing in the stutters. We might need a different strategy when the buffer is small. This will disable waiting on flow control (ACK) packets.

diff --git a/common/SoapyStreamEndpoint.cpp b/common/SoapyStreamEndpoint.cpp
index c3bfa78..f26c320 100644
--- a/common/SoapyStreamEndpoint.cpp
+++ b/common/SoapyStreamEndpoint.cpp
@@ -252,6 +252,7 @@ void SoapyStreamEndpoint::releaseRecv(const size_t handle)
  **********************************************************************/
 bool SoapyStreamEndpoint::waitSend(const long timeoutUs)
 {
+    return true;
     //are we within the allowed number of sequences in flight?
     while (not _receiveInitial or
         uint32_t(_lastRecvSequence+_maxInFlightSeqs) < uint32_t(_lastSendSequence))

Just for some rough maths here, if there is a 2MB window, then roughly 1 million samples (at CS8) can be held in that window. At 1 Msps (sample rate), the remote endpoint has about 1 second before it will block on flow control. And it should see a flow control update every eight of a window. It seems like plenty of time and that 2MB should not be an issue for lower speeds.

cjcliffe commented 9 years ago

CS16 is fixed; seems a little bit smoother with the patch but still stuttering -- though it improves slightly when moving from CF32 to CS16 and from CS16 to CS8, but stutter seems to be of similar consistency whether it's running at 250KHz or 2.56Mhz -- the stuttering increases with rate (i.e. 2.048Mhz seems to have twice as much stuttering as 1.024Mhz)

Window seems to have a larger margin of improvement now, default window vs. 2MB window is much better; but still stuttering quickly such that it sounds "grainy"

cjcliffe commented 9 years ago

Doubt it's related; but I do get this warning several times from a fresh compile:

In file included from /Users/ccliffe/Documents/GitHub/SoapyRemote/common/SoapyURLUtils.cpp:4:
/Users/ccliffe/Documents/GitHub/SoapyRemote/build/common/SoapySocketDefs.hpp:71:13: warning: 'htonll' macro redefined [-Wmacro-redefined]
    #define htonll(x) __builtin_bswap64(x)
            ^
/usr/include/sys/_endian.h:141:9: note: previous definition is here
#define htonll(x)       __DARWIN_OSSwapInt64(x)
        ^
guruofquality commented 9 years ago

Oh, thats nice. I didn't get htonll on my platform. I fixed up the ifdefs, should be ok now.

guruofquality commented 9 years ago

Recap

To recap, you are streaming from the Pi to the OSX around 2 Msps. Increasing the socket buffering helps, increasing the samples per packet helps. But you are still seeing some kind of overflow or loss through the link.

Some thoughts

The OSX side is probably keeping up just fine pulling samples out of the socket. So its probably an issue of the Pi keeping up on the send() side. I guess we are seeing overflow, but the RTL doesn't have an indicator when this happens. Some things worth investigating here

Replicating the issue

I don't have exactly the same hardware, but I think I will give RTL on a Novena to my linux desktop a shot and mess around with the window sizes and some of the ideas mentioned here to see what happens.

Diff for printing S when there is a packet drop or overflow in socket buffer -- packet loss in network layer

diff --git a/common/SoapyStreamEndpoint.cpp b/common/SoapyStreamEndpoint.cpp
index c3bfa78..5e3ab59 100644
--- a/common/SoapyStreamEndpoint.cpp
+++ b/common/SoapyStreamEndpoint.cpp
@@ -208,6 +208,10 @@ int SoapyStreamEndpoint::acquireRecv(size_t &handle, const void **buffs, int &fl
     const int numElemsOrErr = int(ntohl(header->elems));

     //TODO sequence out of order or skip failures
+    if (uint32_t(_lastRecvSequence+1) != uint32_t(ntohl(header->sequence)))
+    {
+        SoapySDR::log(SOAPY_SDR_SSI, "S");
+    }

     //update flow control
     _lastRecvSequence = ntohl(header->sequence);

Maybe we are overflowing on the OSX side and the socket buffer has unaccounted for overhead:

diff --git a/common/SoapyStreamEndpoint.cpp b/common/SoapyStreamEndpoint.cpp
index c3bfa78..1efbf46 100644
--- a/common/SoapyStreamEndpoint.cpp
+++ b/common/SoapyStreamEndpoint.cpp
@@ -100,6 +100,7 @@ SoapyStreamEndpoint::SoapyStreamEndpoint(
     {
         //calculate maximum in-flight sequences allowed
         _maxInFlightSeqs = actualWindow/mtu;
+        _maxInFlightSeqs /= 2;

         //calculate the flow control ACK conditions
         _triggerAckWindow = _maxInFlightSeqs/_numBuffs;
cjcliffe commented 9 years ago

Ok, I'll give these a go -- note that most of my testing since the arithmetic error fix has actually just been to localhost (running SoapySDRServer and CubicSDR on my macbook) since the Pi introduced it's own issues testing over Wifi.

The MTU on lo0 I've been able to push up to 65535; but no actual network device seems to let me go over 1500 on the OSX side. And pushing lo0 to 65535 doesn't seem to affect the 178 element limit connecting to localhost anyways.

Will get back to you soon with results.

guruofquality commented 9 years ago

I should add that for the mtu stuff, you need to set both "remote:mtu=4096" as well as ifconfig lo mtu 4096 -- for example.

cjcliffe commented 9 years ago

Ok, so current status:

First patch: Printing 'S' only really reveals it showing an 'S' when I move the center frequency or do other things that actually interrupt the stream.

Second patch: Overflow patch seems to have no effect.

But Removing all the patches and going back to master I have managed to get it reasonably stable using:

   streamArgs["remote:mtu"] = "8192";
   streamArgs["remote:format"] = "CS8";
   streamArgs["remote:window"] = "16384000";

remote:mtu at 8192 seems to be the maximum; going beyond this it just connects successfully and then no data is sent at all. It looks like remote:window might be capable of going higher though.

Though it complains in the log about "No buffer space available"; cranking the mtu and the window to these levels seems to actually work and changes the endpoint init values in the log.

I had been stopping when I hit the error previously since I assumed they weren't being applied but it appears they are; and they work:

[INFO] SoapyRemote::setupRxStream(remoteFormat=CS8, localFormat=CF32, scaleFactor=128, mtu=8192, window=16384000)
[INFO] Client side stream bound to 127.0.0.1:56215
[INFO] Client side status bound to 127.0.0.1:56215
[INFO] Using format CS8.
[INFO] Server side stream bound to [::ffff:127.0.0.1]:59978
[INFO] Server side stream connected to [::ffff:127.0.0.1]:56215
[INFO] Server side status connected to [::ffff:127.0.0.1]:50924
[ERROR] StreamEndpoint resize socket buffer FAIL: No buffer space available
[INFO] Configured sender endpoint: dgram=8144 bytes, 4060 elements @ 2 bytes, window=16000 KiB
[INFO] Client side stream connected to 127.0.0.1:59978
[ERROR] StreamEndpoint resize socket buffer FAIL: No buffer space available
[INFO] Configured receiver endpoint: dgram=8144 bytes, 4060 elements @ 2 bytes, window=16000 KiB
calculated optimal element count of 40448

Perhaps OSX just doesn't let you tweak the socket buffer manually and is handling buffering behind the scenes in a dynamic way?

At these settings I can also now use SoapyRemote with localhost at up to 3.2Msps somewhat stable -- which is strange since it stutters at 3.2M using just the SoapyRTLSDR module directly and has never really worked properly on my MacBook at that rate..

I do still seem to notice the odd glitch but they're several seconds apart at minimum and sound more like just a skipped sample instead of a constant grain like before. CF32 format is almost usable as well but has more frequent stuttering.

Edit: Tried again with the Raspberry Pi, with the above streamArgs I can almost use 250khz over the hard-wired network but any higher it gets choppy. remote:mtu seems to be Ok with 65535 and I've had remote:window up to 32MB with the Pi remotely, neither settings improves much upon the 8192/16384000.

guruofquality commented 9 years ago

I gave my FM demod demo a shot, and the SoapyRTLSDR is very choppy and causing audio underflows compared to the RTL support from SoapyOsmo. I'm going to take a closer look, but I think there is an issue in SoapyRTLSDR readStream and buffer length somewhere. Just wanted to report this first...

guruofquality commented 9 years ago

I switched to the RTL async API on this branch, and my issue goes away: https://github.com/pothosware/SoapyRTLSDR/tree/async_stream

Just curious if it also fixes any of the stuttering mentioned here.

cjcliffe commented 9 years ago

Yeah, the async version works 100% locally with SoapyRemote and any stream options I throw at it now; I'm going to update the Pi and see how it goes.

I've been using the sync api for quite some time without issue so I'm surprised it's the apparent cause; perhaps because we were requesting buffer sizes that weren't in multiples of 512 or whatever the requirement is during SoapyRemote streaming?

cjcliffe commented 9 years ago

OK, nice work; Pi is awesome now!

I can use it stable at about 1Mhz at default settings; but with the stream args of:

   streamArgs["remote:mtu"] = "8192";
   streamArgs["remote:format"] = "CS8";
   streamArgs["remote:window"] = "16384000";

3.2Mhz now appears to be quite stable! And that's over Wifi :)

Edit: yeah this is much better than rtl_tcp :+1: Edit2: been running it stable for an 1/2 hour now on the Pi, don't recall hearing a single glitch so far

guruofquality commented 9 years ago

The sync api was always using bulk transfers of 16384*16. I did try smaller sizes like 16384 as well. Actually, I wasnt even using SoapyRemote when I mentioned this issue. So it really may be a sync api bug in rtl. I guess we can close this one out at least.