Sampling problems with LimeSDR at sample rates higher than 3Mhz

[Dantali0n]

Whenever the sampling rate is increased above around 3Mhz it fails to sample the data. The output can only be recovered by stopping the device and restarting at lower sampling rate. My machine should be capable of handling these data rates having a 7700hq and a 1050 TI.

The outputs shows the following endlessly in a loop:

SDRThread::readStream(): 2. SoapySDR read failed with code: -1
SDRThread::readStream(): 2. SoapySDR read failed with code: -1
SDRThread::readStream(): 3.1 iqDataOutQueue output queue is full, discard processing of the batch...
SDRThread::readStream(): 2. SoapySDR read failed with code: -1

SoapySDR version: 0.7.1 LMS7Soapy version: 18.10.1 CubicSDR version: c561292

[vsonnier]

Hello @Dantali0n, was the problem present if you revert CubicSDR for instance to 792bf20a9c42e634031f4ef2ad6812f17cdf7b69 (a.k.a before my Spin-lock experiments)

Otherwise the -1 return code means SOAPY_SDR_TIMEOUT and is returned by the Soapy module itself, and the problem is likely to come from there.

[Dantali0n]

Hello @Dantali0n, was the problem present if you revert CubicSDR for instance to 792bf20 (a.k.a before my Spin-lock experiments)

Otherwise the -1 return code means SOAPY_SDR_TIMEOUT and is returned by the Soapy module itself, and the problem is likely to come from there.

No, I was waiting to see if the problem would persist after your spin-lock refactoring.

Regarding the Soapy module I can use almost any Soapy interfacing software such as sdrangel but also the python modules of Soapy at the full 65Mhz rate perfectly. Therefor, I think the problem resides within CubicSDR.

[vsonnier]

Right, I've added traces in master about the used MTU size usage. Can you test and report the ones you are getting depending of the sample rate ? Maybe the problem is a too-small reported getStreamMTU, resulting too much readStream on the Cubic side, since it uses MTU to know how many samples are fetched at a time.

Other clients may not obey the MTU size and request bigger chunks, resulting less overhead or problems.

[Dantali0n]

Anything between 100Khz and 3Mhz has the same MTU size of 4080 I also tried starting the device with 65Mhz just to see if the MTU size would be affected but it was still 4080.

sdrangel seems to have a very similar approach to getting mtu size expect it uses GetNativeStreamFormat instead of "CF32"

https://github.com/f4exb/sdrangel/blob/f520ad60724e94e6b392029b41c15f25f98be348/plugins/samplesource/soapysdrinput/soapysdrinputthread.cpp#L117

[vsonnier]

@Dantali0n Hum, 4080 is indeed awfully small. That leads to a big fat 785 readStream calls per second for 3Msps, not to speak of the higher samplerates.

Still, SDRAngel manages it so the problem is not there. The CF32 vs. Native looks also irrelevant, because basically the same method is used in both cases to fetch samples the only difference being CF32 has another format conversion step.

I think I found it: LimeSDR actually manages the readStream timeoutUs parameter, which was set as a default value of 100ms in Cubic as a C++ default argument.
By comparison SDRAngel adapts it and it is no less than 250ms according to the comments there.

Cubic however doesn't care of this parameter at all, so I changed the code to neutralize its effect entirely. Give it a try !

[Dantali0n]

Unfortunately, It did not help. The errors about timeouts of course disappear from the console but the waterfall drops flat at 3Mhz sample rate and a attempt to stop the device will freeze CubicSDR.

I think we are on the right track though, I managed to setup Clion and probably have time to have a play tomorrow. Surely if other programs such as SDRAngel can do it than we can also make it work with CubicSDR.

[vsonnier]

The errors about timeouts of course disappear ...

That is actually good news, and it was not that obvious. Good to hear though ! The freeze problem may come from the module using the CF32 path though, ex around here: https://github.com/myriadrf/LimeSuite/blob/e3e7203ccf1e52128950347d44eb4e11a0ac7088/src/protocols/Streamer.cpp#L93

[Dantali0n]

Got it! just deactivate and activate stream after adjusting samplerate and fixed.

[vsonnier]

Got it! just deactivate and activate stream after adjusting samplerate and fixed.

Well, that means the Lime module got broken when sample rate changes, which is definitely its fault. I guess some internal buffers were not properly resized/reset. We should submit an issue to https://github.com/myriadrf/LimeSuite/tree/master/SoapyLMS7 I think.

[Dantali0n]

I'll do that. and in the meantime I am writing a small fix I'll send the pull request soon.

Notably, when deactivating and activating device MTU also changes significantly. I have seen MTU sizes of 32640 appear in output of console.

[vsonnier]

How high sample rate you could get with your fix ?

[Dantali0n]

How high sample rate you could get with your fix ?

65Mhz but the performance is not great as the audio & waterfall stutter but it works nonetheless.

[Dantali0n]

@vsonnier Seems MyriadRF is aware of this behavior and does not think it is a bug https://github.com/myriadrf/LimeSuite/issues/177

Looking at the SoapySDR driver guide it never lists a requirement that calls such as setSampleRate are required to behave correctly when the stream is active. Perhaps we can ask @guruofquality if he thinks this behavior is indeed correct.

Driver guide for reference: https://github.com/pothosware/SoapySDR/wiki/DriverGuide

[vsonnier]

The way CubicSDR implements the LimeSDR workaround, it looks like a side-effect to properly reset buffers: 1) SoapySDR::Device::setSampleRate : User command 2) SoapySDR::Device::deactivateStream : Lime workaround 3) SoapySDR::Device::activateStream : Lime workaround 4) SoapySDR::Device::getStreamMTU : Re-do buffer setup on Cubic side

What I would rather have as a "safe" sample rate change is this for all devices if possible:

1) SoapySDR::Device::deactivateStream : be safe, make the device quiet before changes 2) SoapySDR::Device::setSampleRate : User command, changing sample rate 3) SoapySDR::Device::getStreamMTU : Re-do buffer setup on Cubic side 4) SoapySDR::Device::activateStream : back to business.

I don't know if the second solution would break other devices, though. I can test on SDRPlay RSP2 and PlutoSDR at home.

[Dantali0n]

I agree that your proposed safe change is better. I saw that deactivate / activate takes a long time (several seconds) but indeed we need to test this for other devices (I am in Switzerland until August and could only bring the LimeSDR). I do not think that having a few seconds delay when changing the sampling rate is a huge issues because I find it unlikely users will change the sample rate often.

If you have the proposed "safe" sample rate changes ready I will be happy to test it on the LimeSDR again.

[guruofquality]

Looking at the SoapySDR driver guide it never lists a requirement that calls such as setSampleRate are required to behave correctly when the stream is active. Perhaps we can ask @guruofquality if he thinks this behavior is indeed correct.

Well samples arent well defined while the rates are being changed, and certain things may need resetting after PLLs are re-tuned or filters flushed, hardware depending... So I don't think its a terrible idea to deactivate the stream. But I would really rather the soapysdr support module handled this case in whatever way was best so that re-tuning while streaming would be possible.

I think for limesdr, its too confusing to coordinate doing a live sample rate re-tune while streaming. The FPGA support isnt there, and if it was, the software effort might be tedious as well. So they might as well shut off streaming while setting the rate, which is basically what we are discussing. Actually SoapyLMS used to do this, I suppose it was removed to simplify the software.

I saw that deactivate / activate takes a long time (several seconds)

For LimeSDR, activateStream() will re-run some internal calibrations if its the first call, or if the filter bandwidth or frequency has been changed since the last call to activateStream(). I wonder if thats what you are seeing.

Maybe thats ok if its just due to the settings changes since limesdr needs to run internal loopback calibrations. But in general de/activateStream were supposed to be quick calls; where as setupStream() can be somewhat heavy-handed.

[vsonnier]

PR #718 is now available for public consumption, implementing the suggestions above.

[vsonnier]

PR #718 has been merged! Thanks you all for your feedback. Good to have LimeSDR working at full potential at last !