f4exb / sdrangel

SDR Rx/Tx software for Airspy, Airspy HF+, BladeRF, HackRF, LimeSDR, PlutoSDR, RTL-SDR, SDRplay and FunCube
GNU General Public License v3.0
2.83k stars 430 forks source link

Spectrum enhancements #768

Closed srcejon closed 2 years ago

srcejon commented 3 years ago

Hi,

Here's a few ideas for enhancements to the spectrum (I may be able to do some, if needed, but thought they should be discussed first)

f4exb commented 3 years ago

Well in my opinion there is not much things to change. Spectrum display is working productively as it is and some functions you mention already exist at least in some form. However there are interesting ideas that could make their way differently:

What could be actually implemented in the spectrum display but a few questions remain:

srcejon commented 3 years ago

RE markers, I meant markers that automatically track the power level at the frequency the marker is placed at. The current markers are fixed at the level where you manually place them - so don't reflect changes in the signal. (Both manually placed and tracking markers are useful)

With regard to the usefulness of zooming, I would say extremely! Quite often I'll have a wide span (due to a high sample rate), but briefly want to zoom in to see a narrow region in a little more detail, because the signals aren't visible with a wide span. The problem with adjusting the decimation to do this, is that it affects the signal being passed to the demodulators, which may cause them to stop working. It's also a bit fiddly to do like that. This is less of a problem with the channel analyzer, but I really want to work with the main spectrum, as that is what has the largest screen area, and is the main focus of the GUI. (What's the point of the main spectrum if you keep on having to go in to the channel analyzer to do things? I'd rather just have that in the middle)

Yep, SNR calculation might only be valid for CW like signals, but that still is a useful thing to be able to see when fiddling with circuits, anntennas or settings to see the impact on a test signal. With regard to the calculation, one way to calculate the noise, is to remove the fundamental frequency (accounting for a bit of leakage), its harmonics and DC, then just average the remaining bins.

RE autoscale - without having thought much about it, I'd guess the range could just be set so that the minimum value in the the FFT bins is visible (range = max - min) - that's roughly where I set it to by hand +/ a few %. (Note I see autoscale occurring when a user presses a button - not continuously)

With regard to needing dBm in normal SDR work, I think it can be useful for transmit, to get a rough idea of what output power you'll get for a given gain setting (as it isn't obvious). Perhaps not all that useful for most on RX, but I'd find it useful :)

With regards to labels, it's not a case of implementing them because other software has them - but I think more that other s/w has them because they are useful. A couple of use cases: it's useful to easily see what modes should be used within subbands of the HAM bands and where the band limits are. Also, it's useful for labelling things like beacons, so you easily know what you are looking at. Some may be able to remember the frequencies in their head, but I have to continually look them up, so it's nice just to have them on screen. Yes, the span can currently be determined by looking for the corresponding sample rate, but if we added a zoom, that would no longer be true. I think it would probably be helpful for less experienced users if it were displayed in the same place consistently (similarly displaying units (even via tooltips) would be helpful - the units of the waterfall scale aren't obvious to me, for example). I appreciate you may prefer a lightweight UI, which is why I suggested this extra information would be optional.

f4exb commented 3 years ago

RE markers, I meant markers that automatically track the power level at the frequency the marker is placed at.

yes the usage is wide enough so that we may add this instrumentation

With regard to the usefulness of zooming, I would say extremely!...

Zooming is the poor man's choice. It does not improve resolution in any way. However with larger FFTs as the screen resolution becomes lower than the FFT resolution it may improve resolution image wise. The channels docking window can be undocked and be made larger or put to another screen if you have a multiple screen setup. You may also use the local sink / local input couple of plugins to move part of the bandwidth to another device set. Zooming is apparently easier but does come with complexity in the implementation. For example what do you do with the channel markers?

Yep, SNR calculation might only be valid for CW like signals, but that still is a useful thing to be able to see when fiddling with circuits, anntennas or settings to see the impact on a test signal. ...

This is a call for a proper VNA feature plugin isn't it?

RE autoscale - without having thought much about it, I'd guess the range could just be set so that the minimum value in the the FFT bins is visible (range = max - min) ...

Yes this should be a push button. However it will turn out that the min is really low and you end up with too much dynamic range that compresses the useful signals. To make it useful I would rather work something around the average.

With regard to needing dBm in normal SDR work, I think it can be useful for transmit, to get a rough idea of what output power you'll get for a given gain setting ...

Maybe but you would (should!) normally know your output power by other means. This is quite more complex than it seems. You will have to manage the correction curve by frequency and apply it to the spectrum.

With regards to labels, it's not a case of implementing them because other software has them ...

I still do not see the real usefulness of this apart from encumbering the display but if this is optional why not. This is a typical "Ham" goody and just a goody.

Re displaying units. The units are well known and it is not needed to encumber the scale with it. Also the frequency scale itself gives the information about the bandwidth so there is no need to put it on the display.

srcejon commented 3 years ago

Yeah, even with a 4k display, you can't quite have a 4k FFT at 1-1 pixel to bin mapping, because 4k displays aren't actually 4096 pixels. But sometimes, it's easier to use where it's several pixels to a bin. For example, it's quite tricky to place markers on the exact pixel on a 4k display. Much easier to do / see when you've zoomed in a bit, in my opinion, at least.

More importantly though, if you zero-pad the zoomed FFT, so it's at the same number of points, although you are not increasing the frequencies you can resolve, you should get a better/smoother looking spectrum, because of the effective interpolation.

With regard to markers, I would say they should always be drawn at the frequency / level they are placed at. So if you zoom in so that the frequency or level they are placed at isn't visible, then they would not be drawn or get clipped to the edge of the display. (Having said this, this doesn't appear to be how they currently work: E.g. If I move the reference level, the markers don't move with it.)

While a VNA-like feature might be fun, I think there's merit to having SNR available for SDR setups that are receive only (as this is just what many people have). Where it can be used, is by tuning to a nearby CW beacon. You can then fiddle with your antenna / preamps / ferrite beads etc and see if your modifications are actually improving the SNR. While you can try to eyeball it by looking at the spectrum, I think a numeric value is much more reliable.

RE autoscale - yes, if a single FFT bin is too low, an average of the N lowest bins or something like that could work. I'm sure with a little bit of experimentation, some heuristic can be found that will give reasonable results the majority of the time.

You may be right about the power levels. I will give it some more thought.

Hmm, what can I say - I like goodies :)

f4exb commented 3 years ago

Having said this, this doesn't appear to be how they currently work: E.g. If I move the reference level, the markers don't move with it.

Yes as implemented now these markers are purely graphical artifacts. It would require a more complex design to support more functional markers. This probably requires a redesign of the way the spectrum and its representation on screen is designed. This is an old design that hasn't fundamentally changed for ages. Such redesign may be required to support more advanced stuff like frequency zooming. Unlike the features in the two tickets above this requires more thinking to be put into it.

srcejon commented 3 years ago

One further idea for you to consider: allow the frame rate to be increased.

While 20 FPS is fine for continuous signals, high speed packets can be missed. This can be seen with the 802.15.4 modulator and Packet Mod. (Seems to worse on Windows - not sure why).

I did some quick experiments with changing the DSPEngine and MainCore timers to smaller intervals and it seemed to help (also the waterfall scrolling was a bit smoother). I have a fast PC, but there seemed to be no problem doing 4k FFT at > 60 FPS.

(The phosphor effect helps as well, of course - but I'm talking about when this is off).

f4exb commented 3 years ago

I will lift all limits like I just like I did with the websockets: with greater power comes greater responsibility.

The reasonable use is to have a moderate FPS with max "average mode" (it is not an average...) engaged thus transients as short as they could be will always produce a trace.

Lifting the FPS limitation is in fact very easy to do with just one line of code apart from removing the timer related code.

With a 16 us refresh rate (face refresh rate):

image

Thu Feb 11 19:10:26 2021                                                                                                                                                                                                                    
+-----------------------------------------------------------------------------+                                                                                                                                                             
| NVIDIA-SMI 450.102.04   Driver Version: 450.102.04   CUDA Version: 11.0     |                                                                                                                                                             
|-------------------------------+----------------------+----------------------+                                                                                                                                                             
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |                                                                                                                                                             
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |                                                                                                                                                             
|                               |                      |               MIG M. |                                                                                                                                                             
|===============================+======================+======================|                                                                                                                                                             
|   0  GeForce GTX 1070    Off  | 00000000:01:00.0 Off |                  N/A |                                                                                                                                                             
| N/A   66C    P2    70W /  N/A |   1247MiB /  8119MiB |     98%      Default |                                                                                                                                                             
|                               |                      |                  N/A |                                                                                                                                                             
+-------------------------------+----------------------+----------------------+                                                                                                                                                             

+-----------------------------------------------------------------------------+                                                                                                                                                             
| Processes:                                                                  |                                                                                                                                                             
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |                                                                                                                                                             
|        ID   ID                                                   Usage      |                                                                                                                                                             
|=============================================================================|                                                                                                                                                             
|    0   N/A  N/A      1760      G   /usr/lib/xorg/Xorg                379MiB |                                                                                                                                                             
|    0   N/A  N/A    443463      G   ...all/sdrangel/bin/sdrangel       24MiB |                                                                                                                                                             
|    0   N/A  N/A    818922      G   /usr/lib/qt5/bin/designer           2MiB |                                                                                                                                                             
|    0   N/A  N/A    847321      G   ...AAAAAAAAA= --shared-files      131MiB |                                                                                                                                                             
|    0   N/A  N/A   1806134      G   ...gAAAAAAAAA --shared-files      202MiB |                                                                                                                                                             
|    0   N/A  N/A   2111037      C   /opt/conda/bin/python             473MiB |                                                                                                                                                             
|    0   N/A  N/A   2448553      G   /usr/lib/firefox/firefox            1MiB |                                                                                                                                                             
|    0   N/A  N/A   2549802      G   ./sdrangel                         25MiB |                                                                                                                                                             
+-----------------------------------------------------------------------------+    

Reaching close to 100% on my GPU: YES!!!

This is the GPU utilization profile in Grafana during these tests: image

f4exb commented 3 years ago

While turning SDRangel into a GPU burner might be fun I expect a mildly positive acceptance so a FPS capping option should be kept.

srcejon commented 3 years ago

Another request : how about larger FFT sizes (8k, 16k, 32k.. etc)?

(I've been looking at Meteor scatter off the GRAVES radar and the larger FFT's in SpectrumLab (up to 512k) help to show extra detail and increase SNR due to the processing gain).

f4exb commented 3 years ago

It should be possible however there are a few things to consider:

I think the limitation is only in the GUI so in theory it should be possible to use higher values with the REST API. However as soon as you move things around in the GUI it will fall back to its limitations. There is a limitation in SpectrumVis as well.

f4exb commented 3 years ago

While 4k may be a bit short I do not see the point of having FFT sizes larger than 32k. Even a 8k screen will not natively fit anything larger than 8k (not taking into account things on the "sides"). Note that you could still zoom in and keep the details but this comes at the expense of undesirable side effects:

Let's keep in mind that you also get processing gain through decimation and in the particular case of Graves radar you do not need a bandwidth much larger than a few k or at most a few 10's of k. Practically 24k is enough to catch even the fastest objects and 3k is a luxury to just "see" meteor impacts on the atmosphere.

srcejon commented 3 years ago

Thanks - the difference between 16k and 32k is definitely visible when zoomed in.

srcejon commented 3 years ago

For radio astronomy (and other small signal measurements), it might be nice to be able to support calibration to flatten out the non-flat frequency response of some RF front-ends.

E.g.

Connect 50ohm terminator to LNA. Press calibration button in GUI. This starts a "fix" average measurement and then saves the result. Connect antenna to LNA. The calibrated average is then subtracted from future FFT outputs, which should hopefully flatten it out. Might want to add in a median value from the average to all FFT bins so it's displayed at roughly the same level.

N5BRG commented 3 years ago

I like the calibration idea. I also have Noise sources that can be used. The noise sources I have where calibrated at one time and I don’t think they have moved much. To use them properly one may have to build a power supply and possible pule/sync the bias while testing the noise level.

I also have a noise source obtained from Ebay that looks pretty flat on a spectrum analyzer. https://www.ebay.com/itm/142806862253?hash=item213ff3c9ad:g:PnIAAOSwO5xfT2qf This is one that is similar.

Bob N5BRG

On May 17, 2021, at 9:23 AM, srcejon @.**@.>> wrote:

Connect

srcejon commented 3 years ago

I thought it would be interesting to compare a noise source vs 50ohm terminator, in this case with a NooElec hydrogen-line LNA:

image

And immediately thought of another handy feature for the spectrum :) Most spectrum analyzers support multiple traces - so you can do a capture, and then switch to a new trace while seeing the old trace in a different colour, to allow you to easily compare the two (or more) spectra.

f4exb commented 3 years ago

And immediately thought of another handy feature for the spectrum :) Most spectrum analyzers support multiple traces - so you can do a capture, and then switch to a new trace while seeing the old trace in a different colour, to allow you to easily compare the two (or more) spectra.

The phosphor display sort of does that but in a more graphics art manner. The base computation behind is similar however.

f4exb commented 3 years ago

Re calibration: alternatively an equalizer could be implemented as a "specialized" filter plugin as described in #871

Edit/Note: not "alternatively"... Calibration and equalization are different matters. An equalizer can be implemented as a plugin as described above while with calibration you would record the correspondence between a unit-less level in dB and a power level in dBm at certain points in frequency generally over a larger range than the present spectrum width. Then one could use linear interpolation to yield the actual power in dBm at any frequency. Calibration could be attached to the spectrum while equalization would act as a filter between device sets as described in #871.

N5BRG commented 3 years ago

Jon, When you tested with the noise source (NC346B) how did you power the noise source?

I have a HP8970B Noise measurement instrument and an HP346B noise source. The 8970B powers the 346B and I think it is doing so by pulsing the noise source. Possible pulsing a diode to keep the junction temperature at a constant level. One day I need to hook a stop to this power line and see what it is actually doing.

Bob N5BRG

On Jun 1, 2021, at 8:03 AM, Edouard Griffiths @.**@.>> wrote:

Alternatively an equalizer could be implemented as a "specialized" filter plugin as described in #871https://github.com/f4exb/sdrangel/issues/871

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/f4exb/sdrangel/issues/768#issuecomment-852106307, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ATEQNBBQKNZAJ3I47CUYN5TTQTLAXANCNFSM4XHJMD5A.

srcejon commented 3 years ago

Yes, the power to the 346B will be pulsed on and off when doing a noise figure measurement, if it is using the Y-factor method, as that compares the difference in power when the noise source is on and off (This is what the listed Excess Noise Ratio specifies). See:

See: https://www.keysight.com/us/en/assets/7018-06829/application-notes/5952-3706.pdf

srcejon commented 3 years ago

Is there a way for an RX channel to be sent the spectrum rather than IQ samples?

f4exb commented 3 years ago

No this is not possible by design and I think is not even desirable. The very reason of (Rx) channels is to process IQ samples. However a feature can very well do anything else.

srcejon commented 3 years ago

Ok. Just thinking it could save some processing having to recompute it.

How would a feature get the spectrum? Via the websockets API or is there a better way?

f4exb commented 3 years ago

Websockets are for remote so for internal transfer the so called "data pipes" are there. This is not too much used yet so may need some tweaking and enhancements but the idea is there.

f4exb commented 3 years ago

This is called DataFifo in fact. It is used in the Demod Analyzer for example: https://github.com/f4exb/sdrangel/blob/master/plugins/feature/demodanalyzer/demodanalyzerworker.cpp

N5BRG commented 3 years ago

Ability to use the noise source in this way would be interesting. This would be working like an HP8970B which I have here but I have not used as much as I would like.

If you are looking for a way to compare the noise spectrum under two conditions it may be interesting to first have the ability to record the noise spectrum in memory of to a file.

Consider the possibility of also looking at noise levels on a preamp, filter, or other devices in front of the radio. After having the ability to look at noise level on the radio being used you would insert a preamp or other device and reconsider the noise level. One may also evaluate different devices in the front end of the radio. Also, different power supplies.

Other SDR applications capture the noise floor in a quiet portion (minimum radio signals) of the spectrum and then subtract this from the active data stream. Just another possible use of the spectrum if captured.

On Jun 7, 2021, at 7:53 AM, Edouard Griffiths @.**@.>> wrote:

This is called DataFifo in fact. It is used in the Demod Analyzer for example: https://github.com/f4exb/sdrangel/blob/master/plugins/feature/demodanalyzer/demodanalyzerworker.cpp

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/f4exb/sdrangel/issues/768#issuecomment-855899017, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ATEQNBFDXB6JZEFQSOR4TELTRS6NPANCNFSM4XHJMD5A.

f4exb commented 3 years ago

Other SDR applications capture the noise floor in a quiet portion (minimum radio signals) of the spectrum and then subtract this from the active data stream. Just another possible use of the spectrum if captured.

Which other SDR applications? I'd be curious. I see an interest of this only if you monitor signals over a long time in order to compensate for the slow internal noise fluctuations essentially for radio astronomy. But for radio astronomy other techniques like having two LNAs on the same silicon substrate with one pointing to the source of interest and the other off focus would probably yield more interesting results. This could be achievable by exploiting two receivers in parallel and combine their PSD in a feature plugin. Another (old) technique using a single receiver is to periodically switch the input between the source and a dummy load. This can possibly be also orchestrated in a feature plugin.

srcejon commented 3 years ago

I'm personally interested in it for radio astronomy usage.

I think an advantage of capturing the cold sky vs dummy load or noise source, is that you can potentially calibrate out some of the freq response of the antenna, not just from the LNA and SDR. Another often mentioned alternative is using some RF absorbent material in front of the feedhorn (assuming you're using a dish). The s/w would be the same regardless of which of these methods is chosen, of course.

Two LNAs or a Dicke Switch are interesting, but of course require extra h/w that I suspect most beginners don't have. Probably quite a few, like myself, just starting out with Nooelec H1 LNAs, or similar.

N5BRG commented 3 years ago

Take a look at this video of features on the Flex 6600M. Stating at about the 40 second mark he describes “One of the most powerful features that Flex has come out with”. He talks about it in terms of AGC but I think it sets a differential threshold for the noise floor based on a quite part of spectrum. Took me a while to find the place a I saw this.

https://www.flexradio.com/videos/flex-6600m-getting-started-with-some-basic-controls/

Not suggesting you should be chasing the features in these radios but they are interesting...

Here is another link of several videos you may find interesting. This includes a recent ~1 hour talk on the Elecraft K4 and the Flex 6600M. Also a couple of DXpedition talks and a talk on DXlab. DXlab is a open source software package Dave Bernstein AA6YQ has worked on since 1990.

https://ncdxc.org/idxc-2021/

I too am very interested in the Radio Astronomy applications of SDRAngel. I have a 5 meter dish I hope to have operational one day (70 now so I need to hurry up) and this may become a part of the data collection system. Noise is an issue here but RFI is a bigger problem especially with all the satellites in place and being launched. So if we have a way to subtract noise and known QRM sources this would be a good feature.

On Jun 7, 2021, at 11:04 AM, Edouard Griffiths @.**@.>> wrote:

Other SDR applications capture the noise floor in a quiet portion (minimum radio signals) of the spectrum and then subtract this from the active data stream. Just another possible use of the spectrum if captured.

Which other SDR applications? I'd be curious. I see an interest of this only if you monitor signals over a long time in order to compensate for the slow noise fluctuations essentially for radio astronomy. But for radio astronomy other techniques like having two LNAs on the same silicon substrate with one pointing to the source of interest and the other off focus would probably yield more interesting results. This could be achievable by exploiting two receivers in parallel and combine their PSD in a feature plugin. Another (old) technique using a single receiver is to periodically switch the input between a source and a dummy load. This can possibly be also orchestrated in a feature plugin.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/f4exb/sdrangel/issues/768#issuecomment-856066885, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ATEQNBABWUSPQ7F72JBQQZTTRTU2NANCNFSM4XHJMD5A.

f4exb commented 3 years ago

Re the first link on Flex-6600M this exactly what the AGC of the SSB demod does: https://github.com/f4exb/sdrangel/tree/master/plugins/channelrx/demodssb#11-volume-and-agc except that the threshold setting (button no 4) is manual. Originally I developed this because I found annoying to have a permanent background noise particularly when not looking for Dx and listening to conversations btw stations with large enough SNR. This is typically the case of narrowband QO-100 where signals are constant. If you try it you will see it works pretty much the same way as in the video. You have additional controls in particular a delay (button no 3) before the squelch closes preventing squelch chopping at the rhythm of the voice.

N5BRG commented 3 years ago

Thanks for your comment. I have re-read the documentation you noted and now probably better understand the function of these AGC controls in SDRAngel.

I'm thinking this is still not the ideal situation for applications Jon was talking about. The Flex radio solution probably would not work either.

In radio astronomy work you really are trying to get the signals which are buried in the noise. These AGC techniques block signals below some threshold. A Dicky approach is to compare the noise signals to a noise+weak signal (which is at the same level as the noise). So this is a different type of separation of signal and noise. I believe this is what a correlator does in a radio telescope.

The added issue of needing to subtract the QRM from signals is another layer of complexity for radio astronomy.

The AGC approach can work well for a Ham QSO.

On Jun 8, 2021, at 3:17 AM, Edouard Griffiths @.**@.>> wrote:

Re the first link on Flex-6600M this exactly what the AGC of the SSB demod does: https://github.com/f4exb/sdrangel/tree/master/plugins/channelrx/demodssb#11-volume-and-agc except that the threshold setting (button no 4) is manual. Originally I developed this because I found annoying to have a permanent background noise particularly when not looking for Dx and listening to conversations btw stations with large enough SNR. This is typically the case of narrowband QO-100 where signals are constant. If you try it you will see it works pretty much the same way as in the video. You have additional controls in particular a delay (button no 3) before the squelch closes preventing squelch chopping at the rhythm of the voice.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/f4exb/sdrangel/issues/768#issuecomment-856561814, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ATEQNBCUQKB5V7U4R46H2ULTRXGZRANCNFSM4XHJMD5A.

f4exb commented 3 years ago

In fact Radioastronomy is worse than picking a signal into noise it is dealing with a faint noise over a sea of noise. This is what makes it difficult and very different from radio communications. And therefore is also an interesting challenge. Thus indeed the methods are different. Because the noise you are trying to detect is only a fraction of the total noise of the system you want a way to estimate the noise due to the system only particularly if you observe over a certain amount of time and the system noise varies. Hence the various techniques I described earlier. If you observe a spectrum which can be interesting to figure out the speed of hydrogen clouds from the Doppler shift you have to consider also that your system noise may vary over a range of frequencies.

audriusb commented 2 years ago

Sorry if this was discussed already (too much to read) but can there be an option to scroll the zoommed in main spectrum sideways? An API would be even better. I have a small usb DJ Controller that acts as a midi controller and i pretty much coded all my buttons to control sdrangel. The central fader is used to scroll the view, but i do that by changing the center and all channels frequency that isnt very nice as it has mini delays between api calls making SSB demod noisy during slide. Its nothing fancy, but if anyone cares: https://github.com/audriusb/midiangel

f4exb commented 2 years ago

All implemented