Track noise estimates as data

[rct]

The feat-noise branch was merged July 15, which prints periodic noise data to STDERR as a text string.

I think it would be useful to instead output this data the same way as the other decoded data and stats - using the data_make mechanisms so if -F JSON was selected, the noise data should be output as easily ingested data for graphing and tracking over longer periods.

The use case I have in mind is trying to identify sources of periodic interference that inhibit reception.

Right now there are two types of data output: 1) message decodes 2) stat data. Unless the noise data was merged into the stat data reports, it is logically a third message type. It might warrant introducing an additional message type field to make it easier for consumers to identify the message types.

[zuckschwerdt]

Certainly useful. We have a the "msg" key (but check there is no "model" key) to print messages, like here: https://github.com/merbanan/rtl_433/blob/master/src/decoder_util.c#L123-L126 But we might want structured data here.

It would also be nice if we could come up with a heuristic to track the levels and let the user known at least "channel earily quiet. check antenna" and "channel too crowded. check for interference."

[rct]

For the last 1-2 years, I've been aggregating/averaging the noise using what -M level adds to decodes.

It doesn't really change unless something drastic changes, so it hasn't been as useful as some of the other stats, like counting the number of unique devices heard each minute.

Hopefully the new noise floor estimates will give me more clues when interference crops up.

[zuckschwerdt]

Great to see this in action! Are those -11.5dB really average noise levels? If this is automatic gain (on a R820T) I would expect noise of -20 to as low as -30?

[rct]

Are those -11.5dB really average noise levels? If this is automatic gain (on a R820T) I would expect noise of -20 to as low as -30?

I haven't had a chance to integrate the new noise data yet. The old graph is showing the average noise as captured from -M level. That rrdtool archive is putting it into 5 minute buckets. (Should have been 1 minute buckets.)

On another instance with a somewhat better (RTL-SDR.com v3) receiver, I see a range of around -10 to -15.

[rct]

The noise estimates from -M noise:60 are close to what I'm seeing from -M level.

Last 30 minutes, from stderr:

Current level -13.5 dB, estimated noise -13.4 dB
Current level -13.6 dB, estimated noise -13.3 dB
Current level -11.3 dB, estimated noise -13.2 dB
Current level -13.6 dB, estimated noise -13.4 dB
Current level -13.0 dB, estimated noise -13.1 dB
Current level -13.6 dB, estimated noise -13.3 dB
Current level -13.7 dB, estimated noise -13.4 dB
Current level -10.5 dB, estimated noise -12.9 dB
Current level -13.5 dB, estimated noise -13.4 dB
Current level -13.3 dB, estimated noise -13.3 dB
Current level -13.4 dB, estimated noise -12.9 dB
Current level -12.9 dB, estimated noise -12.8 dB
Current level -12.7 dB, estimated noise -13.4 dB
Current level -13.5 dB, estimated noise -12.7 dB
Current level -13.6 dB, estimated noise -13.5 dB
Current level -8.4 dB, estimated noise -13.4 dB
Current level -13.6 dB, estimated noise -13.1 dB
Current level -7.1 dB, estimated noise -13.3 dB
Current level -13.5 dB, estimated noise -13.4 dB
Current level -14.1 dB, estimated noise -13.5 dB
Current level -9.6 dB, estimated noise -13.3 dB
Current level -13.1 dB, estimated noise -13.6 dB
Current level -12.8 dB, estimated noise -13.5 dB
Current level -13.9 dB, estimated noise -13.7 dB
Current level -7.4 dB, estimated noise -13.7 dB
Current level -13.8 dB, estimated noise -13.4 dB
Current level -13.9 dB, estimated noise -13.3 dB
Current level -13.2 dB, estimated noise -13.8 dB
Current level -7.3 dB, estimated noise -13.5 dB
Current level -13.9 dB, estimated noise -13.7 dB

-M level noise, 5 minute buckets, last 18 hours

I haven't added the new options that were suggested from the issue related to librtlsdr yet. Should be easier to do some quick experiments to get the noise floor down with -M noise:10

[zuckschwerdt]

I'm afraid the -Y autolevel won't help here, the default detection level is -12dB and the autolevel goes 3 dB above observed noise, which would be around -10dB :/ That's why I got suspicious about the noise figure. Over the whole range of receivers and locations I get -25 to -35 dB noise.

[rct]

Interesting, Thanks. With two different RTL-SDR.com v3's, if I remove the antenna, I'm seeing a noise floor around -32 - -33 dB. With the antenna back on I'm seeing around -12 to -16 dB. Both on running on RPi's with 6 ft USB extension cables. One antenna is a 1/2 wave dipole, the other is a 2m/440mhz 1/4 wave rubber duck attached directly to the SMA connector. I've got some experiments to do.

Generally my rtl_433 reception is pretty good, but there are times when I loose some devices, like the Acurite weather station.

I should note that rtl_433 does a better job than Acurite's purpose build Access Hub.

[rct]

I noticed something somewhat unexpected. When I fire up rtl_433 with -M noise (no gain settings or new -Y settings) the noise estimate is reported to be around -15 - -16 dB.

While running, if I disconnect the antenna for a minute and then reconnect it, the noise estimate drops to around -33 dB and seems to stay there after reconnecting the antenna. The first time I saw this, I thought I had stumbled upon a better antenna position or better connection. After I saw the same behavior on a second dongle I realized that wasn't that likely.

Maybe disconnecting the antenna is triggering an automatic gain adjustment which doesn't reset back even after a number of minutes have elapsed?

Fresh run with -M noise:60 Disconnected the antenna briefly after 5 minutes.

Tuner gain set to Auto.
Tuned to 433.900MHz.
Allocating 15 zero-copy buffers
MQTT Connected...
baseband_demod_FM: low pass filter for 250000 Hz at cutoff 25000 Hz, 40.0 us
MQTT Connection established.
Current level -15.6 dB, estimated noise -15.5 dB
Current level -15.6 dB, estimated noise -15.4 dB
Current level -15.8 dB, estimated noise -15.6 dB
Current level -14.5 dB, estimated noise -15.9 dB
Current level -15.6 dB, estimated noise -15.6 dB
Current level -12.3 dB, estimated noise -33.0 dB
Current level -12.2 dB, estimated noise -33.0 dB
Current level -12.6 dB, estimated noise -33.0 dB
Current level -11.6 dB, estimated noise -33.0 dB
Current level -12.5 dB, estimated noise -33.0 dB
Current level -7.0 dB, estimated noise -33.0 dB
Current level -12.6 dB, estimated noise -33.0 dB
Current level -7.2 dB, estimated noise -33.0 dB
Current level -6.3 dB, estimated noise -33.0 dB
Current level -12.3 dB, estimated noise -33.0 dB
Current level -12.8 dB, estimated noise -33.0 dB

[zuckschwerdt]

You might want to enable the print in https://github.com/merbanan/rtl_433/blob/master/src/rtl_433.c#L374 to get the level for each frame. The "Current level" values are from just the current frame, likely noise but could also be a transmission. The "estimated noise" only tracks "Current level" for changes within 3 dB (line 381 and 384). The guesswork is to dumb to track the 18 db jump, mistaking that for a signal, and gets stuck :( I think we might need periodic reset of the auto level or something like "if the level is high (thought to be signal) for too many frames take it as new noise estimate".

Perhaps you could explore other frequency bands and see if there is always so much noise? Away from 70cm band the reception should get silent, right? I'd probably say that -15dB is always a genuine signal for me. I have stable reception of one sender with -21 db and sometimes see TPMS with -28 dB even. Using R820T2 and this simple antenna (screwed on the original base with the MCX connector) https://m.media-amazon.com/images/I/511rurRjq+L._AC_SL1000_.jpg

Would be great if other users could report a -M noise reading.

[Sineos]

FWIW this is a short noise reading on my side:

ubuntu@ubuntu:~$ rtl_433 -M noise -R0
rtl_433 version 21.05-54-g5f74389b branch master at 202107181850 inputs file rtl_tcp RTL-SDR
Use -h for usage help and see https://triq.org/ for documentation.
Trying conf file at "rtl_433.conf"...
Trying conf file at "/home/ubuntu/.config/rtl_433/rtl_433.conf"...
Trying conf file at "/usr/local/etc/rtl_433/rtl_433.conf"...
Trying conf file at "/etc/rtl_433/rtl_433.conf"...
Disabling all device decoders.
Registered 0 out of 190 device decoding protocols [ ]
Detached kernel driver
Found Rafael Micro R820T tuner
Exact sample rate is: 250000.000414 Hz
[R82XX] PLL not locked!
Sample rate set to 250000 S/s.
Tuner gain set to Auto.
Tuned to 433.920MHz.
Allocating 15 zero-copy buffers
Current level -8.1 dB, estimated noise -15.1 dB
Current level -8.1 dB, estimated noise -15.1 dB
Current level -8.1 dB, estimated noise -15.1 dB
Current level -8.1 dB, estimated noise -15.1 dB
Current level -8.1 dB, estimated noise -15.1 dB
Current level -8.1 dB, estimated noise -15.1 dB
Current level -8.1 dB, estimated noise -15.1 dB
Current level -8.1 dB, estimated noise -15.1 dB
Current level -8.1 dB, estimated noise -15.1 dB
Current level -8.1 dB, estimated noise -15.1 dB
Current level -8.1 dB, estimated noise -15.1 dB
Current level -8.1 dB, estimated noise -15.1 dB
Current level -8.1 dB, estimated noise -15.1 dB
Current level -8.1 dB, estimated noise -15.1 dB
Current level -8.1 dB, estimated noise -15.1 dB
Current level -8.1 dB, estimated noise -15.1 dB
Current level -8.1 dB, estimated noise -15.1 dB
Current level -8.1 dB, estimated noise -15.1 dB
Current level -8.1 dB, estimated noise -15.1 dB
Current level -8.1 dB, estimated noise -15.1 dB
Current level -8.1 dB, estimated noise -15.1 dB
Current level -8.1 dB, estimated noise -15.1 dB

[zuckschwerdt]

Here are random samples how my band appears. Should be RSSI: -19.0 dB Noise: -25.5 dB RSSI: -15.9 dB Noise: -26.3 dB and RSSI: -19.2 dB Noise: -23.8 dB (with default color map the background just touches the "green" at ~-20dB, set the gain to 15 and range to 6 and that should be what rtl_433 detected.) My own senders are near 0 dB, the neighbours around -10dB.

calm.zip

[zuckschwerdt]

I've fixed the autolevel to rise slowly in addition to the fast fall. This will track changing noise even if it's initially classified as signal. It assumes less than 1/4 (fast fall / slow rise ratio) band utilization, otherwise the noise estimate will skew towards signal level.

[rct]

Been seeing some periodic reception problems lately, so tracking the noise estimates as data is back on my mind. I'm capturing stderr to a log file but the lack of timestamps complicates things.

So looking at https://github.com/merbanan/rtl_433/blob/master/src/rtl_433.c#L398

Would it be safe (and reasonable) to turn that fprintf into a call to the data_make machinery?

Another alternative, what about adding the current level and estimated noise to the existing stats?

Any thoughts on "message types" to be able to differentiate received data from stats, noise level estimates?

[zuckschwerdt]

Already done for graphing in the HTTP UI ;) It's now in the feat-httpevs branch, but unfinished.

[rct]

That looks interesting and potentially very useful for short term analysis! How heavyweight is it? One of my first rtl_433 installations is still happily running on a single core RPi 1B+.

I'd like to capture the RF environment data so that it can be graphed over several day periods. One of the odd things I'm trying to figure out is why when we go away for a few days do I start to have more issues with reception. Our laptops, phones, tablets leave the house, the HVAC system runs less often, so I'd think reception would improve instead of getting worse. So far I can't think of anything that would correlate with more RFI when we are away...

[zuckschwerdt]

You actually only need this single change (addition) https://github.com/merbanan/rtl_433/commit/9fc6b9ad3320122c6903f31ad4ba72e217dcf90f#diff-e67dede6781f117c63290cc1a9458711acc3dbf27c545d107f2dd5bf32b7fd1bR401-R409 But switch the event_occurred_handler_api() to just event_occurred_handler() and probably tune the report time there -- currently each second) then you get noise reports as data. There is no added computational load. It's just a change to output what's already there.

All the other changes are to track undecoded messages (the gray dots in the graph above) and funnel output to the API only.

[rct]

Thanks for this, I've got that working.

Have you had any thoughts about adding a message type to ensure there is no ambiguity during parsing?

It would be nice to make this data available via MQTT as well. It will likely need a new topic, though I could also see it being added to the states topic.

[zuckschwerdt]

The logic currently is that messags with a "model" key are a decode, everthing else is something like logging. There is some overlap with stats reports and such "not a decode, but also not a one-off user message". I guess we need to collect all possible cases, then work through that to get anywhere ;)

[rct]

I see with the recent log changes you now need -F log to get the noise/level estimate logged.

Has any way been added to:

• make the level/noise estimates added to MQTT?
• get a timestamp on the output
• Output it as JSON?

The HTTP view is nice, but I'd really like to make it easier to integrate this with the other data

[zuckschwerdt]

The log output is just a compatibility to still get log to stderr. Use the level on another output, e.g. the "Auto Level" logs at LOG_WARNING = 4 so this will work: -F "json,v=4" {"time" : "2023-03-11 10:17:10.032165", "src" : "Auto Level", "lvl" : 4, "msg" : "Current noise level -14.8 dB, estimated noise -14.6 dB"}

[rct]

Ah got it -- That's neat, Thanks!

If this is documented some place let me know.

What should be used to indicate that the type of this message is log data (as opposed to a data event or statistics)?

What are your thoughts on making this (and possible other RF env) data accessible via MQTT?

Am I on the wrong track thinking it should be there?

[zuckschwerdt]

Well, any day now I'll write some proper docs …just one more feasture to finish :)

Log messages will always be "time", "src", "lvl", "msg".

Yes, the "states" topic should be used for logging in MQTT. It's not fully hooked up yet. Just adding mqtt->output.log_level = 9; near the end of output_mqtt.c should work.

[rct]

Log messages will always be "time", "src", "lvl", "msg".

For consumer, it would be helpful to have a single field that could be used to figure out what type of data it is. Otherwise we wind up with heuristics like:

• if "model" is present, must be a message from a device.
• if "enabled" is present, must be statistics
• if [ "src", "lvl", "msg" ] is present, must be a log message.

[gdt]

That cries out for having a type field.

[rct]

@gdt wrote:

That cries out for having a type field.

Yes, that's the point I was trying to make. Ideally it would be a type field that is a reserved, so decoders can't also use that field.

[rct]

Yes, the "states" topic should be used for logging in MQTT. It's not fully hooked up yet. Just adding mqtt->output.log_level = 9; near the end of output_mqtt.c should work.

While that worked to send this log message over MQTT, I wound up backing this change out.

Overloading what goes in the states topic means you can no longer easily parse the states message with relative simple parsers like Home Assistant. The fields I was getting data from event and frame counts aren't in the more frequently published level log messages, so this generate a lot of errors. When the actual states message gets published, it gets overwritten, so if you want to use that data, you need to be continually connected to get each "states event".

(yes, there are ways to parse that overloaded topic in a consumer like Home Assistant but it becomes more cumbersome than it needs to be.)

[gdt]

My impression is that things are better. In my own system I am seeing every 10s reports that look like:

{"time":"2023-09-29 12:00:50","src":"Auto Level","lvl":4,"msg":"Current noise level -26.0 dB, estimated noise -25.7 dB"}

I'm unclear on noise-with-decode, noise-with-stats, and noise-every-10s. Is this fixed, or is something else desired?

[luconedj]

Well, any day now I'll write some proper docs …just one more feasture to finish :)

Log messages will always be "time", "src", "lvl", "msg".

Yes, the "states" topic should be used for logging in MQTT. It's not fully hooked up yet. Just adding mqtt->output.log_level = 9; near the end of output_mqtt.c should work.

Hello! I'm trying to output to MQTT with no success:

error: ‘struct data_output’ has no member named ‘log_level’
  582 |     mqtt->output.log_level = 9;
      |                 ^
make[2]: *** [src/CMakeFiles/r_433.dir/build.make:328: src/CMakeFiles/r_433.dir/output_mqtt.c.o] Errore 1
make[1]: *** [CMakeFiles/Makefile2:951: src/CMakeFiles/r_433.dir/all] Errore 2
make: *** [Makefile:146: all] Errore 2

How can i route it towards mqtt output? Thanks!

[zuckschwerdt]

Make sure you have the current source code. The member log_level is there, see include/data.h The proper line would be below all the similar ones, currently https://github.com/merbanan/rtl_433/blob/master/src/output_mqtt.c#L603

[luconedj]

Thank you very much @zuckschwerdt! Somehow i forked the repo and cloned locally the wrong one 😶‍🌫️. Cheers, Luca

[luconedj]

My final goal is to implement some kind of stupid anti-jamming detection based on the noise level and some threshold. I think this should be a good start. What is your opinion? Thanks! Luca

[zuckschwerdt]

That's a great idea. Please test some ideas about that and post your findings! Many people would benefit if we could add that and diagnostic messages e.g. saying "it's too quiet, is your antenna broken?" "it's too loud, is there interference?"

Note that Noise is already a good figure (above say -15 dB: too noisy, below -34 dB too quiet) but since the auto-gain changes randomly you should also look at signal dB when one is available and thus SNR.

I would however not use MQTT for the bulk of those messages but rather HTTP. Connect to e.g. :8433/stream, see examples/rtl_433_http_*

If maybe you want noise data for all frames (4x per second or more) we could likely also add that.

[luconedj]

Yes, indeed. For MQTT , it was just because my initial though was to put some automation in Home Assistant. I'll follow your suggestion with stream for testing purpose, i'll try to play with it this weekend :) Thank you! Luca

[zuckschwerdt]

Replace this with just { to get all noise measurements always. https://github.com/merbanan/rtl_433/blob/2122cc2670ea5166d02b4ddd9a36077e69a3b9f9/src/rtl_433.c#L472

[gdt]

Seems like this is quite valid and would be a big improvement. I am a little unclear on where we are and what's next. (Personally, I'm ok with a json-is-the-answer world and a stats key in the dict to separate from a decode.)