Problems decoding ERT SCM messages (protocol 149)

[phrrngtn]

The problem was discussed on a thread in the google group: https://groups.google.com/forum/#!msg/rtl_433/loZFvPhCxOE/BmvuBqDLAQAJ

There are two parts to this issue. One is adding support for other ERT protocols such as SCM+, IDM. This was requested in #1378. The protocols are described in the rtlamr wiki https://github.com/bemasher/rtlamr/wiki/Protocol

The other part is that it seems that rtl_433 is missing a bunch of messages that are successfully decoded by rtlamr. See below for details of the attempts to reproduce. The most promising looking diagnostics came from rtlamr when run with -samplefile. It looks like the program is saving the raw samples to a file but also indicating the sample offset, number of samples and payload of the decode. Given the fixed sample rate (2359296), the baud rate of the signal (32768), the fixed length of the packet 96 bits, it should be possible to find the samples rtlamr is framing.

rtl_tcp -a 192.168.68.126

# start the multiplexer that dups traffic on port 1234 to 7678
rtlmux -h 192.168.68.126 -p 1234

# Run the go-based s/w that successfully decodes a lot of ERT traffic
rtlamr -server=127.0.0.1:7878   -msgtype=scm,scm+,idm

# Run rtl_433 as directed but using sampling rate and frequency copied verbatim from rtlamr diagnostics
rtl_433 -d rtl_tcp://192.168.68.126:7878  -s 2359296  -f 912600155 -R 149 -S all

# 
rtl_433   -s 2359296  -r g004_912.6M_2359.3k.cu8 -W test.sr

I only saw one pulse there.

I ran rtlamr with -samplefile which saves the samples with length and offset information. I assume that it is saving the data in the same format it is getting it from rtl_tcp but I have not checked the source to verify that. phrrngtn@sdrpi:~ $ /home/pi/go/bin/rtlamr -samplefile=foo.cu8 -server=127.0.0.1:1234 -msgtype=scm,scm+,idm

16:54:11.635072 decode.go:45: CenterFreq: 912600155 16:54:11.635723 decode.go:46: SampleRate: 2359296 16:54:11.636156 decode.go:47: DataRate: 32768 16:54:11.636211 decode.go:48: ChipLength: 72 16:54:11.636256 decode.go:49: PreambleSymbols: 32 16:54:11.636302 decode.go:50: PreambleLength: 4608 16:54:11.636346 decode.go:51: PacketSymbols: 736 16:54:11.636387 decode.go:52: PacketLength: 105984 16:54:11.636443 decode.go:59: Protocols: scm+,idm,scm 16:54:11.636591 decode.go:60: Preambles: 0001011010100011,01010101010101010001011010100011,111110010101001100000 16:54:11.636731 main.go:119: GainCount: 29 {Time:2020-05-18T16:54:14.905 Offset:0 Length:229376 SCM:{ID:61105633 Type: 4 Tamper:{Phy:00 Enc:00} Consumption: 66405 CRC:0x833F}} {Time:2020-05-18T16:54:15.359 Offset:229376 Length:229376 SCM:{ID:60xxxxx3 Type: 4 Tamper:{Phy:00 Enc:00} Consumption: 66499 CRC:0x1D14}} {Time:2020-05-18T16:54:16.519 Offset:458752 Length:229376 SCM:{ID:60xxxxx9 Type: 4 Tamper:{Phy:00 Enc:01} Consumption: 93564 CRC:0xA02C}} {Time:2020-05-18T16:54:17.856 Offset:688128 Length:229376 SCM:{ID:61xxxxx3 Type: 4 Tamper:{Phy:00 Enc:00} Consumption: 66405 CRC:0x833F}} {Time:2020-05-18T16:54:18.079 Offset:917504 Length:229376 SCM:{ID:60xxxxx3 Type: 4 Tamper:{Phy:00 Enc:00} Consumption: 66499 CRC:0x1D14}} {Time:2020-05-18T16:54:20.236 Offset:1146880 Length:229376 SCM:{ID:1xxxxx22 Type:12 Tamper:{Phy:03 Enc:00} Consumption: 139242 CRC:0x9EBC}} {Time:2020-05-18T16:54:21.192 Offset:1376256 Length:229376 SCM:{ID:6xxxxx87 Type: 4 Tamper:{Phy:00 Enc:01} Consumption: 46699 CRC:0x8C1A}} {Time:2020-05-18T16:54:21.687 Offset:1605632 Length:229376 SCM:{ID:6xxxxx89 Type: 4 Tamper:{Phy:00 Enc:01} Consumption: 93564 CRC:0xA02C}} {Time:2020-05-18T16:54:24.031 Offset:1835008 Length:229376 SCM:{ID:6xxxxx87 Type: 4 Tamper:{Phy:00 Enc:01} Consumption: 46699 CRC:0x8C1A}} {Time:2020-05-18T16:54:24.195 Offset:2064384 Length:229376 SCM:{ID:6xxxxx03 Type: 4 Tamper:{Phy:00 Enc:00} Consumption: 66499 CRC:0x1D14}} {Time:2020-05-18T16:54:25.526 Offset:2293760 Length:229376 SCM:{ID:6xxxxx33 Type: 4 Tamper:{Phy:00 Enc:00} Consumption: 66405 CRC:0x833F}} {Time:2020-05-18T16:54:27.142 Offset:2523136 Length:229376 SCM:{ID:4xxxxx79 Type:12 Tamper:{Phy:00 Enc:00} Consumption: 629012 CRC:0xDB2C}} {Time:2020-05-18T16:54:30.082 Offset:2752512 Length:229376 SCM:{ID:6xxxxx03 Type: 4 Tamper:{Phy:00 Enc:00} Consumption: 66499 CRC:0x1D14}} {Time:2020-05-18T16:54:31.633 Offset:2981888 Length:229376 SCM:{ID:6xxxxx03 Type: 4 Tamper:{Phy:00 Enc:00} Consumption: 66499 CRC:0x1D14}} {Time:2020-05-18T16:54:33.030 Offset:3211264 Length:229376 SCM:{ID:6xxxxx33 Type: 4 Tamper:{Phy:00 Enc:00} Consumption: 66405 CRC:0x833F}} {Time:2020-05-18T16:54:33.796 Offset:3440640 Length:229376 SCM:{ID:6xxxxx87 Type: 4 Tamper:{Phy:00 Enc:01} Consumption: 46699 CRC:0x8C1A}} {Time:2020-05-18T16:54:36.138 Offset:3670016 Length:229376 SCM:{ID:6xxxxx03 Type: 4 Tamper:{Phy:00 Enc:00} Consumption: 66499 CRC:0x1D14}} {Time:2020-05-18T16:54:37.422 Offset:3899392 Length:229376 SCM:{ID:6xxxxx03 Type: 4 Tamper:{Phy:00 Enc:00} Consumption: 66499 CRC:0x1D14}} {Time:2020-05-18T16:54:38.346 Offset:4128768 Length:229376 SCM:{ID:6xxxxx33 Type: 4 Tamper:{Phy:00 Enc:00} Consumption: 66405 CRC:0x833F}}

I tried to follow the instructions but pulseview would not start up by itself after this invocation. I was able to start it manually and there were samples up to the 1 second mark rtl_433 -R 149 -s 2359296 -r ~/foo.cu8 -W test.sr

[phrrngtn]

I have uploaded the rtlamr sample file to dropbox and renamed it as rtlamr_sample_output.cu8

https://www.dropbox.com/s/zu7331md1y5gh0m/rtlamr_sample_output.cu8?dl=0

It looks like these are not an entire time-series of the signals but rather just the blocks that are deemed to contain messages. These means that they will be very squished together and look like a much smaller sample that indicated by the timestamps in the program output.

[merbanan]

To me this is a clear indication that the signal detector is the part that needs fixing first. There might be other issues also but I also observed this with the pogsac frequencies. Pogsac are fsk modulated signals that are transmitted randomly and it should be easy for the segmenter to pick it up but I've never been able to get it working.

Right now the segmenter uses I^2+Q^2 to get an envelope. If the envelope is larger then a threshold a signal is detected. I am hopeful that this detector could be used instead or as a complement:

%% test demod for research purposes
for i = 3:length(x)

Isum = in_i(i) - in_i(i-2); %% Q1.7
Qsum = in_q(i) - in_q(i-2); %% Q1.7

iprod = Isum * in_q(i-1); %% Q1.7 * Q0.7 -> Q1.14
qprod = Qsum * in_i(i-1); %% Q1.7 * Q0.7 -> Q1.14

out(i) = qprod - iprod; %% Q2.14

out(i) = ((in_q(i)-in_q(i-2)) * in_i(i-1)) - ((in_i(i) - in_i(i-2))*in_q(i-1));
end

Or something fft based. Old ffmpeg code has a 16bit integer fft that could be reused.

[merbanan]

The segmenter is not able to frame the signal. So that is problem one. Then when you look at the AM signal it is very very very low. The pulse recovery sensitivity of the OOK code needs some rework, and that is problem two. With both of these fixed the performance should be comparable. Rtlamr uses matched algorithms and will always be a more better decoder. But it should be possible to get rtl_433 good enough.

[phrrngtn]

Although I don't fully understand your answers (due to my ignorance of DSP), this is still helpful! I note from the https://github.com/bemasher/rtlamr/wiki/Signal-Processing documentation the use of a FIR filter to boost the signal. I was unsure as to what taps to use (note that I have been doing my experiments in LuaRadio; I started doing independent tests in rtl_433 as I was not getting any results at all with my efforts in LuaRadio). Would you mind elaborating(in your own time) a bit on this comment:

Rtlamr uses matched algorithms and will always be a more better decode

To the best of my understanding, 'all' it knows is the OOK, baud rate, the preamble and the length of the packet. To my software-biased perspective, I am perplexed that my efforts (in LuaRadio) have not worked!

BTW, despite my lack of knowledge of DSP, please consider me very motivated to do whatever grunt work needs to be done on this that can be done by by a DSP novice. My day job is in the Reinsurance industry, modeling losses to insurance and reinsurance contracts due to natural catastrophes. I would like to use logged data from electric/PV, gas and water meters to help people understand their resource footprint on a more fine-grained basis than a monthly/quarterly bill and I think that processing the ERT signals may lower the barrier to entry.

pjjH

[zuckschwerdt]

I already have some code to debug amplitude and magnitude histograms in the pulse detect. I'll try to add this later today.

[zuckschwerdt]

Also as a quick test you can use the magnitude estimator -Y magest instead of the amplitude. It has wider dynamic range, theoretically.

[zuckschwerdt]

Test #1387 with e.g. rtl_433 -Y verbose -Y verbose and rtl_433 -Y verbose -Y verbose -Y magest to get a picture what the detector "sees".

[phrrngtn]

I checked out and built feat-atthist and see output like the following (zero entries removed). I can run it for longer if needed but it seems pretty consistent.

[-]
>-22 dB:    44 smps
>-23 dB:   379 smps
>-24 dB:  2015 smps
>-25 dB:  5142 smps
>-26 dB:  6496 smps
>-27 dB:  3879 smps
>-28 dB:  1420 smps
>-29 dB:   305 smps
>-30 dB:    44 smps
>-31 dB:     4 smps
>-32 dB:     1 smps
>-33 dB:     0 smps
Levels low: -36 dB  high: -11 dB  thres: -17 dB  hyst: (-15 to -18 dB)

>-13 dB:   101 smps
>-14 dB:  6265 smps
>-15 dB: 10844 smps
>-16 dB:  2477 smps
>-17 dB:  1150 smps
>-18 dB:   957 smps
>-19 dB:   823 smps
>-20 dB:   805 smps
>-21 dB:   783 smps
>-22 dB:   926 smps
>-23 dB:  1494 smps
>-24 dB:  4716 smps
>-25 dB: 10517 smps
>-26 dB: 13547 smps
>-27 dB:  7253 smps
>-28 dB:  2634 smps
>-29 dB:   473 smps
>-30 dB:    70 smps
>-31 dB:     4 smps
>-32 dB:     0 smps
>-33 dB:     0 smps
Levels low: -36 dB  high: -28 dB  thres: -33 dB  hyst: (-32 to -34 dB)

[merbanan]

I tried with the suggested detector and it is not performing any better then the current code. The threshold code might need some better tuning or we need to move to a fft based approach. Brief testing in octave shows promise.

[zuckschwerdt]

The histograms posted show a clear picture:

1. a block of pure noise, centered around -26 dB
2. a block of data, bimodal distribution with peaks at -15 dB (data) and -26 db (noise) However since the default values for the detector are a minimum of -12.1442 dB to see a signal the "high" level does not account for the data. With a minimum "high" level of -15 or -16 dB (not a cut off, just an initial value to lock on) the algorithm should find a threshold at -21 db (lowest count in this histogram) or at least somewhere between the peaks.

Perhaps capture a sample that shows this behaviour on replay and play with: -Y level=0.0 -Y minlevel=-12.1442 -Y minsnr=9.0 (defaults shown).

[merbanan]

@phrrngtn the biggest difference is the matched filter and that the decoding is always "active". rtl_433 needs to "find" a signal and then try to decode it.

[bemasher]

rtlamr author here. I agree with @merbanan, rtlamr is sort of cheating in this case because it was designed for optimum sensitivity to manchester-coded OOK signals at the cost of being far less efficient than rtl_433. I'm not very familiar with rtl_433's design, but if a bit-decided signal is available to the decoder at a higher sample rate than the symbol rate, a matched filter could be used to provide better sensitivity without increasing resource requirements too much. Especially since rtl_433 was written in C rather than go, where more sophisticated compilers can produce better code for a specific architecture.

[fmuntean]

Any update on this? I see differences even inside rtl_433 itself when using Flex decoder with the same parameters. In order to test this I changed the code for scm to report raw data and record both the flex output and the ert-scm output then used excel to summarize the results:	raw data	SCM	Flex
015306110014D2A1BA46C160	2	129
0153061300A7A0A58E8C197D	1959	2860

So flex decoder seems to "see" more events than the scm decoder and these raw data event are deem correct by the SCM. Maybe somebody can explain this behavior. As I changed the parameters in the code for completeness these are the parameters used

SCM:
r_device ert_amr = {
        .name        = "ERT",
        .modulation  = OOK_PULSE_MANCHESTER_ZEROBIT,
        .short_width = 30,
        .long_width  = 30,
        .gap_limit   = 1000, //0,
        .reset_limit = 500, //64,
        .decode_fn   = &ert_decode,
        .disabled    = 0,
        .fields      = output_fields,
        .tolerance   = 20 //us
};

FLEX:
-X decoder n=FLEX,m=OOK_MC_ZEROBIT,s=30,l=30,t=20,r=500,g=1000,bits=96,unique

[merbanan]

@fmuntean the decoder will reject messages with bad crc. That could be one reason. But if there is any other reason we need signal recordings that works with the flex decoder but not the other decoder.

[gdt]

Status?

[gwww]

I don't know if there is appetite to resurrect this thread... I've been trying to get rtl_433 working reading my water meter. Same reason as some others here, I want a single program that can hop between 433/915 and read a number of radio devices using a single dongle.

Using rtlamr I get 5 reports in under a minute. Using rtl_433 I get 2 reports in about a minute.

Running rtlamr I get the following:

18:15:25.360043 decode.go:45: CenterFreq: 912600155
18:15:25.360284 decode.go:46: SampleRate: 2359296
18:15:25.360299 decode.go:47: DataRate: 32768
18:15:25.360307 decode.go:48: ChipLength: 72
18:15:25.360315 decode.go:49: PreambleSymbols: 21
18:15:25.360324 decode.go:50: PreambleLength: 3024
18:15:25.360332 decode.go:51: PacketSymbols: 96
18:15:25.360340 decode.go:52: PacketLength: 13824
18:15:25.360351 decode.go:59: Protocols: scm
18:15:25.360361 decode.go:60: Preambles: 111110010101001100000
18:15:25.360370 main.go:126: GainCount: 29
{Time:2024-06-30T18:15:31.934 SCM:{ID:34138679 Type:11 Tamper:{Phy:00 Enc:00} Consumption:  669105 CRC:0x91B9}}
{Time:2024-06-30T18:15:44.826 SCM:{ID:34138717 Type:11 Tamper:{Phy:00 Enc:00} Consumption:  188800 CRC:0x539D}}
{Time:2024-06-30T18:15:49.661 SCM:{ID:34138681 Type:11 Tamper:{Phy:00 Enc:00} Consumption:  276841 CRC:0x6225}}
{Time:2024-06-30T18:16:08.656 SCM:{ID:34107077 Type:11 Tamper:{Phy:00 Enc:00} Consumption:  279138 CRC:0x45C7}}
{Time:2024-06-30T18:16:18.658 SCM:{ID:34136051 Type:11 Tamper:{Phy:00 Enc:00} Consumption:  258200 CRC:0x55B7}}

Running rtl_433 I get far fewer water meters being picked up. It is getting some, which is encouraging! Unfortunately, my meter is not being picked up. The command I'm running is: rtl_433 -v -f 912600155 -s 2359296 -X "n=SCM,m=OOK_MC_ZEROBIT,short=30,long=0,reset=64,bits=96,get=@32:{24}:consumption,get=@56:{24}:ID". The output is:

rtl_433 version 23.11-142-g7e4cef28 branch master at 202406281317 inputs file rtl_tcp RTL-SDR with TLS

New defaults active, use "-Y classic -s 250k" if you need the old defaults

[Protocols] Registered 225 out of 260 device decoding protocols [ 1-4 8 10-12 15-17 19-23 25-26 29-36 38-47 49-60 63 67-71 73-85 87-100 102-105 108-116 119-122 124-128 130-149 151-161 163-168 170-175 177-197 199 201-215 217-232 234-241 243-244 246-247 249-259 ]
[Input] The internals of input handling changed, read about and report problems on PR #1978
[SDR] Found 1 device(s)
[SDR] trying device 0: Realtek, RTL2838UHIDIR, SN: 00000001
Detached kernel driver
Found Rafael Micro R820T tuner
[SDR] Using device 0: Realtek, RTL2838UHIDIR, SN: 00000001, "Generic RTL2832U OEM"
[R82XX] PLL not locked!
[SDR] Sample rate set to 2359296 S/s.
[Input] Bit detection level set to 0.0 (Auto).
[SDR] Tuner gain set to Auto.
[Input] Reading samples in async mode...
[SDR] Tuned to 912.600MHz.
[Baseband] low pass filter for 2359296 Hz at cutoff 471859 Hz, 2.1 us
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
time      : 2024-06-30 18:18:47
model     : SCM          count     : 1             num_rows  : 1             rows      :
len       : 96           data      : 02a6085802c2ac10ce784ad1                consumption: 180908       ID        : 1101432
codes     : {96}02a6085802c2ac10ce784ad1
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
time      : 2024-06-30 18:19:18
model     : SCM          count     : 1             num_rows  : 1             rows      :
len       : 96           data      : 02a608580614d811d4da6e8d                consumption: 398552       ID        : 1168602
codes     : {96}02a608580614d811d4da6e8d

I'm pretty new at this. Is there something I can do/provide/run to help with this issue?

[zuckschwerdt]

Does it help if you relax some constraints? E.g. skip the MC decoding (m=OOK_PCM,short=30,long=30) where you might then need to align on a preable like preamble=5559 And allow slightly different length (bits>=90,bits<=110)

[gwww]

I'm still playing. @zuckschwerdt I tried all the suggestion you have. None are working. I also just realized that in my log above the two readings that are there are both bogus. ID and consumption numbers are wrong.

After reading this thread again it seems that changes are required to the rtl_433 code to get this to work.

I'd be happy to test some more if there are more suggestions or if the code base is changed to try and address this.

[gdt]

I lean to a real decoder and not flex. Not sure if that makes it easier to debug.