Open triptec opened 2 years ago
The length value is in bits, so I guess you want {8}
instead of {1}
?
Oh and @8
instead of @1
, @16
for @2
...
Thanks for your reply! Yeah bits, right, I was missing something that's for sure =) I tried the changes (and a multitude of others) but this was the result.
rtl433tomqtt_1 | _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
rtl433tomqtt_1 |
rtl433tomqtt_1 | time : 2022-03-22T06:06:30
rtl433tomqtt_1 | model : MY_DECODER
rtl433tomqtt_1 | count : 1
rtl433tomqtt_1 | num_rows : 1
rtl433tomqtt_1 | rows :
rtl433tomqtt_1 |
rtl433tomqtt_1 | len : 280
rtl433tomqtt_1 | data : aaaaaaaaa39ae54596597659658d62b4cb2ab2ab1795931cb16764b2ab35938696aca0
rtl433tomqtt_1 | len : 170
rtl433tomqtt_1 | to : 170
rtl433tomqtt_1 | from : 170
rtl433tomqtt_1 | id : 170
rtl433tomqtt_1 | flags : 163
rtl433tomqtt_1 | payload : 0
rtl433tomqtt_1 | crc : 0
rtl433tomqtt_1 | codes : {280}aaaaaaaaa39ae54596597659658d62b4cb2ab2ab1795931cb16764b2ab35938696aca0
rtl433tomqtt_1 | _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
rtl433tomqtt_1 |
I've spent some more time reading the radiohead_ask.c and the things going on in this function seems to be more complex than I thought, can that be expressed in the flex decoder?
If any one have a suggestion on another radio transmitting lib that's easier to decode than RadioHead, do tell. I'm going to try TinyRF and see if I can decode that.
I'll probably also create a RadioHead + my data types but is it's possible to do that as some sort of plugin or something because there will probably just be me using it so upstreaming that wouldn't make sense and using my own fork I'd prefer not to. Any suggestions around this?
This is my first exposure to the world of micro controllers and radio and it's been a blast so I will probably make a few other cheap sensors for the garden and whatnot and would really like to decode them with rtl_433 because it works great picking up the data from all my proprietary sensors, so any suggestions around this going forward? I've been thinking about dynamically loaded libs (.so, .dll, .dylib) but that sounds like a lot of work, make the flex decoder able to handle more complex protocols but that would make it even more complex than it already is. So I'm starting to think of perhaps adding something like "Radiohead ASK custom" decoder to the source code but be able to configure name, and what fields to decode the payload into and maybe also some way of matching against headers and/or payload to tell different sensors apart would be nice for custom payload that I'll be sending and perhaps other people doing something similar. Is it something that sounds like a good idea or is it out of scope or is there another solution?
Ah, yes, sorry. You would need to align to a preamble (so the aaa
in the front go away) but then there is the 6to4 coding which can't currently be removed by the flex decoder.
Well, the Radiohead ASK decoder does work, like you see above with "model : RadioHead-ASK". Perhaps use that as input to a Python script and further decode the payload there.
If you only have a few standard values to transmit (temp/hum) maybe transmit in another known protocol, i.e. emulate some well known sensor.
Yeah, I'm doing this for a electricity meter that sends:
double cumulativeActiveImport;
double cumulativeActiveExport;
double cumulativeReactiveImport;
double cumulativeReactiveExport;
double momentaryActiveImport;
double momentaryActiveExport;
double momentaryReactiveImport;
double momentaryReactiveExport;
double momentaryActiveImportL1;
double momentaryActiveExportL1;
double momentaryActiveImportL2;
double momentaryActiveExportL2;
double momentaryActiveImportL3;
double momentaryActiveExportL3;
double momentaryReactiveImportL1;
double momentaryReactiveExportL1;
double momentaryReactiveImportL2;
double momentaryReactiveExportL2;
double momentaryReactiveImportL3;
double momentaryReactiveExportL3;
double voltageL1;
double voltageL2;
double voltageL3;
double currentL1;
double currentL2;
double currentL3;
So piggybacking on existing stuff might be hard.
I've managed to get the TinyRF to work =) with:
-X 'n=MY_DECODER,m=OOK_PPM,s=256,l=508,g=520,r=1520,preamble={119}000000000000000000000000000000,get=@0:{8}:length,get=@8:{8}:crc,get=@16:{8}:seq,get=@24:{64}:msg'
but is there any way to use the length field to pass that to the "get=@24:{64}:msg" but something like "get=@24:{length*8}:msg"? Also I've tried to add format but I get segfaults when I do "get=@24:{64}:msg:%s" to try to print it as a string?
Hmm, is it not possible to decode values into DATA_DOUBLE using the flex decoder?
Note that double
is a bad choice to transmit. Excessively long and machine specific. Including header you are transmitting 2300 bits for what should be maybe 400 to -- which is still very much data.
It will be exceeding regulatory limits, and even if you don't care, it will be interfering with other peoples devices badly.
Known the range you want to transmit, choose endianess, and use a proper coding like e.g. 1-2 bytes per field. Do this first.
You want to apply a scale and offset to every value. E.g for voltage in a 230V system you known that the maximum allowed range is +/- 10%, say you want one decimal, then you are looking to encode e.g. 207.0 to 253.9 which is 469 values. Use 9 bits for that. For a value with wide range (current) pick a mantissa and exponent, e.g. 8 bit mantissa gives you 2.4 digits, and 4 bits exponent (choose a bias and base) can give you e.g. a range of x0.1 to x1000.
Alright, guess I got some work to do =) But in any case this won't make it as a decoder so would it be and idea for me to extend the flex decoder to something being able to handle this?:
get=@24:{9}:(value/10+207):voltage,store=@31:{8}:current_mantissa,store=@39:{4}:current_exponent,get=@31:{12}:(current_mantissa^current_exponent):current
Interesting idea. More flexible rules to decode fields are on the wishlist for the flex decoder for a long time now. But doing that in C won't be smooth. Perhaps a good idea to protoype something in Python first.
Glad to hear it would be in line with the project. When you say prototype in python, do you mean just writing a parser to try out the added syntax of the flex decoder or are we also talking using that parsed config to actually decode messages?
Also when you say "More flexible rules to decode fields" is there more than basic arithmetic?
wmbus can handle several float values. You could look and see if that is an option.
I'm just guessing that parsing even basic arithmetic rules wouldn't be easy in C. That's why I recommended some more advanced and forgiving language like Python.
E.g. extend https://github.com/merbanan/rtl_433/blob/master/examples/rtl_433_custom.py
to grab the raw payload bytes (data["payload"]
) and then process them. No flex decoder needed, just the existing RadioHead-ASK decoder to give a payload.
There should even be some bit handling libs like https://pypi.org/project/bitstring/
The wishlist for the flex decoder would include things like filters (checksum verification) and codings (like 6-to-8 coding and such).
@merbanan I don't think it would work. But then I'm not sure what you're suggesting perhaps but the meter is a landis gyr e360. It has a rj12 jack that produces an ascii output every 10 seconds like:
b"/ELL5\x5c253833635_A\r\n\r\n"
b"0-0:1.0.0(210217184019W)\r\n"
b"1-0:1.8.0(00006678.394*kWh)\r\n"
b"1-0:2.8.0(00000000.000*kWh)\r\n"
b"1-0:3.8.0(00000021.988*kvarh)\r\n"
b"1-0:4.8.0(00001020.971*kvarh)\r\n"
b"1-0:1.7.0(0001.727*kW)\r\n"
b"1-0:2.7.0(0000.000*kW)\r\n"
b"1-0:3.7.0(0000.000*kvar)\r\n"
b"1-0:4.7.0(0000.309*kvar)\r\n"
b"1-0:21.7.0(0001.023*kW)\r\n"
b"1-0:41.7.0(0000.350*kW)\r\n"
b"1-0:61.7.0(0000.353*kW)\r\n"
b"1-0:22.7.0(0000.000*kW)\r\n"
b"1-0:42.7.0(0000.000*kW)\r\n"
b"1-0:62.7.0(0000.000*kW)\r\n"
b"1-0:23.7.0(0000.000*kvar)\r\n"
b"1-0:43.7.0(0000.000*kvar)\r\n"
b"1-0:63.7.0(0000.000*kvar)\r\n"
b"1-0:24.7.0(0000.009*kvar)\r\n"
b"1-0:44.7.0(0000.161*kvar)\r\n"
b"1-0:64.7.0(0000.138*kvar)\r\n"
b"1-0:32.7.0(240.3*V)\r\n"
b"1-0:52.7.0(240.1*V)\r\n"
b"1-0:72.7.0(241.3*V)\r\n"
b"1-0:31.7.0(004.2*A)\r\n"
b"1-0:51.7.0(001.6*A)\r\n"
b"1-0:71.7.0(001.7*A)\r\n!"
There are DIY solutions for this with arduino and esp32/8266 but they all use wifi and I happen to have my meter on the other side of my lawn and my wifi access point isn't very well positioned and it's not as fun (it was to easy) as trying 433mhz. Specially since I already got a rtl-sdr and rtl_433 -> mqtt -> home assistant setup.
My suggestion is to switch protocol. Wmbus can hold arbitrary data.
@zuckschwerdt ok, yeah I guess you're right that it might not be easy but the more I think of it I like this idea the most. And I think the RadioHead protocol still would be too complex (the 6to4 isn't a standard right? and I'm not sure how one would define something like that other than rh_6to4() or somehting that would be available in the arithmetic part) but something simple like the TinyRF would be quite doable.
The rtl_433_custom.py would work, do the decoding and publish to mqtt but it would be another moving part that I'd prefer not to have if I can help it. I'd probably just fork at this point and copy paste the RadioHead decoder and decode the fields there.
Talking about the wishlist, do you guys envision extending the current flex decoder format or doing something new? And do you have an example of how you'd like it to work?
This is off the top of my head but it might be naive. get would be fields that is creates fields in the output like today, store would be to create variables to use in following statements, match would be able to take block and match that against a variable perhaps
get=@xx:{y}:length,
get=@xx:{y}:(rh_6to4(value)/10+207):voltage, // using predefined functions
store=@xx:{y}:current_mantissa,
store=@xx:{y}:current_exponent,
get=(current_mantissa^current_exponent):current,
get=@xx:{length}:some_dynamic_field, // using dynamic length
get=@xx+length:{y}:after_dynamic_field, // using dynamic offset
get=@xx+length+y:{y}:crc,
match=(crc8(data)):crc, // using predefined function when maching/validating
Trying to implement simple arithmetic between parenthesises like (value/10+207) could perhaps be a good test to see how much of a pita it might be?
The 6to4 is not standard. It is needed however as transmitting rows of many 0's or 1's won't work well. A common option would be Manchester coding, but that's at a worse ratio of 2:1. But you would choose a higher bit rate there anyway (more efficient on battery and channel utilization).
The flex decoder format isn't tied to key=value. The idea was a function-like approach like:
check_crc([8..47], 0x31, 0)
and outputs could more simply be something like
temperature_C := [8..15] * 0.1 - 200
or even
temperature_C := [signed 8..15] * 0.1
We could make a nice enough language, but I don't really want to implement a parser (pita, yes).
@merbanan as in implementing it on the microcontroller side, both the protocol and how to send it with a radio transmitter and on the receiving side have something like rtl_wmbus to get the transmissions?
@zuckschwerdt sorry, I'm not suggesting you guys implement it but that if I do you entertain the idea of giving feedback and perhaps in the end accept a PR
rtl_433 can decode wmbus, but yeah there is effort involved to get it working. But you could look at the encoding and reuse those parts.
A custom protocol could be useful. Something OOK but with a dynamic payload and adhering to proper regulations.
Okay, that sounds like you would entertain the idea? If that sounds ok I'll try implement a proof of concept:
Alt 1 Use the RadioHead ASK library on a mcu and send some data (responsible and within regulations) Implement needed functionality to be able to decode that using the flex decoder.
Alt 2 (if the radiohead stuff is over my head) Use the TinyRF library on a mcu and send some data (responsible and within regulations) Implement needed functionality to be able to decode that using the flex decoder.
What do you say?
Sure, try a few things with the flex code. But expect that to be an exploration only ;) There will be dragons.
So I've not even started and I'm having a problem =) As test data I took your example of having odd bits so I created a struct like so:
struct MyStruct
{
unsigned int voltage1 : 9;
unsigned int voltage2 : 9;
unsigned int voltage3 : 9;
unsigned int padding : 5;
};
MyStruct my_struct={229,230,231,0};
const char* msg = (char*)&my_struct;
send((byte*)msg, sizeof (my_struct));
I'll stick to good old 8 or 16 bits for now. But is it my way of send the data or the way I'm decoding it that's the problem?
~ # rtl_433 -X 'n=LOLZ,m=OOK_PPM,s=256,l=508,g=520,r=1520,preamble={119}000000000000000000000000000000,get=@0:{8}:length,get=@8:{8}:crc,get=@16:{8}:seq,get=@24:{9}:v1,get=@33:{9}:v2,get=@42:{9}:v3,get=@51:{5}:p'
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
time : 2022-03-22 21:54:46
model : LOLZ count : 1 num_rows : 2 rows :
len : 0 data : length : 0 crc : 0 seq : 0 v1 : 0 v2 : 0 v3 : 0
p : 0,
len : 56 data : 040905e5cc9d03 length : 4 crc : 9 seq : 5 v1 : 459 v2 : 306 v3 : 232
p : 3
codes : {0}, {56}040905e5cc9d03
Endianess I guess. Bit fields and esp. wider than 8 won't be portable.
E.g. for wind direction (360 values -> 9 bits) we usually see FF WW GG DD
with Flags (battery, 9th bits, ...) and Wind Gust Dir all just the lower 8 bits.
So I've been working on this on and off and I've tried a few alternatives:
So that brings me to perhaps the next alternative which I haven't tried yet but that would basically be like 2 but creating specific devices that decode the payload into a bitbuffer that would be simple to pick values out of.
What do you think?
And if one would go down that path should it's be implemented like all the other devices or would it be possible to do something special?
Right now I pass a data_acquired_handler that just copies the pointer to the data to a static and reading the data as I said is not really ergonomic.
Or perhaps adding a way to load custom devices through dynamic libraries?
load custom devices through dynamic libraries?
No. We discussed this before. Telling (and training) users to load random binaries they found on the net is really bad.
to bind more variables
it should not be needed to predefine (bind) random stuff. There should be a "bit extraction operator". E.g. [signed 3] * [8..16] ^ [4..7]
piggyback on a already defined protocol. option to the flex decoder that will run the decoder first and then basically reference the data fields
Sounds too nested. It would be preferred to make each steps that decoder would do available to the flex. E.g. chksum(), decode_6to4().
Ok, about your suggested thing on top the [signed 3] would be the hardest as I see it now.
would (-([0..1] * 2) + 1) * [1..2] * [8..16] ^ [4..7]
, (-([0..1] * 2) + 1)
being the part that would get the sign?
Because I'm thinking that parsing [x..y] and do an extraction on that and then exchange that for a variables and pass it to the expression lib to do the rest of the calculation. Even something like [func1(func2(x..y))] wouldn't be too hard to handle, basically we'd evaluate things between [ and ] and there we'd only do bit extraction and giving the possibility to call functions on that and the result of "our" processing would be cast into a double and fed to the expression lib.
But would that be enough? Do we need [signed X] or something else?
You are right, for this simple cast -[0] * ...
will work for a "negative flag". But for 2's complement we need e.g. signed( [0..11] )
.
Generally I guess we would want some functions (signed, invert, 6to4_decode, ...), there is always something to transform ;)
In the most abstract form [...]
should be just a slice of bits. If short enough it could be coerced to int or float (dependent on the expression) or manually converted (like a cast). But in other cases it needs to stay a slice of bits (crc_chk([0..79], 0x31, 0)
).
I guess the two pieces are: a parser and executor for arbitrary functions, and more complicated the "slice of bits" operation.
Ok, sounds good. To make things simple and hopefully something to build on I'd prefer to at least start with something like
signed := signed([0..11]) mantissa := [11..5] exponent := [16..5] some_value := signed mantissa 2 ^ exponent crc := crc8(0..79)
Basically start with functions that only take one argument. And also nothing that would be special like crc_chk but we'll see..
If you want to solve this in a general way, then likely parse the input to nested functions, transforming all special cases. E.g.
current_A := signed([0..11]) * 2 ^ [16..5]
becomes
add_output("current_A", mult(signed(slice(0, 11)), pow(2, slice(16, 5))))
Then unroll the tree depth-first to RPN:
5
16
slice
2
pow
11
0
slice
signed
mult
"current_A"
add_output
that's the program you then want to run on a stack-machine.
@zuckschwerdt how would you feel about a radiohead + msgpak device? would you be open to adding the msgpak lib as a dep? more then happy to contrib i currently unwrap payloads using a python script but wouldn't mind if it was self contained
MessagePack is an interesting choice to allow flexible transmissons. I don't think that we would commit to anything but the small fixed bit formats we currently have.
Using an external wrapper/bridge script to unpack data is recommended and we plan to make that easier and better supported.
What is the state of this issue and path forward? While there are some side musings about architectural issues, it reads like a discussion that has concluded, which makes me think just closing is appropriate.
But perhaps a Radiohead decoder which receives and output bits in some container format, if the Radiohead side is standardize, should be PRed.
(I'll close in 30 days absent a plan for progress.)
@gdt i would be more then happy to add a RadioHead decoder, I will sit down and reimplement https://github.com/enavarro222/rtl_433/blob/ASK/src/devices/radiohead_ask.c
We do have Radiohead and also an example of using it: https://github.com/merbanan/rtl_433/blob/master/src/devices/radiohead_ask.c We should enhance the flex decoder to support these types of protocols.
Hello and thanks for a great piece of software!
So I'm making a little arduino serial reader that sends data using RadioHead ask, the protocol works great, it decodes my "Hello world!" as numbers but great. But when I try to make a flex decoder
I get
I intend to implement some kind of format for the data, without modifying the source code if possible but I can't really figure out how to do it, it sounds like it shouldn't be too hard, am I missing somthing?. It would be great with a example of a flex decoder for RadioHead because I think a lot of home tinkerers would be using that lib.
Thanks again.