zuckschwerdt
commented
2 years ago To add an example thought on the unix philosophy (but please don't let this distract from the topic here): I've been thinking for some time if splitting out the hopping logic to an external controller would help. rtl_433 can switch band, rate, and options easily with e.g. the http api. The requirements for hopping logic (conditions, time, timeout), are currently somewhat distracting in the code and don't fit everybodys needs. It is a very nice feature to have ready-to-go otoh.
qwindelzorf
DAVe3283
rct
merbanan
gdt
quaec
For various reasons rtl_433 may output messages with bad data, such as humidity values that fall outside of the range of 0 - 100. For most protocols message integrity checks (MIC) based on checksums, CRCs, and/or parity bit should cause bad messages to be rejected. However this doesn't always work. MIC are not 100% accurate. In some cases it is possibly hardware error with the device itself, a sensor might be bad or have a temporary glitch, but the device calculates the MIC correctly with the bad data. (There have been cases in some decoders identified where a bitbuffer of all zeros could pass the MIC checks like checksum. In a number of cases, those decoders have had checks added to prevent this.
This is a topic that at first glance seems somewhat trivial, but it isn't. Points of view may change after a year or more of operating a system with a variety of sensors. There is no one correct answer to this for all applications. I've been thinking about this for a while, but haven't tried to implement anything yet because I don't have a view yet for what is the correct thing.
The case for not filtering in rtl_433
This leads to the question of whether rtl_433 should do some sanity check of data, An argument against doing sanity checking in rtl_433 is an architectural view that rtl_433 should do its best to handle the RF and lower layers of the bit manipulation in a very general manner and then get out of the way to be of use to a wide range of consumers.
Related to this is the architectural view that rtl_433 should not be maintaining any state for devices, that should be up to consumers and state requirements might be not only consumer specific, but could also be device specific.
Not doing any filtering/sanity checking makes sense when application specific code reads from rtl_433 and can make it's own decisions. For example:
rtl_433's ability to implement sanity checking is limited by not keeping state
The architectural view that rtl_433 shouldn't maintain any state limits the types of sanity checking that can be done within rtl_433. Data that is within range but still potentially bad can't be filtered by rules that need history. For example rejecting values because the rate of change exceeds a threshold.
Not keeping state in rtl_433 does have many benefits for a clean design. It helps keep rtl_433 general and usable for a wide range of uses cases beyond just receiving weather and environmental data. An application that uses rtl_433 to receive data from security sensors has different requirements than a weather console.
Common use cases that need filtering (for lack of sanity checking in other applications)
However in the last few years, rtl_433 has added generalized publishing for messaging (MQTT) and storage (InfluxDB). A common use case is:
In addition to display data, and acting on it an automation platform like Home Assistant may be distributing the data to SQL DB, a time series DB like InfluxDB, and other messages like MQTT. In the Home Assistant case, there is currently no capability for filtering / sanity checking incoming data in a general way. So any bad data winds up everywhere, displays, graphs, alerts, and automations are all impacted. It could be argued that it is consumers like Home Assistant, InfluxDB, that need to change and implement some sanity checking. But the question is how practical is that versus changes in rtl_433?
It could be argued that adding filtering to rtl_433 adds complexity in a place where it doesn't belong. It could also be argued that rtl_433 possibly should not be publishing directly to messaging systems or long term storage systems like Time Series Databases (TSDBs). The question is whether to a large monolithic application or a smaller application that follows the UNIX or microservices-like philosophies.
There is a lot of value in keeping things simple. With Home Assistant, it's possible for people who don't want to get into the details to put together a some what off-the-shelf system with an RTL-SDR to get access to many RF ISM band devices. There are containers for running rtl_433, MQTT broker, and the rtl_433 MQTT auto discovery, without much command line work. This has opened up rtl_433 to a much wider audience.
When not to filter
There are cases when filtering may not make sense or could be detrimental. Messages often contain more than one piece of data. Devices often contain more than one sensor, for example temperature and humidity.. It's possible for one sensor to be bad, but other sensors to still be good.
If the humidity sensor on a weather station goes bad, should all data from the weather station be discarded until the humidity sensor is fixed?
(This is a case I've observed. humidity sensors for outdoor weather stations can become contaminated.)
If some sensors are giving bad data because the battery is low, would filtering prevent messages with the battery low indication from being seen?
(This is a case I've observed. I've had a device where the temperature goes to -40 and the humidity 0 when the battery is very low, but not fully dead yet.)
With filtering enabled, how difficult is it for a user to differentiate between reception problems and a bad sensor causing messages to be discarded?
Where to implement sanity checks and filtering?
Ad hoc checks in decoders
It's tempting to eliminate bad data by implementing ad hoc checks in decoders. Right now this is the only place to get any immediate relief from bad data.
To make it clear that messages are being rejected, there is usually a debug fprintf to stderr, but that is protected by if verbose mode check. So each of those checks is a multi-line block that violates the DRY principle. So the DRY problem is going to multiply as sanity checks are added to more devices.
Given the potential problems mentioned above in when not to filter, it's unlikely that all of the ad hoc checks can be easily enabled or disabled as needed without changing code.
As for the idea that all it takes to figure things out is to restart rtl_433 in verbose mode, consider users who are running rtl_433 in a container on some sort of embedded system like their home automation platform.
Finally, there is an architectural view that decoders should be fairly straightforward to write, maintain, and test. Implementing a lot of ad hoc
Data layer (aka data make)
The data layer could be a useful place for controlling whether to filter or not filter, or even whether certain clients should get a filtered stream or a raw stream.
The data layer could also be used as a way of passing messages (or data elements through) but marking them suspect.
Some data elements like humidity could be filtered without decoder specific knowledge. However something like temperature can not. The range for an indoor sensor is very different than a cooking (oven, bbq) sensor.
The data layer would need info from the decoders about valid ranges. So this would require a reasonably large amount of changes.
A downstream "helper" application
A downstream helper application that reads from rtl_433, filters, and then distributes to some or most of the endpoints that rtl_433 is already supporting could solve a number of problems:
The examples directory is supposed to have things like this, but I don't think any of them are in wide enough use to really gain any traction.
Building a companion app for rtl_433 that handles filtering and distribution might help rtl_433 be less monolithic and might make things easier in the long run. Using an event-based framework that already has support for communicating with various types of data stores and messaging systems could help making things easier and more reusable.
Other considerations
Should the entire message be discarded for one bad data element?
There are cases outlined above where it would be useful to still get the message even though one data element is bad, namly one bad sensor on a multi-sensor device.
On the other hand, if it is a message integrity problem, one bad data element can a helpful indication that other elements might also be invalid even if they pass range checks.
Discard or mark messages with bad data?
Discarding a message with bad data is helpful to simple consumers that accept everything. However this also prevents other consumers from seeing potentially useful information.
Marking messages with bad data, would be more generally useful but would require changes in consumers to discard messages that are marked as suspect. This would currently be difficult in say Home Assistant using the common
rtl_433 -> MQTT -> Home Assistanttopology.Determining why a device is isn't being received or is periodically dropping out is challenging
At the time of this writing, tracking down why a device isn't being received, or worse, is only intermittently being received is currently laborious and requires a not insignificant amount of skill and deduction. As mentioned above if whole messages are discarded, data problems can be difficult to distinguish from reception problems.
The developer experience might be to test one device at a time with running rtl_433 in verbose mode and possibly removing the antenna to help in isolating the one nearby device under test.
However end users, such as the home automation use case may have many devices covering temperature, humidity, water detection, security, presence, energy, etc. So differentiating between an RF problem, a device problem, or a problem with the code presents more challenges in this case. The techniques a developer uses would be fairly disruptive to shut down the home automation system during the length of the debugging, run things manually, relocate devices, remove the antenna etc.
Conclusion
This is a request for comments and discussion. It would be good to collect not only opinions, but also operational experience that might help guide this. Thanks for making it this far.
Edit: Added Other considerations section.