Introduce DEVICE/realtime topic for better realtime reporting

[TurkeyMan]

Feature suggestion

Cross posting to a new issue, since the other place I wrote this is really discussing a different issue.

For reference; I need high quality realtime monitoring of sensors. I'm writing monitoring software which can identify and isolate individual devices present on a circuit by identifying their signatures during operation. To build these signature profiles, I need high resolution raw data, otherwise the signatures are very poor. This allows pseudo-monitoring of individual devices connected to power outlets with just a single meter on the circuit.

Many classes of devices are very easy to identify by their particular waveforms of real and reactive power; things with pumps like fridges, air con, induction cookers, specific tools in the workshop, etc, relatively easy to infer with a high resolution signal.

These devices are realtime raw data sources; the software running on the device itself shouldn't limit the way it's used. The data should be accessible at the highest resolution that the device can record.

Maybe MQTT isn't the best protocol for realtime monitoring... my other modbus-based devices poll at ~20hz (report every 50ms), and I consider that slow (limited by RS232 at 9600bps, much lower than ~100mbps!). It'd be much more useful to have a connection to these devices with an active data stream which just pushed data the instant it samples it, at the same frequency that it samples.

Short of new protocol development, without changing the behaviour of these devices as they are, I can suggest a method for improving realtime usage. To achieve this within MQTT, here's what I think might work well:

1. Since MQTT can only include a single topic/value in a PUBLISH message, we should create a topic for realtime message aggregation; ie, DEVICE/realtime
2. Device should PUBLISH to DEVICE/realtime more-or-less immediately for every hardware sample in realtime (or some specified rate) and include all recent raw samples. Include in the message all topic+value pairs that changed since the last report as native resolution.
3. ...and most importantly; publish with QoS=0 to eliminate all the ACK chatter.
4. Maybe allow configuration for the devices internal sampling frequency, or allow to specific a subset of fiends (ie, current + power) I need to sample at the highest possible frequency if the performance of the device is a limiting factor and sampling all sensors at high frequency would be too much for the chip.

I reckon this could be implemented while leaving all other implementation details as they are, just an additional option to enable DEVICE/realtime reporting and some associated frequency config values.

[openshwprojects]

What about using a scripting system to force publish single large packet with all data every 0.25 seconds?

[TurkeyMan]

0.25 is extremely slow... how about every 0.01 seconds? Will scripting be able to deal with that frequency? I think the essence of what I'm suggesting is because these are very small devices, and writing the logic in C will make it very simple and compact, and improve the sample rate substantially. I reckon that if you implemented this here, esphome and tasmote would probably copy it.

[openshwprojects]

Which devices do you have? None of the devices I know even support that kind of resolution. There is no need to poll every 50ms if, for example, TuyaMCU sends update every 5 seconds.

What you are saying sounds totally impossible, even BL0942 states 400ms/800ms update interval for RMS values: I_FAST_RMS seems slightly better but it's still above 150ms...

Can you provide more information on the setup you are using?

[TurkeyMan]

There is no need to poll every 50ms if, for example, TuyaMCU sends update every 5 seconds.

I'm sorry; of course, this is naturally true, and it's subject to the limitations of the hardware. I want to sample at the fastest possible rate for the hardware; whatever rate that happens to be. If it's 400ms (intensely slow), then standard MQTT messages will be fine, but I do have devices with hardware that's better that... Is it the case there are no devices implemented with high sampling rates?

What I'm getting at here, is that I don't want the device software to be the limiting factor under any circumstances. I want a deliberate and documented feature which I can rely on to report data at the highest possible sample rate, and with no resolution truncation. I can't know the hardware limitations of all the possible devices, and I think it's important that the software presents a dependable method to achieve the closest thing to "raw/realtime sampling" as possible, with respect to the hardware's limits.

I wonder what chips my modbus meters are using; they update at a decent rate (~50ms, apparently limited by 9600 baud rate). I never pulled one apart... they're kind-of expensive.

TuyaMCU sends update every 5 seconds

Really? That Tuya chip is that slow? Can it be polled at a higher rate? What about the direct IO connected chips? Surely the software reads from the chip directly? In that case, it's subject to the maximum polling rate of the chip, which I imagines would be milliseconds, although you show it's 400-800ms in the datasheet above :(

Well, it may be the case that my work doesn't map to any of these devices... but I have a whole bunch of them, and I thought I might as well attempt to use them for testing.

[TurkeyMan]

Thinking on it; you know the software and the hardware, and you probably know what's best.

So ignoring my suggestion, here's the goal: Configuration so that without knowing the device specifications/limitations, the software will report at the maximum possible frequency, and with no precision truncation with respect to the hardware's limitations. That's all. I just want the raw data as fast as it can possibly be sampled, with no delays or precision loss. Maybe this is already achievable? If I set VCPPrecision/VCPPublishThreshold for high sensitivity, and VCPPublishIntervals to a very small number? What values will catch all hardware cases for realtime reporting?

Can I be confident that the software is internally sampling at the maximum possible rate that the hardware supports in all cases?

[openshwprojects]

Can you please tell me what is your end goal with that? What do you want to achieve with this frequent sampling? What are you doing later with that data?

WiFi in general is not recommended for that kind of applications and those chips are not optimized for that.

Futhermore, no one does "report at the maximum possible frequency" because it saturates the network quickly, always only the changes are reported.

[TurkeyMan]

FFT's and other time based event/feature analysis. I mean, I change what I'm doing with it every day as I work on different ideas. The thing I care about here, is having the highest quality data made available by the hardware. The software just needs to sample and report at the highest possible rate. If the rate is too low, then some features are not detectable, if it's higher, certain features become visible. The quality of results with respect to the work I'm doing will always be subject to the capabilities of the device; I just want to be certain I am working at the limits of the device capabilities, and not limits the software added on top by not sampling at the highest rate possible, or reporting with a lower precision than the hardware allows, or delaying the reports so that the events are placed on the waveform at the wrong times.

WiFi in general is not recommended for that kind of applications and those chips are not optimized for that.

You're right, and it would be ideal if the reports were accompanied with a hardware timestamp! I've been thinking about this, but I haven't made a post regarding that yet :P .. I wanted to measure the jitter in report time to determine how much noise the wifi latency introduced, or if the latency was at least fairly consistent.

I felt like I need to work through the initial problem that reports are like 5-10 seconds apart at best... network jitter is irrelevant if several seconds between reports is the best I can get. I'm playing wit the tuning options now...

Futhermore, no one does "report at the maximum possible frequency" because it saturates the network quickly, always only the changes are reported.

Sure, changes are fine... although even the maximum possible rate would definitely not saturate the network, since these devices have sample rates that barely approach kilobits per seconds let-alone megabits per second.

[openshwprojects]

Why would you want to run FFT on RMSs values that are sent over the WiFi? What is the application of that?

As far as I know, in order for FFT to give any meaningful results, you would need to use high-speed ADC directly on the AC wave.

[TurkeyMan]

There are loads of forms of feature detection, I have observed several devices with detectable operating signals measuring RMS fluctuation. Devices with particular duty cycles are easily visible watching RMS that way. I have seen some devices that phase between active power and reactive power; if I see the same frequency present in both signals, then I can determine that feature with decent confidence. 400ms samplerate might be too crude to measure it, but with a long enough signal, the frequencies might appear amplified. My reference device measures at 20hz. I'm just trying some other meters for experimental sake, and that lead me here. I started sticking all the meters I have laying around on all the circuits I can find just to record heaps of data. Yesterday I was recording a bunch of signals from power tools, and I think with some work I could find fingerprint strategies for most of them. I'm experimenting, it's all just ideas, and I'm trying to see how far I can get with the quality of data generally accessible. I was just shocked when I received data about about 0.2hz. I figured that can't be a hardware limitation.

Incidentally, I've set VCPPublishThreshold threshold to very small values, and VCPPublishIntervals to 1ms, and I'm reliably receiving samples about every 400ms, so I guess that tracks... that might correlate with the hardware updating its registers. It doesn't seem that apparent and reactive power reports at all ever? I never received a report for those values, although they appear on the web admin.

Should I be alarmed that the config also notes powerfactor ~1.7? Something's very wrong there... I calibrated the device as best I knew how.

[openshwprojects]

If the wave of AC is 50Hz, how do you want get any meaningful data while sampling at 20Hz? Nyquist–Shannon sampling theorem states that you need much higher sample rate than that.

Which devices, are you talking about BL0942? Both BL0937 and BL0942 are taking a sample every second currently:

[TurkeyMan]

I'm not looking for AC waveform features, need an oscilloscope for that! More like duty cycle features. Try it! Sample some data at a decent resolution and twiddle with it in matlab... there's lots of ways I've found visible features that I think can be refined into reliable fingerprints. RMS signal isn't steady. At very least, they're pretty noisy. In some cases, the noise itself seems to have some structure that's not immediately apparent, and I'm becoming convinced that some of that noise isn't just measurement error.

It's really useful to have the reactive signal aswell... active power doesn't tell the whole story.

This device I'm playing with now is BL0942, it's reporting just a little more than twice per second. Possibly the 400hz that the hardware reference mentions? I'm not sure how I'm seeing updates at that rate considering the code snippet you show here... That said; see that code _RunEverySecond; that's EXACTLY what I'm talking about here... that's bad! The software appears to be limiting the device capability in that case. That should be fixed to match the hardware capabilities in that case rather than just picking an arbitrary polling rate. It's a shame for the software to limit the hardware in any case.

[TurkeyMan]

I just measured it rather than guessing. It does seem to be about 1hz samplerate on average. I see a bit of jitter though, and some samples arrive closer together which made me think the samplerate must be higher.

Can we consider that 1s timer a bug?

[TurkeyMan]

Here's another odd behaviour; every 6th second it seems to report evergycounter_ values instead. On that second it doesn't report the other values. They get skipped. Can that be fixed? Or is it that the chip can only serve a single report in that timeframe, and you ask for the counters instead of the realtime signals, or some explanation like that?