last_updated and last_changed flags, add_ele data point, logging arriving data from the sensors

[AustrianSimon]

I am trying to work out a solution so that the field add_ele, so far quite mysterious, can be accounted properly and be converted into a total_forward_energy (analog to the smartmeters) sum. However, I face several issues, which probably go back to the home assistant but also to localtuya as far as I could analyse so far.

Situation with the original Tuya Cloud:

With the Tuya Cloud it is also impossible to determine whether a new data point packet for add_ele has arrived. However, the Tuya Cloud offers a device log in which each and every single packet, for each data point, is being listed.

Understanding that add_ele means "add electricity" I did a test and just summed up the add_ele datapoints listed in the Tuya log, compared to actual power, voltage and duration, and the data matched, so I found this valid (and pretty accurate):

total_forward_energy (like smartmeters) = Sum of all add_ele values provided by the smart plug

Now, this led to an extensive attempt to implement this onto Home Assistant with localtuya, however, I had to give up for two reasons:

There is no possibility to determine whether a new data packet (and thus an add_ele data point) has arrived. The (default) trigger on a value change (last_changed) doesn't work, because add_ele fields usually are very similiar as long as the conumer does not vary its consumption.

The last update (last_updated) flag also doesn't work, because even in theory it included updates by the user interface, rebooting, etc.

I also noticed during testing this all, that there is NEVER any difference between last updated and last changed dates, they are always identical, even after arrival of the same value (however, triggering on this last_updated or last_changed attribute also doesn't solve it, the routine misses several add_ele packets). In theory the last updated date should be updated on EVERY arrival of a new data point packet, and the last changed flag should only be updated if the value of the data point has indeed changed. But that's not how it works. I noticed the last updated flag remained unchanged several time even though a new data packet had arrived (at the Tuya Cloud) with the very same value as the previous transmission however.

So I regard this fact a bug, that both last_updated and last_changed are always in sync and even last_updated doesn't get updated upon arrival of a new packet though with same data as previously. Whenever a data point packet arrives, the last updated date needs to be updated, and if the value changes that last changed date should also change.

But that is still insufficient as I pointed out above: updates can also occur by system issues, via the web interface, and so on. So this would introduce possibly quite significant errors and discrepancies into the computations.

Hence we need another time stamp of when a real/physical data packet has arrived from the device, and this one needs to be triggerable. As far I could see in my data analysis, this should be possible with localtuya adding a readable attribute to the entity, e.g. last_received data, which then could .e.g be read by an own pyscript or even by pyscript module directly and used for triggering actions. It might also be an idea to have localtuya log, similiar to the Tuya Cloud, each and every packet arriving from devices connected to localtuya and make this log avaible via the APIs of Home Assistant (similiar to how the official Tuya Cloud does it).

The consequence of this would be:

1) the add_ele can be properly converted into a total energy calculation (and similiar packets where every single packet counts) 2) arriving data packets can be logged properly like the Tuya Cloud (device log in debugging devices) 3) external devices (dataloggers, analysis software, ...) via the various APIs (including REST API) could more easily synchronise with the Home Assistant and also track those packets

[AustrianSimon]

Additional note:

After developing a work around, which comes pretty close to the original at the Tuya Cloud, I find, that the DP 17 packets of the smart plug are not being forwarded if the value transmitted is 0 (while the other DPs like current etc. do get transmitted). Hence I can't trigger on those 0 add_ele packets. This also is definitely a bug.

[Harted]

Hi, I also faced the same problem.

I created my own service with codetheweb/tuyapi

// you should store this value somewhere so this doesn't get overwritten every time you start the service
// I use a retain value in my mqtt broker
let consumption = 0 

device.on('dp-refresh', data => {
  // dps17 = add_ele
  const dps17 = data.dps['17'] ? Number(data.dps['17']) : undefined

  // This only gets updated when the accumulated value in the plug is send to the api
  // I tested and can confirm when the previous value is 1, the next update can also be 1
  // I thought it would only update when the value changes (is different), but it is not
  // dps 17 is only in the 'dp-refresh' data when updated
  if (!dps17) return

  // when present in data, add the 'add_ele' value to your consumption
  consumption += dps17
})

device.on('heartbeat', () => device.refresh({}))