New Energy Monitor - Missing data 'no data available'

[NorthernMan54]

I have created a new plugin, and can't find the correct combination to fix the 'no data available' graph under the on/off button. I have created this as a 'Service.Switch' and am populating the historical data with time, power and status.

fakegatoService.addEntry({
          time: Date.now(),
          power: this.accessories.find(accessory => accessory.UUID === element.uuid)?.getService(this.Service.Switch)?.getCharacteristic(this.CustomCharacteristic.CurrentConsumption).value ?? 0,
          status: (this.accessories.find(accessory => accessory.UUID === element.uuid)?.getService(this.Service.Switch)?.getCharacteristic(this.Characteristic.On).value ?? false ? 1 : 0 )

I have tried using an Outlet and no luck either. Does anyone have a working example I can look at ? Or is this just not feasible.

The top portion of this image.

[simont77]

I’ve only implemented the data consumption in my plugin, I don’t know how it is in the real Eve Energy, but I believe that the status should have a place inside the history protocol. Right now fakegato provides only the power value for the Power type.

[ebaauw]

The original Eve Energy exposes itself as Outlet, with history only for power, not for on/off. This is what I use in Homebridge Hue.

I think the newer ones and/or the power cord use a different scheme, but I’ve never got my hands on any of these.

[NorthernMan54]

I was really hoping it was a setup issue on my side, but it looks like I may need to obtain one and do some reverse engineering.

Tks @simont77 @ebaauw

[bwp91]

I own the eve outlet and have decided to get it out the drawer to check this.

Hasn't been plugged in for very long, but it definitely seems like both graphs (on/off and consumption) are being populated.

I would need guidance but is there a way I could help out with this?

[ebaauw]

See https://github.com/simont77/fakegato-history#how-to-contribute.

[bwp91]

See https://github.com/simont77/fakegato-history#how-to-contribute.

Ooh thanks i’ll check that out

[bwp91]

00000023-0000-1000-8000-0026bb765291
Logging

E863F11E-079E-48FF-8F27-9C2605A29F52
 ; 1f00010e2800b8040a00c1f70c76db8eb891 ; 1f00010e2800b8040a00d7f32dcc8948c8aa

E863F112-079E-48FF-8F27-9C2605A29F52
0

E863F11C-079E-48FF-8F27-9C2605A29F52
 ; 01ffeaf9000000

E863F121-079E-48FF-8F27-9C2605A29F52

E863F116-079E-48FF-8F27-9C2605A29F52
 ; bab99f0173989f01ebe73025050b020c020d0207020e010010001031f70000ffffffff0100 ; 36ba9f0173989f01ebe73025050b020c020d0207020e010010001032f70000ffffffff0100

E863F117-079E-48FF-8F27-9C2605A29F52
 ;  ; 1231070100d5b99f010f0000ab0900000000

E863F131-079E-48FF-8F27-9C2605A29F52
 ; 000228000302b804040c46563335473141303138373906020010070431f700000b0200000501000204a83300005f0400000000190296001401030f0400000000450505000000004609050000000e000042064411051c000503000000003b694b42cf737cbf471105731b451cdf1cb81d780000003c00000048060500000000004a060500000000001a0400000000600164d00435789f019b04bab99f01d200 ; 000228000302b804040c46563335473141303138373906020010070432f700000b0200000501000204a83300005f0400000000190296001401030f0400000000450505000000004609050000000e000042064411051c000503000000003b694b42cf737cbf471105731b451cdf1cb81d780000003c00000048060500000000004a060500000000001a0400000000600164d00435789f019b0437ba9f01d200

E863F11D-079E-48FF-8F27-9C2605A29F52

[bwp91]

Two hours later:

00000023-0000-1000-8000-0026bb765291
Logging

E863F11E-079E-48FF-8F27-9C2605A29F52
 ; 1f00010e2800b8040a00696b0cc4565623e0 ; 1f00010e2800b8040a0064543a2ec3bc7d09

E863F112-079E-48FF-8F27-9C2605A29F52
0

E863F11C-079E-48FF-8F27-9C2605A29F52

E863F121-079E-48FF-8F27-9C2605A29F52

E863F116-079E-48FF-8F27-9C2605A29F52
 ; e1d29f0173989f01ebe73025050b020c020d0207020e01001000103cf70000ffffffff0100 ; 10d39f0173989f01ebe73025050b020c020d0207020e01001000103cf70000ffffffff0100

E863F117-079E-48FF-8F27-9C2605A29F52
 ;  ; 

E863F131-079E-48FF-8F27-9C2605A29F52
000228000302b804040c4656333547314130313837390602001007043cf700000b0200000501000204a83300005f0400000000190296001401030f0400000000450505000000004609050000000e000042064411051c000503000000003b694b42cf737cbf471105731b451cdf1cb81d780000003c00000048060500000000004a060500000000001a0400000000600164d00435789f019b044fd39f01d200

E863F11D-079E-48FF-8F27-9C2605A29F52

[bwp91]

A few more logs/pics that may help?

• The first log is after a couple of hours of randomly turning the outlet on/off. Before opening the eve app.
• The middle pictures are from then opening the eve app and refreshing.
• The last log is then after opening the eve app as above.

00000023-0000-1000-8000-0026bb765291
Logging

E863F11E-079E-48FF-8F27-9C2605A29F52
 ; 1f00010e2800b8040a004bd34811d43b0a92 ; 1f00010e2800b8040a004bd34811d43b0a92 ; 1f00010e2800b8040a00e0c1076d6e340ee3 ; 1f00010e2800b8040a002856290cc82263b3

E863F112-079E-48FF-8F27-9C2605A29F52
0

E863F11C-079E-48FF-8F27-9C2605A29F52
 ; 01ff3407010000 ; 01ff3407010000

E863F121-079E-48FF-8F27-9C2605A29F52

E863F116-079E-48FF-8F27-9C2605A29F52
 ; 4ed39f0173989f01ebe73025050b020c020d0207020e01001000103cf70000ffffffff0100

E863F117-079E-48FF-8F27-9C2605A29F52
 ; 123c0701009dd39f010f0000a409000000000b3d07010008d49f011000123e070100f5d59f010f0000030000000000123f0701004dd89f010f00000300000000001240070100a5da9f010f00000300000000001241070100fddc9f010f0000040000000000124207010055df9f010f00000400000000001243070100ade19f010f0000040000000000124407010005e49f010f000003000000000012450701005de69f010f00000400000000001246070100b5e89f010f000004000000000012470701000deb9f010f0000030000000000124807010065ed9f010f00000400000000001249070100bdef9f010f0000040000000000124a07010015f29f010f0000040000000000124b0701006df49f010f0000040000000000124c070100c5f69f010f0000040000000000124d0701001df99f010f0000030000000000124e07010075fb9f010f0000040000000000124f070100cdfd9f010f000004000000000012500701002500a0010f000003000000000012510701007d02a0010f00000400000000001252070100d504a0010f000003000000000012530701002d07a0010f000004000000000012540701008509a0010f00000300000000001255070100dd0ba0010f0000040000000000

E863F131-079E-48FF-8F27-9C2605A29F52
000228000302b804040c4656333547314130313837390602001007043cf700000b0200000501000204a83300005f0400000000190296001401030f0400000000450505000000004609050000000e000042064411051c000503000000003b694b42cf737cbf471105731b451cdf1cb81d780000003c00000048060500000000004a060500000000001a0400000000600164d00435789f019b044fd39f01d200

E863F11D-079E-48FF-8F27-9C2605A29F52

00000023-0000-1000-8000-0026bb765291
Logging

E863F11E-079E-48FF-8F27-9C2605A29F52
 ; 1f00010e2800b8040a00ceca77253979d572

E863F112-079E-48FF-8F27-9C2605A29F52
0

E863F11C-079E-48FF-8F27-9C2605A29F52
 ; 01ff340701000001ff3407010000 ; 01ff340701000001ff3407010000

E863F121-079E-48FF-8F27-9C2605A29F52

E863F116-079E-48FF-8F27-9C2605A29F52
 ; 6b6fa00173989f01ebe73025050b020c020d0207020e01001000108bf70000ffffffff0100

E863F117-079E-48FF-8F27-9C2605A29F52
 ; ;

E863F131-079E-48FF-8F27-9C2605A29F52
 ; 000228000302b804040c4656333547314130313837390602001007048bf700000b0200000501000204983300005f0400000000190296001401030f0400000000450505000000004609050000000e000042064411051c000503000000003b694b42cf737cbf471105731b451cdf1cb81d780000003c00000048060500000000004a060500000000001a0400000000600164d004396ea0019b046c6fa001d200

E863F11D-079E-48FF-8F27-9C2605A29F52

[ebaauw]

You probably need to expose a (dummy) characteristic for the schedule for the history to show. At least that’s the case for the Eve Thermo history.

[NorthernMan54]

Looking at the various characteristics against the existing work for energy outlet's, this device has different signature and data structure

Taking apart the E863F116 characteristic

4ed39f01 73989f01 ebe73025 05 0b020c020d0207020e01 001000 103cf700 00ffffffff0100

The signature is 05 0b020c020d0207020e01 versus 04 0102020207020f03 in the code

And for the E863F117 characteristic - historical data transport

12 3e070100 f5d59f01 0f 00 0003 0000000000

Versus the code

14 6c000000 a6980100 1f 00 0000 0032 0000000000  -> The 0032 is from my test device ie 5.0 Watts

Length is different 12 versus 14 and the type is different 0f versus 1f ( 0f is the same as room )

Looking at the binary data section don't have enough information yet to determine where the various data values are included. I'm expecting both power and status to appear

[NorthernMan54]

I think I missed something when I pulled apart the E863F117 characteristic - historical data transport

E863F117-079E-48FF-8F27-9C2605A29F52
123c0701009dd39f01 0f 0000 a409 00000000
0b3d07010008d49f01 10 00                                  <---- What is this ? the On/Off data?
123e070100f5d59f01 0f 0000 0300 00000000
123f0701004dd89f01 0f 0000 0300 00000000
1240070100a5da9f01 0f 0000 0300 00000000
1241070100fddc9f01 0f 0000 0400 00000000

When I get some time later going to try and mock up this data stream

[NorthernMan54]

And with mocked up data it appears to work

The top graph no longer says ‘no data available’.

Now to work on providing real data.

@bwp91 tks for your efforts on the data dump, looks like we have path forward

[NorthernMan54]

And success

The only thing that is troubling me is the location of the power information in the binary data. The sample from @bwp91 had the power information in bytes 3+4, where for me it needed to be in position 7+8

This is with the same device signature as well.

I have a first draft pull request submitted, but want to do more testing before publishing and look into why I'm double publishing the on/off status.

0b 1d020000e31700001001 
0b 1e020000f21700001001 
12 1f020000011800000f 0000 0000 0000 e00b <-- why byte position 7+8 and not 3+4 like the sample
0b 20020000011800001001 
0b 21020000101800001001 
12 220200001f1800000f 0000 0000 0000 ea0b 
0b 230200001f1800001001 
0b 240200002e1800001001 
12 250200003d1800000f 0000 0000 0000 ea0b 
0b 260200003d1800001001 
0b 270200004c1800001001
12 280200005b1800000f 0000 0000 0000 ea0b 
0b 290200005b1800001001 
0b 2a0200006a1800001001 
12 2b020000791800000f 0000 0000 0000 e00b 
0b 2c020000791800001001 
0b 2d020000881800001001 
12 2e020000971800000f 0000 0000 0000 d60b 
0b 2f020000971800001001 
0b 30020000a61800001001 
12 31020000b51800000f 0000 0000 0000 d60b 
0b 32020000b51800001001
0b 33020000c41800001001 
12 34020000d31800000f 0000 0000 0000 cc0b 
0b 35020000d31800001001 
0b 36020000e21800001001 
12 37020000f11800000f 0000 0000 0000 cc0b 
0b 38020000f11800001001 
0b 39020000001900001001 
12 3a0200000f1900000f 0000 0000 0000 cc0b 
0b 3b0200000f1900001001 
0b 3c0200001e1900001001 
12 3d0200002d1900000f 0000 0000 0000 c20b

[NorthernMan54]

@simont77 @bwp91 @ebaauw While I still have the patch open, anything else that needs taken care of?

I fixed the new Buffer deprecation issue, and an empty data issue for “energy” as well.

I’m doing a few more days of testing, and should be complete early next week.

[ebaauw]

Thanks, Sean, looking forward using this in Homebridge Hue (which currently uses the old energy history scheme, with only consumption).

Haven’t looked at your code, sorry, but the screenshot seems weird. The log above suggests you consume 5.06 Wh per minute or ~304W, so that seems about right, but I would expect one entry per 10 minutes, instead of two per minute.

When playing with a combined motion/temperature history (for the Hue motion sensor), I came to the conclusion that I’d best create a history entry with temp (average temperature) and status (motion) every 10 minutes, and an entry with only status whenever the motion state changes. I would expect here, you’d need an entry every 10 minutes with the status (on/off) and the consumption for the past 10 minutes; and an additional entry with only status when the on/off state changes.

[NorthernMan54]

@ebaauw My power consumption was broken, I was sending the current/amps value and not watts. I have since fixed my code. I'm still testing so may have other minor issues.

When playing with a combined motion/temperature history Do you have this working ? I was looking at adding this as well

In my setup, I'm using fakegato's global timer to manage the power consumption value and not my own code. And I'm sending power updates to fakegato every 5 minutes ( and it averaging it over 10 minutes ) and status update only on events. To make this work I had to tweak the addEntry function to accept separate power and status updates, and route them appropriatly

https://github.com/simont77/fakegato-history/blob/72878500066f5b3dbc1b8bfc842814721ba0e803/fakegato-history.js#L618

For power I'm using the global timer and for status bypassing it and adding it to the data set.

Tks for that breakdown of the signature, let me look at that further.

[ebaauw]

OK, looked at your code, comparing my notes (hit save too early, sorry):

this.accessoryType116 = "05 0b02 0c02 0d02 0702 0e01";

The 116 value defines the structure of the history. Here you expose history with values for five characteristics:

• 0b: unknown, two-byte value;
• 0c: V x10, two-byte value;
• 0d: unknown, two-byte value;
• 07: Wh x10, two-byte value;
• 0e: on/off, one-byte value.

The 117 value is a bitmap of the values included in the history entry.

this.dataStream += Format(
  " 12 %s%s%s 0000 0000 0000 %s", // Maximum value is 6.5 Kwh
  numToHex(swap32(this.currentEntry), 8),
  numToHex(swap32(this.history[this.memoryAddress].time - this.refTime - EPOCH_OFFSET), 8),
  this.accessoryType117,
  numToHex(swap16(this.history[this.memoryAddress].power * 10), 4));

This defines an entry of length 18 bytes (incl. the 12 length byte):

• 4-byte entry #;
• 4 byte timestamp;
• 1-byte bitmap of included values (in order) for: 0b, 0c, 0d and 07
• 2-byte dummy value 0000 for 0b;
• 2-byte dummy value 0000 for 0c;
• 2-byte dummy value 0000 for 0d;
• 2-byte real value for 07, i.e. Wh x10.

this.dataStream += Format(
  " 0b %s%s10%s",
  numToHex(swap32(this.currentEntry), 8),
  numToHex(swap32(this.history[this.memoryAddress].time - this.refTime - EPOCH_OFFSET), 8),
  numToHex(this.history[this.memoryAddress].status, 2));

This defines an entry of length 10 bytes (incl. the 0b length byte):

• 4 byte entry #;
• 4-byte timestamp;
• 1-byte bitmap: only the 5th field 0e;
• 1-byte value for 0e, i.e. on/off state.

You could probably simplify the 116 value to 02 0702 0e01 and post entries 0c %s %s 03 %s %s for power and on/off (in that order) and 0b %s %s 02 %s for only power.

Alternatively, you could try and send V x10 for 0c instead of fixed 0000. I don't think it will work, or you should already see an empty history under Voltage. I think 0b or 0dwould be for Current, but I never got it to work.

When playing with a combined motion/temperature history Do you have this working ? I was looking at adding this as well

Yes, in homebridge-lib, but not yet in Homebridge Hue. Using 117 of 02 1c01 0102 for 1-byte motion and two-byte temperature; and entries 0d %s %s 03 %s %s for both and 0b %s %s 01 %s for only motion.

I'm using fakegato's global timer

Had no luck with that so I'm using my own timer. Also had conflicts with different fakegato-history versions by different plugins.

[ebaauw]

For completeness, these are the codes I've collected so far:

  0102: temperature x 100
  0202: humidity x 100
  0302: air pressure x 10
  0402: ppm
  0601: open/close
  0702: W x10
  0b02: ?
  0c02: volt x10
  0d02: ?
  0e01: on/off
  0f03: ?
  1001: valve %
  1102: temperature setpoint
  1201: ? current heating mode
  1301: ?
  1c01: motion
  1d01: ? target heating mode
  1f03: ?
  2302: ? battery voltage
  2808: ?

[NorthernMan54]

Erik, tks for this. I'm going to take this into account and adjust as needed

[simont77]

@simont77 @bwp91 @ebaauw While I still have the patch open, anything else that needs taken care of?

I fixed the new Buffer deprecation issue, and an empty data issue for “energy” as well.

Seems ok to me. Which was the empty data issue?

[bwp91]

I am very excited for this! :)

[NorthernMan54]

@simont77 The empty data issue was caused by having the global timer running but no value initial loaded, so it was putting to the persist file only a time record.

Looking at the code you had fixed it for weather, and was copying it to energy.

[An00bIS47]

hi @ebaauw

You have suggested the following:

You could probably simplify the 116 value to 02 0702 0e01 and post entries 0c %s %s 03 %s %s for power and on/off (in that order) and 0b %s %s 02 %s for only power.

0c = Length of this entry %s = 4 Bytes: entry # %s = 4 Bytes: timestamp 03 = 1 Byte: Number of bytes to follow <- is this correct ? %s = 2 Bytes: power consumption %s = 1 Byte: on-off state

If I understand you correct, the accessoryType117 is only the number of bytes of the sensor data fields of a history entry?

In the current implementation for the weather-type the values are specified like this:

this.accessoryType116 = "03 0102 0202 0302";
this.accessoryType117 = "07";

Shouldn't the this.accessoryType117 be 06 for the weather-type instead?

I have tested this with my own implementation (which is on an ESP32 on not using this implementation) and I can confirm, that this works as well but I don't have an EVE device which exposes a weather type to test myself why the value is here a 07.

Thank you :)

[An00bIS47]

If I understand you correct, the accessoryType117 is only the number of bytes of the sensor data fields of a history entry? In the current implementation for the weather-type the values are specified like this: this.accessoryType116 = "03 0102 0202 0302"; this.accessoryType117 = "07"; Shouldn't the this.accessoryType117 be 06 for the weather-type instead? I have tested this with my own implementation (which is on an ESP32 on not using this implementation) and I can confirm, that this works as well but I don't have an EVE device which exposes a weather type to test myself why the value is here a 07.

Please forget this part! After I checked the docs again, this is not correct

[ebaauw]

If I understand you correct, the accessoryType117 is only the number of bytes of the sensor data fields of a history entry?

No, see my previous post:

The 117 value is a bitmap of the values included in the history entry.

[NorthernMan54]

Erik,

Thanks very much for that detailed explanation of the device signature and the 117 bitmap, I was able to create a motion, temperature, humidity, and pressure sensor with graphing in a few minutes. I also tried voltage, 0x0b and 0x0d and no luck.

With this added information about the protocol it raises a dilemma though. The original accessory API developed by @simont77 is based on a static device signature based on existing devices, and this allows a more dynamic approach to accessory definition. So am thinking a refactoring of the API and the internal data management layer may be required, and just expose the signature to end user and allow them to create ( max 8 based on the 117 bitmap ) their own signature, and allow the addEntry function to mimic the 117 message format more closely. At the same time keep the existing API for legacy users. This would negate the need to create an energy 2 ( power and on/off ) or a weather 2 ( temp and motion ). The existing data averaging functionality would require some thought and likely not be usable for the new API

I'm still thinking this thru, but something like this

During initialization, specify the device signature

And when storing history information, have it determine the 117 value programmatically

[ebaauw]

You might want to check out what I’ve done in homebridge-lib. I simply link the characteristic delegates to the history service and need not worry about generating entries at all. I still use a couple of fixed history services, but already with some optional delegates. I suppose you could use an array of delegates, and check the characteristic type.

I’m phasing out the use of this library; I think its footprint is too big. I don’t use the google storage stuff, and it has been the source of security issues in the past. As I mentioned, I’ve had no luck with the global timer. Also there’s no need for moment; it can be handled natively in Javascript. Finally, homebridge-lib stores the history in cachedAccessories instead of in separate files. I still need to refactor Homebridge Hue - that will be a nice stress test to see if history can be persisted in cachedAccessories in bulk.

[An00bIS47]

@ebaauw Thanks for the fast response and sorry! After reading your previous comment again this is clear now. I was a bit confused with the wording „bitmap“ but it is the same as bitmask :)

[NorthernMan54]

I have gone a totally rethought my approach for adding these new graphing options, and instead of creating a custom configuration for each potential configuration went with a dynamic approach instead. To create the 116 device signature, am scanning thru the linked accessory and creating the signature based on the available characteristics within the accessory. So far have added auto discovery for the temperature, humidity, pressure, motion and power characteristics. ( others are possible, just don't have devices available to test them with handy ).

With this approach, the global timer is non functional and averaging must be handled on the plugin side.

For the addEntry object, needed to change the entry naming logic from the terms used today ( ie temp, status etc ) to the HomeKit Characteristic UUID ( Can be the short form to reduce lengths ). Did this to reduce the amount of logic required to be included within fakegato. Usage of the UUID is pretty simple on the plugin side, as the value is part of the actual characteristic object. So addEntry looks like this.

https://github.com/NorthernMan54/homebridge-tasmota/blob/60f9e2b17345fb11f4d168da0158fbbc51febff7/src/tasmotaSensorService.ts#L238

[ebaauw]

I think you could still be using the short names by including them in this.signatures, instead of the uuid. Would probably need to split up status, though.

The missing UUIDs:

• PPM: https://github.com/ebaauw/homebridge-lib/blob/b2c7f4e69d2b4b7ef4b3846c9de4c8b981ef84df/lib/EveHomeKitTypes.js#L161
• Open / Close: ContactSensorState

[NorthernMan54]

@ebaauw I just went back and looked that approach, and it was a nothing change. I think when I started coding it I had already ruled that approach out and focused on UUID. Looking at what the developer will see as the API, it is simple and makes sense.

For status, I split them to motion, contact and status ( on/off ).

[NorthernMan54]

Am thinking that the pull request is ready to go

I also removed the moment dependancy.

@simont77 @ebaauw Any final comments before we can publish ?

[ebaauw]

No further comments from my side. Thanks for ditching moment and correcting the use of new Buffer().

[simont77]

Seems ok to me.

[NorthernMan54]

I'm thinking that with this change is pretty big so we should increment the version, so this would start the 0.6.0 version level? Any thoughts ? I will add this to the pull request.

[NorthernMan54]

And with that @simont77 could you merge and publish on NPM?

[simont77]

Do you think that the changes will break something? I don't think so. The main point is the addition of the custom accessory, and so far you have always bumped the patch version for that.

[simont77]

Please, update also the changelog.

[NorthernMan54]

Changelog is updated

In regards to breaking something, I'm around all weekend ( in the the Toronto time zone ) to deal with any issues. May need to push a patch if something does come up, even though it is not likely.

End user rollback is tricky, as a end user would need to install the previous version from the command line ie

npm install fakegate-history@v0.5.6

But they would need to be in the node_modules directory of the plugin to do it ( needs to be an intermediate or advanced user )

[simont77]

ok, let's keep 0.6.0, which should avoid automatic update if not explicitly requested by the plugin developer. Merging and publishing.

[NorthernMan54]

Tks everyone

🤞 fingers crossed

On Oct 30, 2020, at 1:18 PM, simont77 notifications@github.com wrote:

 ok, let's keep 0.6.0, which should avoid automatic update if not explicitly requested by the plugin developer. Merging and publishing.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or unsubscribe.

[thncode]

Thanks to these descriptions I managed to implement an ESP8266 version of a custom switch/power accessory. The only problem I still do have is that it needs a pull down for updating the switch graph after tapping onto on/off. Is there any other option to force graph update?

[NorthernMan54]

@thncode If you locally switch your device on / off ( from the unit itself ) does HomeKit update in real time ?

ie does you device support HomeKit Notifications ( Section 6.8 of the HAP Spec )?

[thncode]

I can only do that indirectly: I made two web based buttons (via web server on the device) and switched on/off. (1) All the other characteristics do recognize this instantly, (2) the switch cha + graph is ignoring it till I pull down the screen. (1) lets me assume that notifications are managed correctly...

What kind of characteristic is this first entry, if not HAP 9.70?