Found a new endpoint, but it returns a binary file

vloschiavo / powerwall2

Tesla Powerwall 2 - Local Gateway API documentation

Apache License 2.0

288 stars 51 forks source link

Found a new endpoint, but it returns a binary file #51

Closed darryllee closed 1 year ago

darryllee commented 3 years ago

https://teg/api/devices/vitals

Good old strings returns some interesting tidbits:

THC_STATE_AUTONOMOUSCONTROL
THC_AmbientTemp!

I'd love to get temperature info from my gateway/inverter/battery since I was having problems with my brand-new Tesla 7.6kW inverter because it was installed with low coolant, and until it got topped off last week, was throwing "Inverter Over Temperature" alerts (which after a recent update, were only visible after logging on as Installers instead of Customers. Super-shady.)

vloschiavo commented 3 years ago

Nice find! (And nice work on your project https://github.com/darryllee/teslog).

Feel free to issue a pull request with what you know so far, I'm sure the rest of the community will be interested and can help learn more.

Thanks!

darryllee commented 3 years ago

DOH, I just saw you already published a bash script that's far more elegant than mine. I will create a PR for the vitals endpoint I discovered. :-}

vloschiavo commented 3 years ago

I'm taking a look at the binary file output on my powerwall now. It looks like the various serial numbers of the gateway and the battery units are plastered all over this file. So I suggest you scrub those out before posting an example.

I don't know if there is some special encoding or not yet. But I suspect that these are key-value pairs and probably some standard format from Redis or something similar. I'll put into a hex editor and see what I can reverse engineer.

example: I'm looking at the output in VIM:

Line 24: ^OTHC_AmbientTemp!^@^@^@^@^@^@=@

THOC_AmbientTemp is likely the key, and the value is encoded in the hexidecimal value here: ^@^@^@^@^@^@=@ ; I'll need to see the hex values vs the ASCII translation to see if the numbers make sense.

These would be interesting to have the values too: 37 ^XPOD_nom_energy_remaining!^@^@^@^@^@~Á@^R# 38 ^XPOD_nom_full_pack_energy!^@^@^@^@^@×Ê@^R% 39 ^ZPOD_available_charge_power!^@^@^@^@^@X»@^R% 40 ^ZPOD_available_dischg_power!^@^@^@^@^@X»@^R% 41 ^ZPOD_curtailed_charge_power!^@^@^@^@^@^@^@^@^R% 42 ^ZPOD_curtailed_dischgpower!^@^@^@^@^@^@^@^@^R^

Also, setting up a "charge completed" notification would be useful in my automation to then enable charging for the car on those days when I'm making more solar than I can use/store in the powerwall. But it looks like these are broken out for each powerwall (if you have multiples on the same gatway).

Powerwall 1: Line 50: 50 ^RPOD_ChargeComplete0^@^R^Y

Others: 51 ^UPOD_DischargeComplete0^@^R^[ 52 ^WPOD_PersistentlyFaulted0^@^R^Z 53 ^VPOD_PermanentlyFaulted0^@^R^U 54 ^QPOD_ChargeRequest0^@^R^U 55 ^QPOD_ActiveHeating0^@^R^O 56 ^KPOD_Resting0^@^R^O 57 ^KPOD_CCVhold0^@^R^Z 58 ^VPOD_HeatZeroCurrentReg0^@^R^\ 59 ^XPOD_ExtendedCapacityMode0^@^R^Y 60 ^UPOD_IpcRequestedBusUp0^@^R^Y 61 ^UPOD_SelfTestRequested0^A

Powerwall 2: 134 ^RPOD_ChargeComplete0^@^R^Y 135 ^UPOD_DischargeComplete0^@^R^[ 136 ^WPOD_PersistentlyFaulted0^@^R^Z 137 ^VPOD_PermanentlyFaulted0^@^R^U 138 ^QPOD_ChargeRequest0^@^R^U 139 ^QPOD_ActiveHeating0^@^R^O 140 ^KPOD_Resting0^@^R^O 141 ^KPOD_CCVhold0^@^R^Z 142 ^VPOD_HeatZeroCurrentReg0^@^R^\ 143 ^XPOD_ExtendedCapacityMode0^@^R^Y 144 ^UPOD_IpcRequestedBusUp0^@^R^Y 145 ^UPOD_SelfTestRequested0^A

This is also interesting: 172 ^SPINV_MainRelayState^LRELAY_CLOSED^R& 173 ^VPINV_NeutralRelayState^LRELAY_CLOSED^R^V

darryllee commented 3 years ago

Ha yes, I just created a PR without including my vitals file because of it contained my serial numbers, and I didn't feel like loading up a hex editor and scrubbing that info.

But yes, tons of very interesting info that I've not seen in other endpoints.

vloschiavo commented 3 years ago

Not sure if I'm going the right direction but I'm looking at this: THC_AmbientTemp!......<@

0x0A (Hex value 0A) plus another hex byte always precedes each Value. (maybe 0x0A indicates the end of the line?)

The ! = 0x21 - This looks like a the delimiter for the end of each of the keys. The @ = 0x40 - This also looks like a delimiter as well.

So the hex in between those delimiters should be the value. In my case I see the following HEX as the value: 00 00 00 00 00 80 3C

0x80 = 128 0x3C = 60

Does this mean 128.60 °F ?

Seems a bit hot. My ambient outside of the chassis is 80.24 °F.

Maybe 60 ° C?

What are your hex values there?

vloschiavo commented 3 years ago

Ha yes, I just created a PR without including my vitals file because of it contained my serial numbers, and I didn't feel like loading up a hex editor and scrubbing that info.

But yes, tons of very interesting info that I've not seen in other endpoints.

Merged.

darryllee commented 3 years ago

Here's a bit of data between THC_AmbientTemp! up to " TEPOD:

At 2:36PM (ambient temp 88ºF) 00 00 00 00 40 42 40 0A EF 06 0A 97 01 0A 94 01 0A

At 6:06PM (82ºF) CE CC CC CC CC 4C 42 40 0A 80 07 0A 97 01 0A 94 01 0A

At 6:26PM (80ºF) 9A 99 99 99 99 19 42 40 0A 80 07 0A 97 01 0A 94 01 0A

So this seems consistent for end of data: 0A 97 01 0A 94 01 0A

darryllee commented 3 years ago

From user power.saver on the teslamotorclub.com forums:

The binary data after the text strings is the value, in 64-bit IEEE-754 Floating Point notation. There is a ! character separating the text from the number, and the 8 bytes are in little-endian format.

For example, in the vitals file is the text: THC_AmbientTemp!

This is followed by 8 bytes (shown here as hexadecimal digits): CE CC CC CC CC CC 40 40

Here they are in big-endian format (again shown as hexadecimal digits): 40 40 CC CC CC CC CC CE

Those 8 bytes comprise the 64-bits of the IEEE-754 Floating Point number.

The decimal equivalent of that is 33.6 which is the temperature in Celsius. (92.5°F)

Text fields that are followed by a * character have a text value after them (enumeration).

The object: THC_State* has the value: THC_STATE_AUTONOMOUSCONTROL

An interesting one in this file is: POD_DC_Bus_Voltage! which has a value of 428.1 volts.

sean-user commented 3 years ago

I'm power.saver on the TMC forums. Thanks to @darryllee for bringing my post over here.

Some additional info on the objects with after them (text enumerations). The first byte after the character is the length of the text string, followed by the actual text. That wasn't explained in my post on the TMC forums.

sean-user commented 3 years ago

With a little work, I was able to parse out most of the vitals file. Here is the text result. The first column is the number of bytes into the file where the parser found the start of an object. This will probably vary since differences in the text string lengths affects the file size.

A new delimiter was found: if a 0 (ascii zero) appears after the object then the value is TRUE or FALSE, and is represented by a binary 1 or 0 in the next byte. I show TRUE or FALSE when this occurs.

0031 : 1118431-00-J = 
00b4 : 1092170-03-G = 
0128 : THC_State = THC_STATE_AUTONOMOUSCONTROL
0152 : THC_AmbientTemp = 33.600000
019b : 1081100-79-J = 
020b : POD_nom_energy_to_be_charged = 2756.000000
0234 : POD_nom_energy_remaining = 11115.000000
0259 : POD_nom_full_pack_energy = 13707.000000
027e : POD_available_charge_power = 7000.000000
02a5 : POD_available_dischg_power = 7000.000000
02cc : POD_curtailed_charge_power = 0.000000
02f3 : POD_curtailed_dischg_power = 0.000000
031a : POD_est_DC_Bus_Power = -2320.000000
033b : POD_Ibatt = -50.600000
0351 : POD_Vbatt = 46.600000
0367 : POD_DC_Bus_Voltage = 428.100000
0386 : POD_state = POD_ACTIVE
039f : POD_enable_line = TRUE
03b4 : POD_ChargeComplete = FALSE
03cc : POD_DischargeComplete = FALSE
03e7 : POD_PersistentlyFaulted = FALSE
0404 : POD_PermanentlyFaulted = FALSE
0420 : POD_ChargeRequest = FALSE
0437 : POD_ActiveHeating = FALSE
044e : POD_Resting = FALSE
045f : POD_CCVhold = FALSE
0470 : POD_HeatZeroCurrentReg = FALSE
048c : POD_ExtendedCapacityMode = FALSE
04aa : POD_IpcRequestedBusUp = FALSE
04c5 : POD_SelfTestRequested = TRUE
050e : 1081100-79-J = 
057e : PINV_EnergyDischarged = 4173130.000000
05a0 : PINV_EnergyCharged = 4827370.000000
05bf : PINV_VSplit1 = 123.300000
05d8 : PINV_VSplit2 = 123.200000
05f1 : PINV_PllFrequency = 59.995000
060f : PINV_PllLocked = TRUE
0623 : PINV_Pout = 2.320000
0639 : PINV_Qout = -0.020000
064f : PINV_Vout = 246.500000
0665 : PINV_Fout = 60.011000
067b : PINV_ReadyForGridForming = TRUE
0699 : PINV_State = PINV_GridFollowing
06bb : PINV_GridState = Grid_Compliant
06dd : PINV_MainRelayState = RELAY_CLOSED
0702 : PINV_NeutralRelayState = RELAY_CLOSED
072a : PINV_Iavail = 32.400000
0742 : PINV_Savail = 7.990000
075a : PINV_Vdc = 428.100000
076f : PINV_HardwareEnableLine = TRUE
078c : PINV_PowerLimiter = PWRLIM_No_Power_Limit
080d : 1092170-03-G = 
0881 : THC_State = THC_STATE_AUTONOMOUSCONTROL
08ab : THC_AmbientTemp = 29.100000
08f4 : 1081100-59-J = 
0964 : POD_nom_energy_to_be_charged = 2758.000000
098d : POD_nom_energy_remaining = 11207.000000
09b2 : POD_nom_full_pack_energy = 13810.000000
09d7 : POD_available_charge_power = 7000.000000
09fe : POD_available_dischg_power = 7000.000000
0a25 : POD_curtailed_charge_power = 0.000000
0a4c : POD_curtailed_dischg_power = 0.000000
0a73 : POD_est_DC_Bus_Power = -2350.000000
0a94 : POD_Ibatt = -51.200000
0aaa : POD_Vbatt = 46.600000
0ac0 : POD_DC_Bus_Voltage = 428.000000
0adf : POD_state = POD_ACTIVE
0af8 : POD_enable_line = TRUE
0b0d : POD_ChargeComplete = FALSE
0b25 : POD_DischargeComplete = FALSE
0b40 : POD_PersistentlyFaulted = FALSE
0b5d : POD_PermanentlyFaulted = FALSE
0b79 : POD_ChargeRequest = FALSE
0b90 : POD_ActiveHeating = FALSE
0ba7 : POD_Resting = FALSE
0bb8 : POD_CCVhold = FALSE
0bc9 : POD_HeatZeroCurrentReg = FALSE
0be5 : POD_ExtendedCapacityMode = FALSE
0c03 : POD_IpcRequestedBusUp = FALSE
0c1e : POD_SelfTestRequested = TRUE
0c67 : 1081100-59-J = 
0cd7 : PINV_EnergyDischarged = 4179150.000000
0cf9 : PINV_EnergyCharged = 4857010.000000
0d18 : PINV_VSplit1 = 123.400000
0d31 : PINV_VSplit2 = 123.100000
0d4a : PINV_PllFrequency = 60.007000
0d68 : PINV_PllLocked = TRUE
0d7c : PINV_Pout = 2.330000
0d92 : PINV_Qout = -0.030000
0da8 : PINV_Vout = 246.600000
0dbe : PINV_Fout = 60.013000
0dd4 : PINV_ReadyForGridForming = TRUE
0df2 : PINV_State = PINV_GridFollowing
0e14 : PINV_GridState = Grid_Compliant
0e36 : PINV_MainRelayState = RELAY_CLOSED
0e5b : PINV_NeutralRelayState = RELAY_CLOSED
0e83 : PINV_Iavail = 32.400000
0e9b : PINV_Savail = 7.980000
0eb3 : PINV_Vdc = 428.100000
0ec8 : PINV_HardwareEnableLine = TRUE
0ee5 : PINV_PowerLimiter = PWRLIM_No_Power_Limit
0f67 : 1118431-00-J = 
0fdb : ISLAND_VL1N_Main = 122.500000
0ff8 : ISLAND_FreqL1_Main = 60.010000
1017 : ISLAND_VL1N_Load = 122.500000
1034 : ISLAND_FreqL1_Load = 60.010000
1053 : ISLAND_PhaseL1_Main_Load = 0.000000
1078 : ISLAND_VL2N_Main = 121.000000
1095 : ISLAND_FreqL2_Main = 60.010000
10b4 : ISLAND_VL2N_Load = 121.000000
10d1 : ISLAND_FreqL2_Load = 60.010000
10f0 : ISLAND_PhaseL2_Main_Load = 0.000000
1115 : ISLAND_VL3N_Main = 0.000000
1132 : ISLAND_FreqL3_Main = 0.000000
1151 : ISLAND_VL3N_Load = 0.000000
116e : ISLAND_FreqL3_Load = 0.000000
118d : ISLAND_PhaseL3_Main_Load = -256.000000
11b2 : ISLAND_L1L2PhaseDelta = -256.000000
11d4 : ISLAND_L1L3PhaseDelta = -256.000000
11f6 : ISLAND_L2L3PhaseDelta = -256.000000
1218 : ISLAND_GridState = ISLAND_GridState_Grid_Compliant
124d : ISLAND_L1MicrogridOk = TRUE
1267 : ISLAND_L2MicrogridOk = TRUE
1281 : ISLAND_L3MicrogridOk = FALSE
129b : ISLAND_ReadyForSynchronization = TRUE
12bf : ISLAND_GridConnected = TRUE
12d9 : SYNC_ExternallyPowered = FALSE
12f5 : SYNC_SiteSwitchEnabled = TRUE

fkhera commented 3 years ago

Unsubscribe

On Sat, Aug 7, 2021 at 9:47 PM sean-user @.***> wrote:

With a little work, I was able to parse out most of the vitals file. Here is the text result. The first column is the number of bytes into the file where the parser found the start of an object. This will probably vary since differences in the text string lengths affects the file size.

A new delimiter was found: if a 0 (ascii zero) appears after the object then the value is TRUE or FALSE, and is represented by a binary 1 or 0 in the next byte. I show TRUE or FALSE when this occurs.

0031 : 1118431-00-J = 00b4 : 1092170-03-G = 0128 : THC_State = THC_STATE_AUTONOMOUSCONTROL 0152 : THC_AmbientTemp = 33.600000 019b : 1081100-79-J = 020b : POD_nom_energy_to_be_charged = 2756.000000 0234 : POD_nom_energy_remaining = 11115.000000 0259 : POD_nom_full_pack_energy = 13707.000000 027e : POD_available_charge_power = 7000.000000 02a5 : POD_available_dischg_power = 7000.000000 02cc : POD_curtailed_charge_power = 0.000000 02f3 : POD_curtailed_dischg_power = 0.000000 031a : POD_est_DC_Bus_Power = -2320.000000 033b : POD_Ibatt = -50.600000 0351 : POD_Vbatt = 46.600000 0367 : POD_DC_Bus_Voltage = 428.100000 0386 : POD_state = POD_ACTIVE 039f : POD_enable_line = TRUE 03b4 : POD_ChargeComplete = FALSE 03cc : POD_DischargeComplete = FALSE 03e7 : POD_PersistentlyFaulted = FALSE 0404 : POD_PermanentlyFaulted = FALSE 0420 : POD_ChargeRequest = FALSE 0437 : POD_ActiveHeating = FALSE 044e : POD_Resting = FALSE 045f : POD_CCVhold = FALSE 0470 : POD_HeatZeroCurrentReg = FALSE 048c : POD_ExtendedCapacityMode = FALSE 04aa : POD_IpcRequestedBusUp = FALSE 04c5 : POD_SelfTestRequested = TRUE 050e : 1081100-79-J = 057e : PINV_EnergyDischarged = 4173130.000000 05a0 : PINV_EnergyCharged = 4827370.000000 05bf : PINV_VSplit1 = 123.300000 05d8 : PINV_VSplit2 = 123.200000 05f1 : PINV_PllFrequency = 59.995000 060f : PINV_PllLocked = TRUE 0623 : PINV_Pout = 2.320000 0639 : PINV_Qout = -0.020000 064f : PINV_Vout = 246.500000 0665 : PINV_Fout = 60.011000 067b : PINV_ReadyForGridForming = TRUE 0699 : PINV_State = PINV_GridFollowing 06bb : PINV_GridState = Grid_Compliant 06dd : PINV_MainRelayState = RELAY_CLOSED 0702 : PINV_NeutralRelayState = RELAY_CLOSED 072a : PINV_Iavail = 32.400000 0742 : PINV_Savail = 7.990000 075a : PINV_Vdc = 428.100000 076f : PINV_HardwareEnableLine = TRUE 078c : PINV_PowerLimiter = PWRLIM_No_Power_Limit 080d : 1092170-03-G = 0881 : THC_State = THC_STATE_AUTONOMOUSCONTROL 08ab : THC_AmbientTemp = 29.100000 08f4 : 1081100-59-J = 0964 : POD_nom_energy_to_be_charged = 2758.000000 098d : POD_nom_energy_remaining = 11207.000000 09b2 : POD_nom_full_pack_energy = 13810.000000 09d7 : POD_available_charge_power = 7000.000000 09fe : POD_available_dischg_power = 7000.000000 0a25 : POD_curtailed_charge_power = 0.000000 0a4c : POD_curtailed_dischg_power = 0.000000 0a73 : POD_est_DC_Bus_Power = -2350.000000 0a94 : POD_Ibatt = -51.200000 0aaa : POD_Vbatt = 46.600000 0ac0 : POD_DC_Bus_Voltage = 428.000000 0adf : POD_state = POD_ACTIVE 0af8 : POD_enable_line = TRUE 0b0d : POD_ChargeComplete = FALSE 0b25 : POD_DischargeComplete = FALSE 0b40 : POD_PersistentlyFaulted = FALSE 0b5d : POD_PermanentlyFaulted = FALSE 0b79 : POD_ChargeRequest = FALSE 0b90 : POD_ActiveHeating = FALSE 0ba7 : POD_Resting = FALSE 0bb8 : POD_CCVhold = FALSE 0bc9 : POD_HeatZeroCurrentReg = FALSE 0be5 : POD_ExtendedCapacityMode = FALSE 0c03 : POD_IpcRequestedBusUp = FALSE 0c1e : POD_SelfTestRequested = TRUE 0c67 : 1081100-59-J = 0cd7 : PINV_EnergyDischarged = 4179150.000000 0cf9 : PINV_EnergyCharged = 4857010.000000 0d18 : PINV_VSplit1 = 123.400000 0d31 : PINV_VSplit2 = 123.100000 0d4a : PINV_PllFrequency = 60.007000 0d68 : PINV_PllLocked = TRUE 0d7c : PINV_Pout = 2.330000 0d92 : PINV_Qout = -0.030000 0da8 : PINV_Vout = 246.600000 0dbe : PINV_Fout = 60.013000 0dd4 : PINV_ReadyForGridForming = TRUE 0df2 : PINV_State = PINV_GridFollowing 0e14 : PINV_GridState = Grid_Compliant 0e36 : PINV_MainRelayState = RELAY_CLOSED 0e5b : PINV_NeutralRelayState = RELAY_CLOSED 0e83 : PINV_Iavail = 32.400000 0e9b : PINV_Savail = 7.980000 0eb3 : PINV_Vdc = 428.100000 0ec8 : PINV_HardwareEnableLine = TRUE 0ee5 : PINV_PowerLimiter = PWRLIM_No_Power_Limit 0f67 : 1118431-00-J = 0fdb : ISLAND_VL1N_Main = 122.500000 0ff8 : ISLAND_FreqL1_Main = 60.010000 1017 : ISLAND_VL1N_Load = 122.500000 1034 : ISLAND_FreqL1_Load = 60.010000 1053 : ISLAND_PhaseL1_Main_Load = 0.000000 1078 : ISLAND_VL2N_Main = 121.000000 1095 : ISLAND_FreqL2_Main = 60.010000 10b4 : ISLAND_VL2N_Load = 121.000000 10d1 : ISLAND_FreqL2_Load = 60.010000 10f0 : ISLAND_PhaseL2_Main_Load = 0.000000 1115 : ISLAND_VL3N_Main = 0.000000 1132 : ISLAND_FreqL3_Main = 0.000000 1151 : ISLAND_VL3N_Load = 0.000000 116e : ISLAND_FreqL3_Load = 0.000000 118d : ISLAND_PhaseL3_Main_Load = -256.000000 11b2 : ISLAND_L1L2PhaseDelta = -256.000000 11d4 : ISLAND_L1L3PhaseDelta = -256.000000 11f6 : ISLAND_L2L3PhaseDelta = -256.000000 1218 : ISLAND_GridState = ISLAND_GridState_Grid_Compliant 124d : ISLAND_L1MicrogridOk = TRUE 1267 : ISLAND_L2MicrogridOk = TRUE 1281 : ISLAND_L3MicrogridOk = FALSE 129b : ISLAND_ReadyForSynchronization = TRUE 12bf : ISLAND_GridConnected = TRUE 12d9 : SYNC_ExternallyPowered = FALSE 12f5 : SYNC_SiteSwitchEnabled = TRUE

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://urldefense.com/v3/__https://github.com/vloschiavo/powerwall2/issues/51*issuecomment-894742452__;Iw!!IKRxdwAv5BmarQ!MX696kmbCCiWOdDVri9n4kbcZNAfTDfRuzw6YSf3m6zBFGBGfo2OsV01g5ngPg$, or unsubscribe https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ABU4JYCDTKZRP35PQGK7GHDT3YD4NANCNFSM5BVJR2CQ__;!!IKRxdwAv5BmarQ!MX696kmbCCiWOdDVri9n4kbcZNAfTDfRuzw6YSf3m6zBFGBGfo2OsV3uEC3dQw$ . Triage notifications on the go with GitHub Mobile for iOS https://urldefense.com/v3/__https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675__;!!IKRxdwAv5BmarQ!MX696kmbCCiWOdDVri9n4kbcZNAfTDfRuzw6YSf3m6zBFGBGfo2OsV0M-v6DlA$ or Android https://urldefense.com/v3/__https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email__;!!IKRxdwAv5BmarQ!MX696kmbCCiWOdDVri9n4kbcZNAfTDfRuzw6YSf3m6zBFGBGfo2OsV3ihpVmiQ$ .

vloschiavo commented 3 years ago

Nice work everyone! Thanks!

Is anyone up for creating a sample decoder script?

darryllee commented 3 years ago

Got new firmware last night (21.31.0), and noticed that the vitals files that I've just been dumping (every 5 min) into a directory was consistently slightly bigger.

I did a diff -a and eyeballing it, this is the only thing that I could see that is "new":

PVS_a031_Mci3PvVoltage
PVS_a032_Mci4PvVoltage

That's "below"(?) a previously undocumented PVS section.

Full dump. Before:

        PVS_State*
PVS_Active^R$
^QPVS_SelfTestState*^OPVS_SelfTestOff^R^T
^PPVS_EnableOutput0^A^R^Y
^UPVS_StringA_Connected0^A^R^Y
^UPVS_StringB_Connected0^A^R^Y
^UPVS_StringC_Connected0^@^R^Y
^UPVS_StringD_Connected0^@^Z^SPVS_a019_MciStringC^Z^SPVS_a020_MciStringD

After:

        PVS_State*
PVS_Active^R$
^QPVS_SelfTestState*^OPVS_SelfTestOff^R^T
^PPVS_EnableOutput0^A^R^Y
^UPVS_StringA_Connected0^A^R^Y
^UPVS_StringB_Connected0^A^R^Y
^UPVS_StringC_Connected0^@^R^Y
^UPVS_StringD_Connected0^@^Z^SPVS_a019_MciStringC^Z^SPVS_a020_MciStringD^Z^VPVS_a031_Mci3PvVoltage^Z^VPVS_a032_Mci4PvVoltage

Yeah... a decoder would be nice. :-}

sean-user commented 3 years ago

I've attached the parser I wrote in C. Sample output file included as well.

vitals.zip

darryllee commented 3 years ago

Awesome! To get it to build on my Raspberry Pi I just had to remove:

#include <conio.h>

And flip the slashes here:

#include <sys/types.h>
#include <sys/stat.h>

Then I ran it (so the input file has to be named vitals. Got it.)

Do you think those are new delimiters for the PVS... MciString values that your code isn't detecting? As I mentioned, I only started seeing them in this latest 21.31.0 firmware.

sean-user commented 3 years ago

I don't have 21.31.0 yet, so can't check the file, but all others were preceded by 0x12 0x?? 0x0a before the key. 0x?? is any value. So if it follows that format, it should get parsed out. I read all the way to the end of the file. Yes, the input file is called vitals since that's what the PW downloaded, but you could change that in the code if you like.

darryllee commented 3 years ago

Would a hexdump help?

00000da0: 0a15 5056 535f 5374 7269 6e67 435f 436f  ..PVS_StringC_Co
00000db0: 6e6e 6563 7465 6430 0012 190a 1550 5653  nnected0.....PVS
00000dc0: 5f53 7472 696e 6744 5f43 6f6e 6e65 6374  _StringD_Connect
00000dd0: 6564 3000 1a13 5056 535f 6130 3139 5f4d  ed0...PVS_a019_M
00000de0: 6369 5374 7269 6e67 431a 1350 5653 5f61  ciStringC..PVS_a
00000df0: 3032 305f 4d63 6953 7472 696e 6744 1a16  020_MciStringD..
00000e00: 5056 535f 6130 3331 5f4d 6369 3350 7656  PVS_a031_Mci3PvV
00000e10: 6f6c 7461 6765 1a16 5056 535f 6130 3332  oltage..PVS_a032
00000e20: 5f4d 6369 3450 7656 6f6c 7461 6765 0aa5  _Mci4PvVoltage..

sean-user commented 3 years ago

The first one is cut off, but the the second one, starting at 0db9 = 12, 0dba = 19 (don't care) and 0dbc = 0a so the parser should find it. The trailing character is "0" so this is a TRUE/FALSE key, and the value is 00 (FALSE). But the next one is different, there are no starting bytes and no trailing delimiter, so the parser won't find that this one (PVS_a019_MciStringC) or the rest of them. Looks like they may be a list with no values?

darryllee commented 3 years ago

Sorry, yeah, I cut it off too soon:

00000d10: 396e 4012 170a 0950 5653 5f53 7461 7465  9n@....PVS_State
00000d20: 2a0a 5056 535f 4163 7469 7665 1224 0a11  *.PVS_Active.$..
00000d30: 5056 535f 5365 6c66 5465 7374 5374 6174  PVS_SelfTestStat
00000d40: 652a 0f50 5653 5f53 656c 6654 6573 744f  e*.PVS_SelfTestO
00000d50: 6666 1214 0a10 5056 535f 456e 6162 6c65  ff....PVS_Enable
00000d60: 4f75 7470 7574 3001 1219 0a15 5056 535f  Output0.....PVS_
00000d70: 5374 7269 6e67 415f 436f 6e6e 6563 7465  StringA_Connecte
00000d80: 6430 0112 190a 1550 5653 5f53 7472 696e  d0.....PVS_Strin
00000d90: 6742 5f43 6f6e 6e65 6374 6564 3001 1219  gB_Connected0...
00000da0: 0a15 5056 535f 5374 7269 6e67 435f 436f  ..PVS_StringC_Co
00000db0: 6e6e 6563 7465 6430 0012 190a 1550 5653  nnected0.....PVS
00000dc0: 5f53 7472 696e 6744 5f43 6f6e 6e65 6374  _StringD_Connect
00000dd0: 6564 3000 1a13 5056 535f 6130 3139 5f4d  ed0...PVS_a019_M
00000de0: 6369 5374 7269 6e67 431a 1350 5653 5f61  ciStringC..PVS_a
00000df0: 3032 305f 4d63 6953 7472 696e 6744 1a16  020_MciStringD..
00000e00: 5056 535f 6130 3331 5f4d 6369 3350 7656  PVS_a031_Mci3PvV
00000e10: 6f6c 7461 6765 1a16 5056 535f 6130 3332  oltage..PVS_a032
00000e20: 5f4d 6369 3450 7656 6f6c 7461 6765 0aa5  _Mci4PvVoltage..
00000e30: 0f0a 9f01 0a9c 010a 260a 2454 4553 594e  ........&.$TESYN

If I'm reading this right, I think PVS_a031_Mci3PvVoltage and PVS_a031_Mci3PvVoltage both start with: 1a16, and well, then there's a bunch of data after that that I cut off in my last post. Sorry about that.

sean-user commented 3 years ago

The 16 is the string length (22 characters). The 1a is unknown because it immediately follows the previous key, which should have been a delimiter and a value, but they are not there. It's possible they added these new keys without fully implementing them, since no values appear after them like all the others. Or they are just text which is part of PVS_StringD_Connected.

wiedmann commented 3 years ago

I'll just add here that putting my output through a protobuf decoder yields something at least half-way sensible, so this may be a serialization format based on a similar principle. Try this: https://protogen.marcgravell.com/decode

The strings and varints seem to decode correctly, but the fixed64 values don't appear correct, so it's probably some other format. In terms of unpacking the data, though, this might be helpful.

I'm still on version 21.20.6, FWIW.

brianhealey commented 3 years ago

Not perfect.. but close. It's definitely protobuf.

I am able to parse my file using this but I still have a few defects.

The "root" is the DevicesWithVitals entity which contains multiple devices. The c script above was dumping the aggregates across multiple devices.

@sean-user

brianhealey commented 3 years ago

`syntax = "proto3"; option go_package = "github.com/example/path/gen;gen"; package teslapower;

import "google/protobuf/timestamp.proto";

message DeviceVital { optional string name = 1; oneof value { int64 intValue = 3; double floatValue = 4; string stringValue = 5; bool boolValue = 6; } }

message StringValue { string value = 1; }

message UInt32Value { uint32 value = 1; }

message ConnectionParameters { optional string ipAddress = 1; optional string serialPort = 2; optional string serialBaud = 3; optional uint32 modbusId = 4; }

message TeslaHardwareId { optional UInt32Value pcbaId = 1; optional UInt32Value assemblyId = 2; optional UInt32Value usageId = 3;

}

message TeslaEnergyEcuAttributes { optional int32 ecuType = 1; optional TeslaHardwareId hardwareId = 2;

}

message GeneratorAttributes { optional uint64 nameplateRealPowerW = 1; optional uint64 nameplateApparentPowerVa = 2; }

message PVInverterAttributes { optional uint64 nameplateRealPowerW = 1; }

message MeterAttributes { repeated uint32 meterLocation = 1; }

message DeviceAttributes { oneof deviceAttributes { TeslaEnergyEcuAttributes teslaEnergyEcuAttributes = 1; GeneratorAttributes generatorAttributes = 2; PVInverterAttributes pvInverterAttributes = 3; MeterAttributes meterAttributes = 4; }; }

message Device { optional StringValue din = 1; optional StringValue partNumber = 2; optional StringValue serialNumber = 3; optional StringValue manufacturer = 4; optional StringValue siteLabel = 5; optional StringValue componentParentDin = 6; optional StringValue firmwareVersion = 7; optional google.protobuf.Timestamp firstCommunicationTime = 8; optional google.protobuf.Timestamp lastCommunicationTime = 9; optional ConnectionParameters connectionParameters = 10; repeated DeviceAttributes deviceAttributes = 11; }

message SiteControllerConnectedDevice { optional Device device = 1; }

message SiteControllerConnectedDeviceWithVitals { repeated SiteControllerConnectedDevice device = 1; repeated DeviceVital vitals = 2; repeated string alerts = 3; }

message DevicesWithVitals { repeated SiteControllerConnectedDeviceWithVitals devices = 1; }`

fkhera commented 3 years ago

Unsubscribe

On Mon, Sep 20, 2021 at 8:13 PM brianhealey @.***> wrote:

`syntax = "proto3"; option go_package = "github.com/example/path/gen;gen"; package teslapower;

import "google/protobuf/timestamp.proto";

message DeviceVital { optional string name = 1; oneof value { int64 intValue = 3; double floatValue = 4; string stringValue = 5; bool boolValue = 6; } }

message StringValue { string value = 1; }

message UInt32Value { uint32 value = 1; }

message ConnectionParameters { optional string ipAddress = 1; optional string serialPort = 2; optional string serialBaud = 3; optional uint32 modbusId = 4; }

message TeslaHardwareId { optional UInt32Value pcbaId = 1; optional UInt32Value assemblyId = 2; optional UInt32Value usageId = 3;

}

message TeslaEnergyEcuAttributes { optional int32 ecuType = 1; optional TeslaHardwareId hardwareId = 2;

}

message GeneratorAttributes { optional uint64 nameplateRealPowerW = 1; optional uint64 nameplateApparentPowerVa = 2; }

message PVInverterAttributes { optional uint64 nameplateRealPowerW = 1; }

message MeterAttributes { repeated uint32 meterLocation = 1; }

message DeviceAttributes { oneof deviceAttributes { TeslaEnergyEcuAttributes teslaEnergyEcuAttributes = 1; GeneratorAttributes generatorAttributes = 2; PVInverterAttributes pvInverterAttributes = 3; MeterAttributes meterAttributes = 4; }; }

message Device { optional StringValue din = 1; optional StringValue partNumber = 2; optional StringValue serialNumber = 3; optional StringValue manufacturer = 4; optional StringValue siteLabel = 5; optional StringValue componentParentDin = 6; optional StringValue firmwareVersion = 7; optional google.protobuf.Timestamp firstCommunicationTime = 8; optional google.protobuf.Timestamp lastCommunicationTime = 9; optional ConnectionParameters connectionParameters = 10; repeated DeviceAttributes deviceAttributes = 11; }

message SiteControllerConnectedDevice { optional Device device = 1; }

message SiteControllerConnectedDeviceWithVitals { repeated SiteControllerConnectedDevice device = 1; repeated DeviceVital vitals = 2; repeated string alerts = 3; }

message DevicesWithVitals { repeated SiteControllerConnectedDeviceWithVitals devices = 1; }`

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://urldefense.com/v3/__https://github.com/vloschiavo/powerwall2/issues/51*issuecomment-923574346__;Iw!!IKRxdwAv5BmarQ!LTuYq4U0-M37z9UhvlyLhI5EqgLPKxRN6t5ooRYeRD7pUn_FKXYz6nePTWtq3Q$, or unsubscribe https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ABU4JYGT53TXE3GNAW4TNJTUC7Z65ANCNFSM5BVJR2CQ__;!!IKRxdwAv5BmarQ!LTuYq4U0-M37z9UhvlyLhI5EqgLPKxRN6t5ooRYeRD7pUn_FKXYz6nfBCvfKRQ$ . Triage notifications on the go with GitHub Mobile for iOS https://urldefense.com/v3/__https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675__;!!IKRxdwAv5BmarQ!LTuYq4U0-M37z9UhvlyLhI5EqgLPKxRN6t5ooRYeRD7pUn_FKXYz6ne6enLNpA$ or Android https://urldefense.com/v3/__https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign*3Dnotification-email*26utm_medium*3Demail*26utm_source*3Dgithub__;JSUlJSU!!IKRxdwAv5BmarQ!LTuYq4U0-M37z9UhvlyLhI5EqgLPKxRN6t5ooRYeRD7pUn_FKXYz6nfTWBXPQA$.

mir0810 commented 3 years ago

I logged into my TEG on its internal wifi with installer access. the "System" page shows all the vital statistics nicely represented. The reason I was in there was to figure out how much electricity each solar string was generating. But it would be nice to parse out the additional values like temp and frequency as well. Since the website shows it clear text, I assume there is a javascript on the page that does the decoding but I haven't looked at it in detail yet :)

jasonacox commented 2 years ago

@sean-user Thanks for your code. I ported a version to python here but it isn't quite right. The protobuf path seems very interesting.

@brianhealey Were you able to parse with that? Any quick tips for someone just learning protobuf? :) I just brew install protobuf and checking out how to get a proto definition into python to parse these vitals payloads. Going through protoc now...

I was able to use protoc to get the raw decode, so this is clearly a protobuf payload:

protoc --decode_raw < vitals.bin

It represents the data in { ... } groupings with number prefixing and key values in hex, strings or 1/0 boolean - example snip:

1 {
  1 {
    1 {
      1 {
        1: "TETHC--1234567-25-E--TG123456789012"
      }
      2 {
        1: "1234567-25-E"
      }
      3 {
        1: "TG123456789012"
      }
      4 {
        1: "TESLA"
      }
      6 {
        1: "STSTSM--1234567-00-E--TG123456789012"
      }
      7 {
        1: "67f943cb05d12d"
      }
      9 {
        1: 1637708577
      }
      11 {
        1 {
          1: 224
        }
      }
    }
  }
  2 {
    1: "THC_State"
    5: "THC_STATE_AUTONOMOUSCONTROL"
  }
  2 {
    1: "THC_AmbientTemp"
    4: 0x403899999999999c
  }
}

I'm particularly interested in getting power data on strings. This shows up in the PVAC section - example:

 2 {
    1: "PVAC_PVMeasuredVoltage_A"
    4: 0x406cb66666666667
  }
  2 {
    1: "PVAC_PVMeasuredVoltage_B"
    4: 0xc003333333333332
  }
  2 {
    1: "PVAC_PVMeasuredVoltage_C"
    4: 0x406c09999999999a
  }
  2 {
    1: "PVAC_PVMeasuredVoltage_D"
    4: 0x406c09999999999a
  }
  2 {
    1: "PVAC_PVMeasuredPower_A"
    4: 0x4069c00000000000
  }
  2 {
    1: "PVAC_PVMeasuredPower_B"
    4: 0x0000000000000000
  }
  2 {
    1: "PVAC_PVMeasuredPower_C"
    4: 0x4078f00000000000
  }
  2 {
    1: "PVAC_PVMeasuredPower_D"
    4: 0x4078b00000000000
  }
  2 {
    1: "PVAC_LifetimeEnergyPV_Total"
    4: 0x4133401600000000
  }

Boolean examples:

 2 {
    1: "ISLAND_L1MicrogridOk"
    6: 1
  }
  2 {
    1: "ISLAND_L2MicrogridOk"
    6: 1
  }
  2 {
    1: "ISLAND_L3MicrogridOk"
    6: 0
  }
  2 {
    1: "ISLAND_ReadyForSynchronization"
    6: 1
  }
  2 {
    1: "ISLAND_GridConnected"
    6: 1
  }

jasonacox commented 2 years ago

I used the great work by @brianhealey (tesla.proto definition) to process the vitals payload and produce a json representation. Work in progress: https://github.com/jasonacox/pypowerwall/tree/main/examples/vitals

UPDATE: I added the protobuf handling to the pypowerwall python library to make it easy to pull the vitals data. My goal was to get the solar panel "String" data so I added a function to parse that data from the vitals payload. Both functions, vitals() and strings() return python dictionary or JSON payloads.

# Install pyPowerwall package
pip install pypowerwall

import pypowerwall

# Credentials for your Powerwall - Customer Login Data
password='password'
email='email@example.com'
host = "localhost"                # Change to the IP of your Powerwall
timezone = "America/Los_Angeles"  # Change to your local timezone/tz

# Connect to Powerwall
pw = pypowerwall.Powerwall(host,password,email,timezone)

# Display Vitals (returns details as dictionary or JSON)
vitals = pw.vitals(jsonformat=False)
print(vitals)

# Display String Data 
strings = pw.strings(jsonformat=False)
print(strings)

reli3 commented 2 years ago

What would be the best way to format the output a bit better?

I am Pre-PTO I think there are issues with my MCI - Maybe once a week / day etc. I have to keep an eye on my strings. Sometimes 1 or maybe 2 strings are not functioning so I have to do a reboot of the inverters to make them show up again.

If getting the fromat correct and comparing it from a baseline everyday if something turns False for a string some type of email notifaction would be amazing for a lot of people I believe when they think there solar is under producing but in the end it could be a MCI problem or a string issue since people complain alot about under performance this could be the reason why!

After executing this is what I get for the Dispaly String Data:

{'A': {'Current': 1.1300000000000001, 'Voltage': 141.20000000000002, 'Power': 173.0, 'State': 'PV_Active', 'Connected': True}, 'B': {'Current': 0.54, 'Voltage': 339.0, 'Power': 178.0, 'State': 'PV_Active', 'Connected': True}, 'C': {'Current': 0.59, 'Voltage': 324.6, 'Power': 197.0, 'State': 'PV_Active', 'Connected': True}, 'D': {'Current': 0.48, 'Voltage': 99.4, 'Power': 64.0, 'State': 'PV_Active', 'Connected': True}, 'A1': {'Current': 1.1, 'Voltage': 302.90000000000003, 'Power': 327.0, 'State': 'PV_Active', 'Connected': True}, 'B1': {'Current': 1.09, 'Voltage': 303.5, 'Power': 320.0, 'State': 'PV_Active_Parallel', 'Connected': True}, 'C1': {'Current': 0.0, 'Voltage': -1.299999999999999, 'Power': 0.0, 'State': 'PV_Active', 'Connected': False}, 'D1': {'Current': 0.0, 'Voltage': -0.29999999999999893, 'Power': 0.0, 'State': 'PV_Active', 'Connected': False}}

jasonacox commented 2 years ago

Hi @reli3 - I want to keep an eye on my strings as well, which is why I wrote this. I also wanted to keep an eye on the Neruio solar meter which is also in the vitals payload. It looks like you have 6 active strings in your system (A, B, C, D, A1, and B1). For string data, if you want to see it in human readable format, just specify jsonformat as True like this:

strings = pw.strings(jsonformat=True)
print(strings)

Output:

{
    "A": {
        "Connected": true,
        "Current": 0.48,
        "Power": 115.0,
        "State": "PV_Active",
        "Voltage": 251.8
    },
    "B": {
        "Connected": false,
        "Current": 0.0,
        "Power": 0.0,
        "State": "PV_Active",
        "Voltage": -2.3999999999999995
    },
    "C": {
        "Connected": true,
        "Current": 1.82,
        "Power": 526.0,
        "State": "PV_Active",
        "Voltage": 293.6
    },
    "D": {
        "Connected": true,
        "Current": 1.81,
        "Power": 521.0,
        "State": "PV_Active_Parallel",
        "Voltage": 294.0
    }
}

I push that into a tiny installation of Splunk (see TinySplunk) and get a graph like this:

I have a lot of trouble with the Neruio solar meter that Tesla installed in my Powerwall+ so I also have scripts to watch that. It shows up in the vitals payload as a device that I monitor and capture to create alerts (e.g. if it doesn't show up in the vitals payload that means it has lost connection with the gateway and shut down solar production):

"NEURIO--VAHxxxxxxxxxx": {
        "NEURIO_CT0_InstRealPower": 1333.5606370817025,
        "NEURIO_CT0_Location": "solar",
        "Parent": "STSTSM--xxxxxxx-00-E--TGxxxxxxxxxxxx",
        "firmwareVersion": "1.6.1-Tesla",
        "lastCommunicationTime": "1638720520",
        "manufacturer": "NEURIO",
        "partNumber": "",
        "serialNumber": "VAHxxxxxxxxxx"
    }

reli3 commented 2 years ago

@jasonacox omg thata is amazing exactly what I think I am looking for....any chance for a write up to set this up via tinysplunk on how you are getting the data in? Does it pull the string data every minute?

This output will help in the morning and ill run it via cron to put to what I need instead of having to login via installer and flipping the powerwall switch when I think its not producing the correct amount of solar as expected!

I loved your other dashboard with grafana was thinking of making a PI display just for that eventually.

I believe I have the NEURIO as well since I have Two Powerwall+ and 4x Powerwalls. I haven't had any issues with the Neurio at least so far besides sometimes the inveter likes to discount from the Wi-Fi every once in a while which what I found was to re join it via the App surpisingly and it fixes it.

Thanks for your amazing work so far with this!

jasonacox commented 2 years ago

Thanks @reli3 ! I cross posted this as an issue under pypowerwall for Strings Data- (see https://github.com/jasonacox/pypowerwall/issues/1) since I don't know how relevant it would be for this issue. I included the script I use to gather and push the string data into Splunk. I plan to add this to the Grafana dashboard as well. :)

brianhealey commented 2 years ago

Glad to see that the protocol buffer example helped. :) There were some errors in what I put together, so I forked your branch to update it. Hopefully this update will help flush out any issues your might be seeing and/or include fields I didn't have before that might be silently ignored.