search

jbuehl / solaredge

SolarEdge inverter logging data capture
GNU General Public License v3.0
288 stars 60 forks source link

Stopped working! #8

Closed martynwendon closed 6 years ago

martynwendon commented 8 years ago

Hello again,

This has been working perfectly for over a month now, but last Monday (7th) it suddenly stopped logging data :-(

I'm not sure why, nothing has changed with the config at all. Restarting the scripts / rebooting etc doesn't help.

The solaredge web portal is still receiving data and updating correctly.

The pcap files are still getting created.

I enabled debugging and took a look at the log output - there seemed to be a lot of "solaredge unknown seType", more than I remember from previously watching the logs when I first got up and running.

Could you take a look at one of the pcap files and see if it seems "normal" and produces expected data for you?

solaredge-20151212230003.zip

Thanks for your help!

Martyn

rippiedoos commented 8 years ago

Someone in that topic tried to brute force the checksum and simply tried all 180 billion combinations. He concluded it's not CRC of any kind over the entire packet. He wanted to try and brake it up over multiple parts of the data packet but hasn't posted back yet.

Op 1 feb. 2016 om 21:21 heeft jbuehl notifications@github.com het volgende geschreven:

I just found this http://gathering.tweakers.net/forum/list_messages/1489893 which has some posts relevant to SolarEdge. I didn't see anything new there, but I am relying on google to translate it so maybe I missed something.

— Reply to this email directly or view it on GitHub.

jbuehl commented 8 years ago

More information...

The 0302 message is the one that causes the slave to send the 0500 message. If the 0500 message contains data, the master eventually responds with a 0080 message that contains the sequence number that was in the 0500 message. This is obviously an acknowledgement because if the slave doesn't get the 0080 message it will send the same data again in the next 0500 message.

I still haven't figured out the checksum algorithm. I tried this http://reveng.sourceforge.net/ but it wasn't able to find a match. Unless the 0302 message has a valid checksum, the slave will ignore it and won't send any messages at all. I was able to repeatedly send a hard coded 0302 message that was an exact duplicate of one that had earlier been sent by the master and the slave responded to it with a 0500 message, but it was always the same data because it hadn't received the 0080 acknowledgement.

So emulating a SolarEdge master is definitely possible once the checksum algorithm is determined, but until that happens, spending 500 USD on a CCG is the only option for being able to get the data.

martynwendon commented 8 years ago

That's good progress so far, so we need to replicate:

master sends 0302 + sequence + checksum

slave responds with 0500 + sequence (from above) + checksum

master sends 0080 + sequence number (from above) + checksum to acknowledge receipt

...repeat

I think you mentioned before the sequence numbers are incremental, but presume they must restart / wrap around at some point?

When you get time can you post a bunch of raw data showing the above message flow so we can get eyes on the bytes and see if we can help figure out the checksum?

jbuehl commented 8 years ago

Almost correct, martynwendon. The sequence number in the 0302 message is repeated in the 039a message from the slave. A different sequence is in the 0500 and 0080 messages. The 0302 message seems to cause the 0500 message to occasionally be sent, but not always. Often the 0500 message doesn't contain any data and in this case there is no 0080 sent in response, however the 0500 sequence is incremented in the next 0500 message. The slave always replies to the 0302 with the 039a regardless. I think the 0347-039f message pair is a request for status data. This has its own sequence and seems to happen concurrently with the other conversation.

dir  flgs seq   func data
---- ---- ----   ---- ----------------------------
M->S FFFF seq1   0347
M->S FFFF seq2   0302
M<-S FFF1 seq1   039F status data ?
M<-S FFFF seq2   039A
M->S FFFF seq2+1 0302
M<-S FFFF seq3   0500
M<-S FFFF seq2+1 039A
M->S FFFF seq2+2 0302
M<-S FE2F seq3+1 0500 inverter + optimizer data
M<-S FFFF seq2+2 039A
M->S FFFF seq3+1 0080
jbuehl commented 8 years ago

Here's a file containing some captured RS485 data https://www.dropbox.com/sh/rlg6ennmibicrxt/AAAj_QAJSwmX_xtutAov2cfDa?dl=0.

You can display it by running python semonitor.py -l -vvvv 20160201.dat

jbuehl commented 8 years ago

Here is something interesting I observed. Looking at 8 successive 0080 messages where only the sequence number is being incremented, the checksum follows a pattern.

0000ffff5c04feffffff164a107f80002b23
0000ffff5d04feffffff164a107f80002fdf
0000ffff5e04feffffff164a107f8000209b
0000ffff5f04feffffff164a107f80002467
0000ffff6004feffffff164a107f8000eb73
0000ffff6104feffffff164a107f8000ef8f
0000ffff6204feffffff164a107f8000e0cb
0000ffff6304feffffff164a107f8000e437

In the first 4 messages when the sequence number is 045c-045f, the first digit of the checksum is always 2. In the last 4 messages containing sequence numbers 0460-0463 the first digit of the checksum is always e. The 2nd checksum digit cycles through b-f-0-4 and the 4th checksum digit cycles through 3-f-b-7. This may be a useful clue to what the algorithm is.

martynwendon commented 8 years ago

Yes I noticed that too :-)

Near identical message where only a few bits in a single byte have changed produce such different checksums.

123456790000ffff9c6e164a107f2049107f9a030250 123456790000ffff9d6e164a107f2049107f9a0306ac

It definitely looks like a CRC of some sort but doesn't immediately present anything obvious when running some of the data through http://www.lammertbies.nl/comm/info/crc-calculation.html

I would think it's something pretty standard, it's only RS485 after all, I don't see why you would reinvent the wheel for something as trivial as serial comms?

jbuehl commented 8 years ago

This looks like a good resource http://www.cosc.canterbury.ac.nz/greg.ewing/essays/CRC-Reverse-Engineering.html.

martynwendon commented 8 years ago

That looks pretty good, I like the technique for manipulating bits in the message to see what parts of the data the CRC might cover, that's pretty clever!

gns1l commented 8 years ago

Very impressed with your progress and findings. On the pycrc website several CRC models are mentioned. Among them is crc-16-modbus, perhaps worth a try?

crc-16-modbus Width 16 Poly 0x8005 Reflect In True XOR In 0xffff Reflect Out True XOR Out 0x0000 Short command pycrc.py --model crc-16-modbus Extended command pycrc.py --width 16 --poly 0x8005 --reflect-in True --xor-in 0xffff --reflect-out True --xor-out 0x0000 Check 0x4b37

jbuehl commented 8 years ago

I mentioned above that I used the search function of reveng, which tests a message-checksum pair against a list of known models http://reveng.sourceforge.net/crc-catalogue/16.htm#crc.cat-bits.16 which includes the modbus model but there were no matches.

Using the procedures described in http://www.cosc.canterbury.ac.nz/greg.ewing/essays/CRC-Reverse-Engineering.html I have determined that it passes the superposition test, so it is definitely a CRC calculation of some sort. Using the method to deduce the polynomial, I did determine that the polynomial may be 0xa001, which, according to that article, really means the polynomial is 0x8005 with refIn and refOut = True. I say "may be" because the method described didn't exactly work without an additional step. That still leaves some more parameters to be determined, and also the question of how much of the message is participating in the CRC (i.e. is the magic number 0x12345679 included).

I think this is going to eventually be solved and in the meantime I'm learning more than I ever wanted to about the arcane details of CRC calculations :)

Jerrythafast commented 8 years ago

Hey everyone,

I think it's time I'd add myself to the list of participants in this discussion. I sort of implemented my own version of the scripts in this repository back in December (in hindsight that must have been around the time they turned the key at SolarEdge). We only got our SE3000 since last Wednesday, so back in December I did this using data that was made available by a member of gathering.tweakers.net (the forum that was mentioned above; I'm that person who literally tried 180 billion combinations of checksum computations only to find out none of them worked). While reverse-engineering the communication myself, I was able to fill in some of the remaining gaps, but now that they changed their data packets that work seems kind of wasted and I haven't even been able to use it on our SE3000!

However, by reading the above I see that SE's serial protocol is identical to their Ethernet protocol, and that they are still using the old packet types over the serial connection. So I may still be of help with some things:

The SolarEdge message header:

struct SEHeader {
  UINT8 magic[4]; // = {0x12, 0x34, 0x56, 0x79}
  UINT16 dataSize;
  UINT16 dataSizeInv; // The one's complement of dataSize, i.e., ~dataSize
  UINT16 sequenceNumber; // [note2]
  INT32 sourceID; // The hardware ID of the sender of the message [note1]
  INT32 targetID; // The hardware ID of the receiver of the message [note1]
  UINT16 mesageType; // Might also include flags, i.e., the MSb is generally set on replies
}

The header is followed by dataSize bytes of data, and a two-byte checksum.

Note 1; hardware IDs can be: (1) the inverter serial number, (2) the inverter serial number with an extra bit set, (3) a special value below zero which I think correspond to various pieces of software that each have their specific purpose. For the type (3) hardware ID's, I've seen -1, -2, and -3, where -2 seems to be the logging software of the SolarEdge servers. Note 2: Separate sequence numbering is used for each combination of sourceID and targetID.

And perhaps I also have some gaps filled in the 0500 packets, but that data is on a different PC so you'll have to do with this for now.

Anyway, I only have access to the Ethernet logging of my SE3000, so I'll have to work with that and actually logged what happened since they installed the thing here. Most interesting is what happened today. The inverter started sending the old-style 0500 type messages when it came online last Wednesday. It kept doing so for two days. Now, all data is in 003d type messages since this afternoon, after a number of TCP retransmits and connection resets that occurred for about 20 seconds. According to Wireshark there are even outgoing IP packets with source=0.0.0.0, so the inverter was certainly doing something funny there. If it was updating firmware, the firmware did not come over TCP since tcpdump hasn't logged anything looking like firmware.

The SE3000 was nicely sending 0500 packets, addressed to targetID=-2. Then all of a sudden sourceID=-3 called in with eight 0012 packets, then 0034, 0011, 0030. Then came the 20 seconds of connection madness; in this time, a 0012 packet was being repeated by sourceID=-3 which never got an answer.

When the inverter came back online, it first sent a 0503 packet to sourceID=-2. From this moment onwards, it only ever sent 003d packets and sporadically a 0503. The outgoing packets always have targetID=-2, but the responses from SolarEdge are weird since they always have sourceID=-3 and targetID=-1. So I guess I am now seeing the same garbage as you guys are and I haven't got a clue yet of what it is.

Cheers, Jerry

jbuehl commented 8 years ago

Thanks for the info Jerry. I hadn't realized that the word following the data size is the complement of the data size. I wonder how that is useful.

When you say you tried 180 billion checksum combinations, how do you mean that? There are only 65K possible checksum values. I was able to determine the valid checksum for a specific message by sending out all combinations until the inverter responded, however that is only useful for a fixed message. To be able to find valid checksums for all possible 0080 acknowledgement messages would "only" require 4 billion combinations.

Good to know about the inverter reverting to the old format. Maybe when an inverter starts up it uses the old format, and then the server tells it to switch at some point. If that's the case, then spoofing the SolarEdge server over ethernet starts to look more attractive since the checksum issue goes away.

Jerrythafast commented 8 years ago

Thanks for the info Jerry. I hadn't realized that the word following the data size is the complement of the data size. I wonder how that is useful.

I think I know how it would be useful. Since the communication includes checksum bytes, obviously their software is checking whether the messages came through correctly by computing the checksum. Since the messages have variable length, you'll need to know the length of the message to find the checksum. So I think it makes sense to send the message length twice like this. (On a TCP connection, all of this is overkill. But since they use the same protocol over the serial port it makes sense to do this.)

When you say you tried 180 billion checksum combinations, how do you mean that?

I used the following code (which is written and commented in English) to try various combinations of operations on each input byte (e.g., bit rotation, XOR, addition, bit-complement, etc.) with the goal of computing the checksum bytes from the content of the message. I included the magic number and the rest of the header in this. http://gathering.tweakers.net/forum/list_message/45507202#45507202

I also ran this code a second time, this time including the checksum bytes in the calculation, with the goal of finding zero as the result. But this failed as well; there was no configuration of options (out of 180 billion possible) that computed the correct checksum for all input messages. Obviously millions of configurations worked for any particular message, but none worked for all of them.

Some textbook examples like Fletcher's checksum and the IP header checksum are covered by this code, but CRC-like checksums are not.

Good to know about the inverter reverting to the old format.

It doesn't really seem to revert to the old format; the 0503 packets look ugly as well. They also use distinct sequence numbering from the 003d packets though it does seem mostly increasing. The first 0503 packets got sequence numbers 133 1d2 1e3 2c2 3e5 3e7 3ee 02d.

This is what I logged after the last 0500 message, up to and including the first 0503 message. It is only sending 003d and occasionally 0503 packets since this happened.

SEQ  SOURCE   TARGET   TYPE  DATA
09d3 fffffffd <invert> 0012  19 00
09d3 <invert> fffffffd 0090  03 00 00 00 02 00
09d4 fffffffd <invert> 0012  4a 01
09d4 <invert> fffffffd 0090  4e 00 00 00 00 00
09d5 fffffffd <invert> 0012  29 03
09d5 <invert> fffffffd 0090  00 00 00 00 00 00
09d6 fffffffd <invert> 0012  39 02
09d6 <invert> fffffffd 0090  85 de b9 3a 00 00
09d7 fffffffd <invert> 0012  3a 02
09d7 <invert> fffffffd 0090  b4 ca e9 7d 00 00
09d8 fffffffd <invert> 0012  3b 02
09d8 <invert> fffffffd 0090  18 69 ff 60 00 00
09d9 fffffffd <invert> 0012  3c 02
09d9 <invert> fffffffd 0090  d3 83 a6 40 00 00
09da fffffffd <invert> 0012  29 03
09da <invert> fffffffd 0090  00 00 00 00 00 00
09db fffffffd <invert> 0034
09db <invert> fffffffd 00b1  00 03
09dc fffffffd <invert> 0011  29 03 01 00 00 00
09dc <invert> fffffffd 0080
09dd fffffffd <invert> 0030  f4 01 00 00 00 00
09dd <invert> fffffffd 0080
09de fffffffd <invert> 0012  29 03
<last message was repeated 12 more times as inverter lost its connection a couple of times in 20 seconds>
0133 <invert> fffffffe 0503  52 5c cd 6f 06 8e ed 50 59 a0 bf 61 d8 80 90 19 1f 87 df 6d 48 3b 15 1e 62 0a ec 7e c7 d9 e4 43 44 34

Maybe the 0503 packet contains some sort of encryption key.

I also realised that it is highly unlikely that a firmware update led to this. As far as I know there is no other network protocol like TCP in common use that offers the same amount of reliability. So given the fact that SolarEdge is using TCP for logging (which is relatively unimportant), I wouldn't expect them to transfer firmware updates without TCP. And since tcpdump has not recorded anything other than the above, the only conclusion I can make is that there was no firmware update.

So that means that possibly, the above communication caused the inverter to change its output format. And if we would be able to spoof similar messages we might be able to switch it back to reporting in the old format...

Jerry

jbuehl commented 8 years ago

If the length and its complement are used in the verification of message integrity then maybe they aren't included in the checksum calculation. That's something to try. If you didn't eliminate CRC-like checksums then I'm going to continue along that route since there is a good indication that it is one of those.

Whizkidzz commented 8 years ago

jbuehl,

Hi I also did some reverse engineering on the CRC16 from the configuration tool software and came up with the following which works. See code below:

TPacket = packed record
       Barker  : array[0..3] of byte;
       PLength : smallint;
       Nlength : smallint;
       ID     : smallint;
       Source  : longint;
       Destination    : longint;
       Opcode  : smallint;
       Data    : array[0..480] of Byte;
  end;                              

`const
crc16_table : array[0..255] of cardinal = (
0,  49345,  49537,    320,  49921,    960,    640,  49729,
50689,   1728,   1920,  51009,   1280,  50625,  50305,   1088,
52225,   3264,   3456,  52545,   3840,  53185,  52865,   3648,
2560,  51905,  52097,   2880,  51457,   2496,   2176,  51265,
55297,   6336,   6528,  55617,   6912,  56257,  55937,   6720,
7680,  57025,  57217,   8000,  56577,   7616,   7296,  56385,
5120,  54465,  54657,   5440,  55041,   6080,   5760,  54849,
53761,   4800,   4992,  54081,   4352,  53697,  53377,   4160,
61441,  12480,  12672,  61761,  13056,  62401,  62081,  12864,
13824,  63169,  63361,  14144,  62721,  13760,  13440,  62529,
15360,  64705,  64897,  15680,  65281,  16320,  16000,  65089,
64001,  15040,  15232,  64321,  14592,  63937,  63617,  14400,
10240,  59585,  59777,  10560,  60161,  11200,  10880,  59969,
60929,  11968,  12160,  61249,  11520,  60865,  60545,  11328,
58369,   9408,   9600,  58689,   9984,  59329,  59009,   9792,
8704,  58049,  58241,   9024,  57601,   8640,   8320,  57409,
40961,  24768,  24960,  41281,  25344,  41921,  41601,  25152,
26112,  42689,  42881,  26432,  42241,  26048,  25728,  42049,
27648,  44225,  44417,  27968,  44801,  28608,  28288,  44609,
43521,  27328,  27520,  43841,  26880,  43457,  43137,  26688,
30720,  47297,  47489,  31040,  47873,  31680,  31360,  47681,
48641,  32448,  32640,  48961,  32000,  48577,  48257,  31808,
46081,  29888,  30080,  46401,  30464,  47041,  46721,  30272,
29184,  45761,  45953,  29504,  45313,  29120,  28800,  45121,
20480,  37057,  37249,  20800,  37633,  21440,  21120,  37441,
38401,  22208,  22400,  38721,  21760,  38337,  38017,  21568,
39937,  23744,  23936,  40257,  24320,  40897,  40577,  24128,
23040,  39617,  39809,  23360,  39169,  22976,  22656,  38977,
34817,  18624,  18816,  35137,  19200,  35777,  35457,  19008,
19968,  36545,  36737,  20288,  36097,  19904,  19584,  35905,
17408,  33985,  34177,  17728,  34561,  18368,  18048,  34369,
33281,  17088,  17280,  33601,  16640,  33217,  32897,  16448);

Function Getchecksum(Base:Tpacket):Cardinal;
var
  crc    : cardinal;
begin
  crc := $5A5A;
  crc := Calc16MsbFirst(crc, base.ID);
  crc := Calc32MsbFirst(crc, base.Source);
  crc := Calc32MsbFirst(crc, base.Destination);
  crc := Calc16MsbFirst(crc, base.OpCode);
  result:= Calc(crc, base.Data, base.PLength);
end;

Function CalcByte(crc:cardinal; data : byte):Cardinal;
begin
  result := crc16_table[(crc xor data) and $ff] xor (crc shr 8);
end;

Function Calc16MsbFirst(crc:cardinal; data : smallint):Cardinal;
begin
  crc:= CalcByte(crc,(data shr 8));
  result := CalcByte(crc,(data and 255));
end;

Function Calc32MsbFirst(crc:cardinal; data : longint):Cardinal;
begin
  crc:= CalcByte(crc,((data shr 24) and 255));
  crc:= CalcByte(crc,((data shr 16) and 255));
  crc:= CalcByte(crc,((data shr 8) and 255));
  result := CalcByte(crc,(data and 255));
end;

Function Calc(crc:cardinal; data : array of byte; len : smallint):Cardinal;
var
  i      : Integer;
begin
  For i:= 0 to len-1 do
    crc := CalcByte(crc,data[i]);

  result:=crc;
end;              `

I have some more information derived from the configtool software, but it seams that in the configtool software the optimizer part is not working. (for me).

Whizkidzz commented 8 years ago

JerryThafast

I have some answers on your data

SEQ  SOURCE   TARGET   TYPE  DATA
09d3 fffffffd <invert> 0012  19 00 --> 0012 means request parameter with number 0019
09d3 <invert> fffffffd 0090  03 00 00 00 02 00 --> 0090 means return parameter value ( first 4 bytes) In type (last 2 bytes) type 0 = Int32, type 1 = float32, type 2 = unsigned int 32, type 3 = string
So value here is 03
09d4 fffffffd <invert> 0012  4a 01
09d4 <invert> fffffffd 0090  4e 00 00 00 00 00
09d5 fffffffd <invert> 0012  29 03
09d5 <invert> fffffffd 0090  00 00 00 00 00 00
09d6 fffffffd <invert> 0012  39 02
09d6 <invert> fffffffd 0090  85 de b9 3a 00 00
09d7 fffffffd <invert> 0012  3a 02
09d7 <invert> fffffffd 0090  b4 ca e9 7d 00 00
09d8 fffffffd <invert> 0012  3b 02
09d8 <invert> fffffffd 0090  18 69 ff 60 00 00
09d9 fffffffd <invert> 0012  3c 02
09d9 <invert> fffffffd 0090  d3 83 a6 40 00 00
09da fffffffd <invert> 0012  29 03
09da <invert> fffffffd 0090  00 00 00 00 00 00
09db fffffffd <invert> 0034
09db <invert> fffffffd 00b1  00 03
09dc fffffffd <invert> 0011  29 03 01 00 00 00
09dc <invert> fffffffd 0080
09dd fffffffd <invert> 0030  f4 01 00 00 00 00
09dd <invert> fffffffd 0080
jbuehl commented 8 years ago

That's good news, Whizkidzz. What computer language is that? Pascal?

Whizkidzz commented 8 years ago

Computer language is free pascal (lazarus)

jbuehl commented 8 years ago

That does indeed work! I will incorporate that into semonitor.py.

Jerrythafast commented 8 years ago

Awesome!! So we finally got the checksum. Perhaps this opens opportunities for us to craft a message that we can send to the inverter to convince it that it should send the nice old-style monitoring portal updates.

Or Whizkidzz could work his magic on the encrypted monitoring portal updates? Heck, how did you even crack the CRC to begin with?!

jbuehl commented 8 years ago

Here is an implementation in python of this checksum method. It doesn't use the table method that was posted above, so it won't be as fast, but it is more understandable and produces identical results. This is a standard CRC algorithm with the following Rocksoft parameters:

width=16 poly=0x8005 init=0x5a5a refin=true refout=true xorout=0x0000

The odd thing about is is that the CRC is calculated on a message with big-endian elements, but the transmitted message has little-endian elements. Also there's the data length and ones complement of the length that aren't included in the CRC. Not the way I would have designed it, but it is what it is.

    # parse the message
    (magic) = struct.unpack(">L", rec[0:4])[0]
    (dataLen, dataLenInv, msgSeq, invFrom, invTo, function) = struct.unpack("<HHHLLH", rec[4:dataPtr])
    (checksum) = struct.unpack("<H", rec[-2:])[0]
    # calculate what the checksum should be
    calcsum = calcCrc(struct.pack(">HLLH", msgSeq, invFrom, invTo, function)+rec[dataPtr:-2])

# crc calculation
def calcCrc(data):
    crc = 0x5a5a    # initial value
    for d in data:
        crc ^= struct.unpack("!B",d)[0]
        for i in range(8):
            if (crc & 0x0001) != 0:
                crc >>= 1
                crc ^= 0xa001   # reflected polynomial 0x8005
            else:
                crc >>= 1
    return crc
Jerrythafast commented 8 years ago

With thanks to Whizkidzz above and richardt at Tweakers.net (for pointing me to a zip file on the SolarEdge website that contains very informative DLLs...), I managed to decode some more of the communication that happened when my inverter switched from the old to the new format.

SOURCE     TARGET     COMMAND                DATA
<conftool> <inverter> CMD_PARAMS_GET_SINGLE  Get value of PARAM_INVGRID_DUTY_CYCLE
<inverter> <conftool> RESP_PARAMS_SINGLE     INT: 3
<conftool> <inverter> CMD_PARAMS_GET_SINGLE  Get value of 0x14a
<inverter> <conftool> RESP_PARAMS_SINGLE     UINT32: 78
<conftool> <inverter> CMD_PARAMS_GET_SINGLE  Get value of 0x329
<inverter> <conftool> RESP_PARAMS_SINGLE     UINT32: 0
<conftool> <inverter> CMD_PARAMS_GET_SINGLE  Get value of 0x239 (possibly PORTIA_PARAM_ACTIVATION_MAGIC_START)
<inverter> <conftool> RESP_PARAMS_SINGLE     UINT32: 85 de b9 3a
<conftool> <inverter> CMD_PARAMS_GET_SINGLE  Get value of 0x23a
<inverter> <conftool> RESP_PARAMS_SINGLE     UINT32: b4 ca e9 7d
<conftool> <inverter> CMD_PARAMS_GET_SINGLE  Get value of 0x23b
<inverter> <conftool> RESP_PARAMS_SINGLE     UINT32: 18 69 ff 60
<conftool> <inverter> CMD_PARAMS_GET_SINGLE  Get value of 0x23c
<inverter> <conftool> RESP_PARAMS_SINGLE     UINT32: d3 83 a6 40
<conftool> <inverter> CMD_PARAMS_GET_SINGLE  Get value of 0x329
<inverter> <conftool> RESP_PARAMS_SINGLE     UINT32: 0
<conftool> <inverter> CMD_MISC_GET_TYPE
<inverter> <conftool> RESP_MISC_GET_TYPE     POLESTAR (or PORTIA)
<conftool> <inverter> CMD_PARAMS_SET_SINGLE  Set value of 0x329 to 1
<inverter> <conftool> RESP_ACK
<conftool> <inverter> CMD_MISC_RESET         Reset device after 500ms
<inverter> <conftool> RESP_ACK
<conftool> <inverter> CMD_PARAMS_GET_SINGLE  Get value of 0x329

Some notes:

Particularly important notes:

TL;DR: Someone with a serial connection to the inverter should try to have the following conversation with it:

SEQ  SOURCE   DEST     TYPE  DATA               EXPLANATION
0123 fffffffd <invert> 0011  29 03 00 00 00 00  Set parameter 0x329 to 0
0123 <invert> fffffffd 0080                     Expected response (acknowledgement)
0124 fffffffd <invert> 0030  f4 01 00 00 00 00  Reset inverter after 500ms
0124 <invert> fffffffd 0080                     Expected response (acknowledgement)

Shortly after this, the inverter should reboot and it might start sending messages in the old format again.

Jerry

jbuehl commented 8 years ago

Well, it looks like Jerry's suggestion works! I sent the commands to set the value of 0x0329 to 0, followed by the 0030 reset function, and the inverter has now begun sending 0500 messages to the SolarEdge server. Of course, they will no doubt change it back again, but I have a plan to address that.

I can cause the inverter to connect to my application running on a local server instead of solaredge.com using a lightweight dhcp and dns server that I have implemented. The app receives the 0500 messages and sends acknowledgements. There may be some additional functions that the SolarEdge server performs that will need to be added. I will be posting it on github as soon as it is debugged and documented.

Jerrythafast commented 8 years ago

That's quite exciting news, @jbuehl! Now that your inverter is sending 0500 messages again, does the SolarEdge monitoring website still show the up-to-date data? That is, do they still accept data in the old format like before?

jbuehl commented 8 years ago

Yes, the monitoring server also accepts data from the 0500 messages, however after about a day they reset it back to the new format as I expected. When that happened, the sequence of messages that I saw was very similar to what you posted earlier. It included a request for the value of parameter 19, which in my case was also 3, so I don't think that it represents what you were thinking it was since my inverters have been running for 2-1/2 years. The 0034 command was also responded to with a 00b1 function with the value 0300.

<conftool> <inverter> 0012 0019
<inverter> <conftool> 0090 00000003 0002
<conftool> <inverter> 0012 014a
<inverter> <conftool> 0090 00000003 0000
<conftool> <inverter> 0012 0329
<inverter> <conftool> 0090 00000000 0000
<conftool> <inverter> 0012 0239
<inverter> <conftool> 0090 232599e1 0000
<conftool> <inverter> 0012 023a
<inverter> <conftool> 0090 47bbddf4 0000
<conftool> <inverter> 0012 023b
<inverter> <conftool> 0090 6f6f33d5 0000
<conftool> <inverter> 0012 023c
<inverter> <conftool> 0090 78327e17 0000
<conftool> <inverter> 0012 0329
<inverter> <conftool> 0090 00000000 0000
<conftool> <inverter> 0034
<inverter> <conftool> 00b1 0300
<conftool> <inverter> 0011 0329 00000001
<inverter> <conftool> 0080
<conftool> <inverter> 0030 000001f4 0000
<inverter> <conftool> 0080

Somehow the time on one of my inverters got set 2 hours later than local time. I think it happened a few days ago, probably caused by something that I did while testing. I need to find out the command to set the time and time zone, which is going to be necessary in order to replace the monitoring server.

Jerrythafast commented 8 years ago

That's propbably not something you did while testing. On the tweakers.net forum there were some similar reports from other people last weekend. People who are not fiddling with their inverters.

jbuehl commented 8 years ago

Can you send me the link to that thread in the tweakers.net form? I can't locate it. Thanks.

There is a way to set the time on an inverter through the configuration tool, but it requires an elevated access level which seems to be controlled by SolarEdge.

Whizkidzz commented 8 years ago

Jbuehl

When i have time Will post the code and program to calculator the password for the engineer and advanced engineer levels The other levels requires a computer in the solaredge domain

Jerrythafast commented 8 years ago

Hi @jbuehl,

Here are the posts at the tweakers.net forum that I could find. (Note they are all in Dutch.) Gasschuif 27 February janesh 24 February Stef87 27 February rik11 27 February

@Whizkidzz, do you have any ideas about how we could decode the 003d and 0503 packets? It would be great if I could decrypt the data I gathered in the past four weeks. And it would also save ourselves from the hassle of having to defend against SolarEdge's attempts to turn the encryption back on.

Whizkidzz commented 8 years ago

Jerrythagast

For finding anything on the 003d and 0503 messages i need a recent firmware file with a version of 3.14xx or higher Firmware files have a extension of bsuf or sus .bsuf being a binary file and a .sus file is a xml file contaning the 3 binary bsuf's as base64 encoded strings

So if anyone can provide me with one of these i can certainly try

Jerrythafast commented 8 years ago

Perhaps there is a firmware download command to get the current firmware from the inverter. There is command 0023 (UPGRADE_READ_DATA). It takes two Int32 parameters, offset and size. The inverter will (or should) respond with a 0160 packet containing the requested data. Since these functions are in the same group as the ones that write new firmware to the inverter, I am kind of hoping you can get the inverter to dump its firmware this way.

If you would like to give it a shot, here's what I think the conversation should look like:

SEQ  SOURCE   DEST     TYPE  DATA               EXPLANATION
0123 fffffffd <invert> 0024                     UPGRADE_READ_SIZE
0123 <invert> fffffffd 00a1  xxxxxxxx           Size (in bytes) of some data that can be read
0124 fffffffd <invert> 0023  00000000 xxxxxxxx  UPGRADE_READ_DATA, offset=0, size=<value received from inverter>
0124 <invert> fffffffd 00a0  ...                Inverter sends the requested data

In words: you request the size of the data that can be read from the inverter. Then, you request this amount of data.

jbuehl commented 8 years ago

@Jerrythafast - if google translated those pages correctly, it looks like the person with the time problem called SolarEdge to fix it. I'm going to wait until I know how to set it myself.

@Whizkidzz - Thanks, I look forward to seeing how to generate the password.

When I used the configuration tool for the first time and it told me that I needed to update the firmware on one of my inverters. I didn't select a local file, I just clicked on the Update button and it began updating. I don't see a .sus or .bsuf file anywhere on my computer so maybe the tool downloaded it from SolarEdge as it updated.

jbuehl commented 8 years ago

I think I just found a way to set the time. Looking at some old pcap files I just found 2 messages that seem to be time related. The inverter occasionally sends a 0501 message that contains no data and the server replies with a 0580 message containing 8 bytes of data which appear to be a timestamp and a time zone offset.

<inverter> fffffffe 0501
fffffffe <inverter> 0580 56360a18 ffff8f80

0x56360a18 = 'Sun Nov 1 04:48:24 2015' which was the time the data was captured 0xffff8f80 = -28800 which is my TZ offset in seconds (UTC -8)

Whizkidzz commented 8 years ago

@jbuehl The configuration tool contains a firmware file with the version 2.0496 Do you remember the Firmware after this update and what the firmware is now?

For writing the time use the following command fffffffd inverter 0303 56360a18 ffff8f80 response will be inverter fffffffd 0080

The commands mentioned in your post are the following PROT_CMD_SERVER_POST_DATA = 0x0500, PROT_CMD_SERVER_GET_GMT = 0x0501, PROT_CMD_SERVER_GET_NAME = 0x0502, PROT_RESP_SERVER_GMT = 0x0580, PROT_RESP_SERVER_NAME = 0x0581,

If you want a look for your self in the inner workings of the configtool goto http://www.ilspy.net/

Whizkidzz commented 8 years ago

@jbuehl. Look at my github page for the configtool V2 and access codes

jbuehl commented 8 years ago

@Whizkidzz - The configuration tool updated my inverter to 2.0496. I didn't note what it was before, but it was a lower version. My other inverter (the one connected to the internet) is at 2.1049. That has obviously been updated by Solaredge because both inverters would have been at the same version when I bought them.

I think I understand your code for generating the passwords, it appears that they are generated as a function of the inverter serial number, check digits, and the current date, so they have to be regenerated each day. Is that correct? I'll try it out as soon as I translate the Pascal to Python.

Whizkidzz commented 8 years ago

@jbuehl

Indeed the date is included in the access code But the code is valid Until the year 2130

The firmware version 2.0496 is included in the configtool v3 Tommorow Will try to extract the latest firmware from the inverter after the suggestion from jerrythafast

Jerrythafast commented 8 years ago

@Whizkidzz, this popped into my mind today: when you are going to try to extract the firmware from the inverter, you'll probably have to do multiple requests for data (up to 32 kB or perhaps 64 kB per request) because the 'data size' packet header field is only two bytes, which puts a limit to the amount that can be delivered in a single response.

jbuehl commented 8 years ago

The config tool 3.0 didn't show the tabs to do any configuration of communication parameters or system time even when I entered the password. Version 2.0 has those functions and doesn't require the password. It seems odd that SolarEdge only has the 3.0 tool and the 2.0 documentation available for download on their site.

I was able to set the time from my software using the 0303 command, so I didn't need the config tool.

jbuehl commented 8 years ago

@Jerrythafast - I tried sending the 0024 command over the RS232 connection and the inverter didn't respond. I tried sending the 0023 command with an offset of 0 and an arbitrary value of 0x100 and the inverter responded with an 0081 message containing no data.

Jerrythafast commented 8 years ago

The 0081 message is called PROT_RESP_NACK, so that's a general error saying "I received your command but I didn't do anything".

You might want to try sending these commands to your inverter with "the bit" in its serial number set (i.e., add 0x00800000 to the inverter ID). This is called the "DSP offset".

(Also, minor note: the inverter's response to 0023 should be a 00a0 packet. I erroneously listed that as 0160 previously. I have now corrected my previous comment.)

jbuehl commented 8 years ago

That's what I thought 0081 probably meant. I just tried both commands with bit 23 set in the inverter ID and it responded to both with a 00a1. I'm sure this means that the ID is invalid. Can someone please post the full list of message codes and names? Thanks.

Jerrythafast commented 8 years ago

00a1 is PROT_RESP_UPGRADE_READ_SIZE. That's what I would expect as the reponse to 0024. It should contain a 32-bit integer that tells you (presumably!) the data size. It should respond to the 0023 command with a 00a0 (PROT_RESP_UPGRADE_READ_DATA).

I'll compile you a list of these codes.

jbuehl commented 8 years ago

Doh! I should have looked back to your earlier post... I didn't notice the data in the 00a1 message due to a bug in my code. The 00a1 message is returning the value 0x18000. When I send the 0023 message with that value, there is no response. I also tried sending the 0023 with a smaller value of 0x100, but no response either. I have used both fffffffe and fffffffd as the sending address.

<fffffffd> <inverter> 0024
<inverter> <fffffffd> 00a1 00018000
<fffffffd> <inverter> 0023 00000000 00018000
...

I may try this over the ethernet interface later.

Jerrythafast commented 8 years ago

I think it's quite exciting how far we got already. We're sort-of speaking the protocol :D

jbuehl commented 8 years ago

Thanks ;)

jbuehl commented 8 years ago

I have noticed an occasional entry in the 0500 messages that has a type of 0300. These messages occur twice a day for each inverter, around the times of sunrise and sunset. Each 0500 message contains two 0300 entries, and never any 0000 optimizer or 0010 inverter entries. The length of the data is 28 bytes and contains values, some of which are valid timestamps. I am sure that these correspond to the times that the inverters are waking up and going to sleep.

<stdin> --> message: 3777 length: 101 
data:       12 34 56 79 48 00 b7 ff 12 00 16 4a 10 7f fe ff 
data:       ff ff 00 05 00 03 16 4a 10 7f 1c 00 76 7e 35 56 
data:       00 00 00 00 76 7e 35 56 f6 0d 35 56 00 00 00 00 
data:       00 00 00 00 00 00 00 00 00 03 16 4a 10 7f 1c 00 
data:       7b 7e 35 56 01 00 00 00 7a 7e 35 56 80 8f ff ff 
data:       7b 7e 35 56 00 00 00 00 00 00 00 00 ca 6f 00 00 
data:       00 00 00 00 00 
dataLen:    0048 
dataLenInv: ffb7 
sequence:   0012 
source:     7f104a16 
dest:       fffffffe 
function:   0500 
time:          
    type : 0300 
    id : 7F104A16 
    len : 001c 
    Date : 2015-10-31 
    Time : 19:52:38 
    ID : 7F104A16 
    Type  : 00000000
    Time1 : Sat Oct 31 19:52:38 2015 
    Time2 : Sat Oct 31 11:52:38 2015 
    Data1 : 00000000 
    Data2 : 00000000 
    Data3 : 00000000 
time:          
    type : 0300 
    id : 7F104A16 
    len : 001c 
    Date : 2015-10-31 
    Time : 19:52:43 
    ID : 7F104A16 
    Type  : 00000001 
    Time1 : Sat Oct 31 19:52:42 2015 
    TZoff : -28800 
    Time2 : Sat Oct 31 19:52:43 2015 
    Data1 : 00000000 
    Data2 : 00000000
Jerrythafast commented 8 years ago

I ran into ConfigTool code that receives and processes the 0500 messages yesterday. I didn't thoroughly investigate because I was looking for hints to find out why your inverter didn't want to respond to 0023. I'm still kind of hoping we can get the current firmware so that we can decode the current data format. That would also be transparent to SolarEdge.

Maybe you could poke at it a little with different offsets. E.g., just request 10 bytes with different offsets spaced 1000 bytes apart. My reasoning is that we might need to look at a certain offset to get the loot.

jbuehl commented 8 years ago

I am not sure that the 0023 commands can be used to access firmware in the inverter, however I have found a better solution. I have identified the pcap file that contains the upgrade for my inverter. The file that I captured on 24-Nov-2015 was unusually large. It contains a 0033 PROT_CMD_MISC_GET_VER that returned 2.474, followed by a 0020 PROT_CMD_UPGRADE_START command, followed by 2184 0021 PROT_CMD_UPGRADE_WRITE commands, followed by a 0030 PROT_CMD_MISC_RESET command. A 0033 PROT_CMD_MISC_GET_VER sent after that returned 2.1049, so the firmware update is contained in the 0021 messages. 

<fffffffe> <inverter> 0033
<inverter> <fffffffe> 00b0 0002.0474 
<fffffffe> <inverter> 0020
<inverter> <fffffffe> 0080
<fffffffe> <inverter> 0021 data
<inverter> <fffffffe> 0080
... repeated 2184 times
<fffffffe> <inverter> 0030
<inverter> <fffffffe> 0080
<fffffffe> <inverter> 0033
<inverter> <fffffffe> 00b0 0002.1049

Each 0021 message has 248 bytes of data. It looks like the first 8 bytes contain a 32 bit offset, followed by a 32 bit length, which is almost always 0x0000f0 = 240 bytes. I created a 512K file solaredge-0002.1049.dat from these messages that should correspond to the firmware image.

You can access these files at https://www.dropbox.com/sh/rlg6ennmibicrxt/AAAj_QAJSwmX_xtutAov2cfDa?dl=0.