Add Version 5 protocol support

[XtheOne]

Implement support for V5 protocol as used in new embedded ethernet loggers. Needs documentation from Omnik and or iGEN / Solarman.

This is requested. Also a firmware update file is requested.

[XtheOne]

Development done in branch: "Experimental_Frame_Version_5_support"

[abc1000]

Tried 00-ff for the first bit, some works for 70%.. But with these empty, i'v got also a error.. frame_hdr = '' command = ''

[XtheOne]

frame_hdr = byte(headCode) + short(dataFieldLength) + short(contrlCode) + short(serialNumber) Change only contrlCode and serialNumber (which is a frame sequential number) if you change command length then change dataFieldLength to match the nr of bytes.

[XtheOne]

As there is still no more help from Omnik or SolarMan about the V5 frame data needed to trigger the logger we are a bit stuck at the moment. Will now wait for updates for the Android Apps or firmware from/for the ethernet module as none of the apps now gets data from the new logger in local mode. It seems that the firmware still is buggy as it frequently locks up so hopefully an update will come.

[PhyxionNL]

Added a PR, thanks for this script 👍

[XtheOne]

The reply is only 99 bytes, have to check the docs what frame type this is.

[PhyxionNL]

FYI, the main_fwver and slave_fwver both still fail in InverterMsg with this logger. Most likely because 101 and 121 don't exist due to 99 bytes.

[XtheOne]

Yes, cannot get data past EOF. Still need to fix this. No time too many work to do before end of this year :-(

[XtheOne]

Still NO help from Omnik or Solarman dev...

[coraminformatica]

Hello, I need also the V5 support. How can I help you? I tried your frame reader for V5 but I get right only the serial number. I can't decode e_total, e_today or other important values.

Regards

[XtheOne]

Still........ Nothing from Omnik. :-(

[stenjo]

I'm really interested in this. Have a data logger stick from Ginlong where they obviously are using frame format V5 as the inverter is logging new sets of data like total energy yesterday, this month and last month. Think I have debugged where to find the main firmware version and yesterdays data, but my energy reading does not match what my app claims to be the current production level. Any way I can contribute?

[XtheOne]

I would really need some trace data. I still did not find what magic string would trigger the logger. In the mean time I am working on a listen server which waits for a logger message and then processes the message. You can then add a server in the logger pointing to the python server IP/port

[stenjo]

Solis Three phase communication protocal.pdf Got hold of this document. Will that cover Version 5 of the protocol? Looks to be Version 7...

[XtheOne]

The Wi-Fi data logger communicates to the inverter trough RS485, the protocol is MODBUS. The IP network frames are in iGEN own format. And V4 format has been in use for a long time but V5 was recently introduced. I first found it on the new wired ethernet logger module from Omnik. I hope that they will implement it in the android app soon. I know the sent frame format, but the V5 frame which has to be sent to the logger to trigger it is still unknown. You can monitor the frames with logger data which are sent to the public server with wireshark on your network.

[XtheOne]

I might get an ethernet logger myself later this year. No replies from Omnik or iGEN until now.

[nielsvn92]

Not sure if you're still working on this, but I tried your experimental branch in attempt to get my inverter working with your script, and I actually do get a response now. I didn't get that on the master branch.

This is the error I'm getting: 2019-01-10 17:42:13,020 INFO Loggers found on the network: ['192.168.1.xxx', '640xxxxxx']. 2019-01-10 17:42:13,020 INFO Connecting to logger with IP: 192.168.1.xxx and SN 640xxxxxx 2019-01-10 17:42:13,021 INFO connecting to 192.168.1.xxx port 8899 2019-01-10 17:42:19,127 DEBUG RAW sent Packet (len=15): a5:02:00:10:45:00:00:7b:8a:26:26:01:00:a9:15 E{&& 2019-01-10 17:42:19,164 DEBUG RAW received Packet (len=38): a5:19:00:10:15:00:18:7b:8a:26:26:01:01:29:da:0f:00:e1:98:00:00:c1:9b:27:5c:2b:45:52:52:3d:2d:31:0d:0a:0d:0a:f0:15 {&&)'\+ERR=-1 2019-01-10 17:42:19,164 DEBUG DATA len=640.0: Traceback (most recent call last): File "LiveStats.py", line 17, in <module> inverter_exporter.run() File "/<path>/Inverter-Data-Logger/InverterExport.py", line 137, in run if (msg.msg)[:9] == 'DATA SEND': File "/<path>/Inverter-Data-Logger/InverterMsg.py", line 88, in msg return self.__get_string(self.Frame_offset + 0, self.len + self.Frame_offset + 0) File "/<path>/Inverter-Data-Logger/InverterMsg.py", line 28, in __get_string return self.raw_msg[begin:end] TypeError: slice indices must be integers or None or have an __index__ method

I have an Omnik 3K TL3, and the Wi-Fi data logger has firmware version H4.01.51MW.2.01W1.0.64(2018-01-251-D). I too am not able to configure any other remote server.

[ppiwowar]

nielsvn92, I am glad to know someone with the same wifi firmware version ( H4.01.51MW.2.01W1.0.64(2018-01-251-D) ) got ANY answer at last. It must be a kind of error-frame as I see +ERR=-1 asscii sequence at the end of your received packet .... Maybe your inverter is not connected or sent packet is still invalid. I will check myself in few hours ... XtheOne, your advice, please.

[ppiwowar]

After experimenting with our omnik/wifi we get the same error result. I think we have a problem with calling packet frame.

[XtheOne]

The problem is that I do not have a new WiFi module or able to get a firmware update from Omnik. I asked them several times for a firmware file. This started with the new Ethernet module which had the first buggy implementation of the V5 protocol from iGEN. This interface crashes on the magic packet so we stopped there. It looks that this new WiFi firmware also has V5 but is responding to the magic frame. I will look what data it returns. It is also strange that entering more than the default server is not possible anymore. This is why the server script was made to also support in decoding the V5 frame layout. Until now I only got some information of the frame layout from the android app. but the implementation there is incomplete and won't work.

[XtheOne]

ERR=-1 probably shows that the V5 magic frame is not correct. What does scanloggers returns?

[ppiwowar]

We did not use scanloggers.py. LiveStats.py with setting in the config file: gateways = auto did not discover any logger automatically. Only setting logger manually in the config file: gateways = 192.168.1.160,638xxxxxx found the logger and returned the frame. We will check with scan loggers.py too. Could you please specify, what is the 'android app' you use so we could check it here. Standard Omnik app in the manual mode does not give the access to electricity parameters of inverter. It gives a possibility to change some config parameters .... General question: Has Omnik may limited access to the inverter by third party software so the portal is the only option?

[XtheOne]

iGEN is the company which creates the logger. they upgraded the V4 protocol to V5 last year. The app on android I use is from Omnik and the one from iGEN which is newer and has some parts about V5 but incomplete. Until now no luck on getting a firmware update file for the newer loggers... Or any documentation. I hoped that someone could enter my IP in their logger so I would get some frames to analyse.

[nielsvn92]

What we are able to do is setting the internal server of port 8899 to client mode, and insert any IP/port as server. I've tried it before, but it seems there is no data coming out. If I remember correctly there was only the following TCP flow:

• Handshake
• Directly FIN, FIN-ACK (I believe from both sides, will have to double check)

So no data.

If you want, I could move my logger to another subnet and grant you access to the inverter WiFi module via a VPN @XtheOne? Or would physical access be required? When I had the gateway set to auto, it could detect my logger btw with the correct SN.

Is there anything else we can try for you?

[ppiwowar]

I have already tried setting the internal server of port 8899 to client mode before. Linux or Node-Red have got connected but no data coming in - exactly as with your trial... The only thing comes to my mind is sniffing .... ? Or else could I possibly downgrade WiFi card firmware ? Risky, as it might not talk to the inverter any more ....

[nielsvn92]

I used Wireshark to capture the data, and some program to use as a TCP server. It could of course be because of the server, because it doesn't send a message though. Not sure how the protocol works.

I have checked for firmware before, but could not find anything. But looking at XtheOne's messages, he has found firmware before. Perhaps a downgrade would work indeed.

[XtheOne]

This implementation has a few tasks. Decode V5 frames, these frames are the same as sent to the public site. Connect to the local logger and trigger it to sent data by sending a magic frame. Update listen server.

[XtheOne]

Recently I asked for an update file but they asked me for the SN and a picture of my logger. I did not yet respond as I don't have the newest logger.

[nielsvn92]

If you have some way for me to send the magic frame to my local logger, e.g. run some Python script or something, I can show you the results from Wireshark, or even send you a capture file?

I can also supply you with the SN and picture of my logger if you'd like, in a DM.

[XtheOne]

For testing I set my logger @home to sent data to my office and test the decoder remotely. Under Advanced > Server B: 46.182.220.24 Port 10004/TCP I understand now that this is not possible with these new loggers? This would enable me to determine the offsets of the data block.

I don't know if the newer WiFi module is still based on the Hi-Fly HF-A11 module.

[nielsvn92]

We have two different options, I might be able to share screenshots when I get home to clarify. The options are (both under Advanced):

• Remote server -- this is set to some default value which we can't change, and is probably the Omnik portal. We can't add a remote server as well.
• Port settings (by heart, have to double check the exact text) -- this can be set to Client or Server. By default this is Server, and port 8899. We can also set this to client, and try and point it to your server (46.182.220.24:10004?)

Which one do you mean?

[ppiwowar]

I am very much interested in the result of second option as I have tried it some time ago. Setting: client with ipaddress and port. Receiving server (RaspberryPi) was indicating the connection but no further data coming in.... Maybe the protocol needs some starting frame to be sent? Please tell me your result.

[nielsvn92]

I just enabled the second option and selected your server and port. It should send data now, or at least try to.

Edit: it only sends data when the inverter is generating electricity of course... good to know. It didn't send any data yet, I'm checking with Wireshark.

[ppiwowar]

Nielsvn92, have you got any data transmission?

[nielsvn92]

I don't know. I can't sniff when I'm not at home. I also came to the conclusion that I'm not going to see the network traffic to another host than mine without doing something with monitor mode on my WiFi card.

I think it's best to check with @XtheOne if he has received data today from me on the specified host/port.

[XtheOne]

Sorry guys, mu untangle server was down due to bad memory in that server. It is up now and I will start the omnik server project now.

[nielsvn92]

Please let me know if you at least receive some network traffic. Preferably data of course, but it's also good to know if you receive any TCP packets at all.

[ppiwowar]

Nielsvn92, you may also want to direct the traffic to: piwowar.dynalias.com:8989 My server is up & running all the times.

[nielsvn92]

I can't send it to two servers, only one.

[ppiwowar]

Sure, I understand ..... Are you sending to me now?

[nielsvn92]

No to XtheOne, since he's trying to figure out the data structure for us :p

[ppiwowar]

OK. Understood. Is he getting any data ?Regards, Pawel W piątek, 18 stycznia 2019, 13:29:02 CET, nielsvn92 notifications@github.com napisał(-a):

No to XtheOne, since he's trying to figure out the data structure for us :p

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[nielsvn92]

I don't know, the only communication we have is via this ticket.

[nielsvn92]

Perhaps it's easier to use Discord, instead of chatting in this issue? I created a server which you can join: https://discord.gg/URK9pP

[XtheOne]

I am working on this now and I just got to the point that I get the ERR=-1 after sending the magic frame. Maybe a wireshark trace between the logger and the omnik portal can show hot the portal starts the data frame reply. It looks that the magic packet is not ok. I first got connected, but no data was sent.

[loekd]

By adding an additional 0x00 to the endcode, making the frame 16 bytes in length I've managed to get a series of 29 byte messages back from the inverter. No idea yet about how to parse it. InverterLib.py ln 65, change into:

    endCode = '\x15\x00'

Additional perk; sending a 16 byte frame, the module no longer crashes.

[ppiwowar]

loekd, Could you possibly paste the hex dump of this 29 bytes here, please ?

[XtheOne]

Interessting, it is not in the java code. But it is still not working. Here a sniplet of the code which encodes a frame.

public byte[] encode() {
    ChannelBuffer buf = ChannelBuffers.buffer(ByteOrder.LITTLE_ENDIAN, getLength());
    buf.writeByte(this.headCode);
    this.dataFieldLength = createDataFieldLength();
    buf.writeShort(this.dataFieldLength);
    buf.writeShort(this.contrlCode);
    buf.writeShort(this.serialNumber);
    buf.writeInt(this.sn);
    this.dataFields = createDataField();
    buf.writeBytes(this.dataFields);
    buf.writeByte(EncodeUtil.getCheckDigit(buf, 1, buf.readableBytes()));
    buf.writeByte(this.endCode);
    return buf.array();
}

[loekd]

@ppiwowar here you go:

"A5-10-00-10-15-00-05-0D-2E-E3-25-01-01-10-24-0C-00-25-6B-00-00-00-00-00-00-01-00-50-15"
"A5-10-00-10-15-00-06-0D-2E-E3-25-01-01-1E-24-0C-00-56-6B-00-00-00-00-00-00-01-00-90-15"
"A5-10-00-10-15-00-07-0D-2E-E3-25-01-01-20-24-0C-00-5D-6B-00-00-00-00-00-00-01-00-9A-15"
"A5-10-00-10-15-00-08-0D-2E-E3-25-01-01-21-24-0C-00-62-6B-00-00-00-00-00-00-01-00-A1-15"

@XtheOne based on that I can't say how many bytes are written into buf. I just figured it may crash because of a mismatch between expected frame length and actual length. This seems to be the case. Do you have code that parses the returned bytes?

[ppiwowar]

What I see here is 4 x the magic packets, I guess, but not a response frame. Are you getting any response? You mentioned a frame with 29 bytes...?

Pozdrawiam/Regards, Pawel

Wiadomość napisana przez loekd notifications@github.com w dniu 10.02.2019, o godz. 09:54:

@ppiwowar here you go:

"A5-10-00-10-15-00-05-0D-2E-E3-25-01-01-10-24-0C-00-25-6B-00-00-00-00-00-00-01-00-50-15" "A5-10-00-10-15-00-06-0D-2E-E3-25-01-01-1E-24-0C-00-56-6B-00-00-00-00-00-00-01-00-90-15" "A5-10-00-10-15-00-07-0D-2E-E3-25-01-01-20-24-0C-00-5D-6B-00-00-00-00-00-00-01-00-9A-15" "A5-10-00-10-15-00-08-0D-2E-E3-25-01-01-21-24-0C-00-62-6B-00-00-00-00-00-00-01-00-A1-15" @XtheOne based on that I can't say how many bytes are written into buf. I just figured it may crash because of a mismatch between expected frame length and actual length. This seems to be the case. Do you have code that parses the returned bytes?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.