search

XtheOne / Inverter-Data-Logger

Data logger for Omnik/Hosola and other Solarman Wi-Fi kit powered Solar Inverters
GNU General Public License v3.0
116 stars 28 forks source link

Add Version 5 protocol support #3

Open XtheOne opened 6 years ago

XtheOne commented 6 years ago

Implement support for V5 protocol as used in new embedded ethernet loggers. Needs documentation from Omnik and or iGEN / Solarman.

This is requested. Also a firmware update file is requested.

loekd commented 5 years ago

These are some 29 byte responses.

XtheOne commented 5 years ago

This is what I make of these frames. Data seems only something with time, could be a TM frame. headCode A5 (-91) dataFieldLength 10 00 (16) contrlCode 10 15 (21/16) serialNumber 00 05 (5) sn 0D 2E E3 25 (635645453) dataField 01 01 10 24 0C 00 25 6B 00 00 00 00 00 00 01 00 (...$..%k........) (fr6) 01-01-1E-24-0C-00-56-6B-00-00-00-00-00-00-01-00 (fr7) 01-01-20-24-0C-00-5D-6B-00-00-00-00-00-00-01-00 (fr8) 01-01-21-24-0C-00-62-6B-00-00-00-00-00-00-01-00 checksum 50 (80) endCode 15 (21)

XtheOne commented 5 years ago

contrlCode 0x1510 (1010100010000) version = V5 frametype = 5 sendOrReplyCommand = 1 (Reply)

nielsvn92 commented 5 years ago

@XtheOne could you perhaps check Discord? You're missed there :-).

To update everyone on what we've tried to do as well: I tried the Solarman app to connect to the inverter locally, but even the app can't connect. I have the same result as with the script in this repo.

pbouwen3 commented 5 years ago

@nielsvn92 any chance you could share the discord invite again? It would be great to be able to follow up.

Thanks!

nielsvn92 commented 5 years ago

@pbouwen3 https://discord.gg/krjReA

peterzandbergen commented 5 years ago

I have been sniffing the packets that the wifi module sends to the Omnik site. Will that information help? I see a lot of different sized packets being sent. Let me know.

lennardk commented 5 years ago

Same as peter I have access to packet dumps, in my case for firmware H4.01.51MW.2.01W1.0.64(2018-01-251-D) connected to a Omnik3000tl. Happy to join you guys on discord if that's still going on.

XtheOne commented 5 years ago

I really have no time now to work on this project. Maybe in a few weeks I would get some time but for now I have to get a big project done...

stingone commented 5 years ago

Any update on this the V5 protocol? :)

picanl commented 4 years ago

any updates??

MrClever commented 4 years ago

While I'm waiting for this to be resolved, I've hacked an egregious obscenity that polls the inverter's web interface and scrapes the current power and energy production. This is then fed into a python script which cleans up the output a little more and does the upload to PVOutput...or you can roll your own to go somewhere else. The SoFar web interface (on my inverter) doesn't expose all the data available (like volatages, current, grid AC freq. etc) but it's better than nothing ¯\_(ツ)_/¯

I take no responsibility for this; it was born out of desperation!

#!/usr/bin/env bash

username=admin
password=admin
inverter=127.0.0.1 # Change to your inverter's IP/hostname
curlOpts="-s -o - --user ${username}:${password}"

poll(){
    curl ${curlOpts} http://${inverter}/status.html 2>&1 |\
    egrep '^var webdata_(now|today)' |\
        sed 's/var webdata_//' |\
        sed 's/["\;\=]//g' |\
        sed 's/\r$//g'
}

output=$(poll)

if [[ ! -z "${output}" ]]; then
    echo -e "${output} OK"
else
    echo "FAIL"
fi

To see what else is exposed try curl -s -o - --user admin:admin http://<your_inverter>/status.html

If anyone is interested, I can share my Python PVOutput uploader too.

droneando commented 4 years ago

Hi! My wifi stick serial number starts with 0522xxxxx and I'm having the time outs responses from your script. Any update in getting it to work with V5 protocol? In the meanwhile do you think it's possible to extract the inverter data from the solarman web page where this info is shown to the user?

netadair commented 4 years ago

I have tested it with a RHI inverter and found this so far:

ppiwowar commented 4 years ago

Yes, please.

netadair commented 4 years ago

Yes, please share the documentation with us, if this is ok with you !

I also did some further analysis of the data sent back and forth between the stick and the portal, but so far didn't find a way to actively poll the stick. The data I found resembles boot info, WLAN info, inverter data, a ping and alert messages, but all actively send from the stick to the portal and just acknowledged there.

XtheOne commented 4 years ago

TNX!!!

XtheOne commented 4 years ago

I had a quick look and I think I see how to trigger (pierce) the logger to sent data. Will try this later. Looks similar as the android code but that is still V4 only...

ppiwowar commented 4 years ago

Could you be so kind and help an old boy with ZERO java knowledge how to trigger or what is the magic frame, please.

stravinci commented 4 years ago

I think you need this:

Pierce pi = new Pierce();
int timeZone = 60;
short comandType = 0x01;
String content= "abcd";
int datalogger = 501000001;
int ssn = 10;
int csn = 2;
boolean needAck = true;
System.out.println("       result : "+Util.bytesToHexString(pi.createPierce(60, 3,comandType, content, datalogger, csn, ssn, needAck)));
ppiwowar commented 4 years ago

I appreciate your quick answer, but this is still too difficult for me. Could you possible translate this to Python or (ideally) C so I could use it "independently" of SDK (like in XtheOne logger). Thank you for your patience. FYI: My logger is based on ESP32 sniffing (hack in) TCP packets flowing from inverter to the portal. I look forward to able too trigger inverter locally.

stravinci commented 4 years ago

@ppiwowar sorry for deayed response. I publish this SDK because I do not have many time to understand and implement this solution. :) But I decode java code from jar file, and here i createPierce implementation:

public byte[] createPierce(final int timeZone, final int x0, final short sensor, final String context, final int datalogger, final int csn, final int ssn, final boolean needAck) {
        if (x0 <= 0 || x0 >= 16) {
            Pierce.LOG.log(Level.SEVERE, "x0 shoud ge 1 and le 15  ...");
        }
        if (csn <= -1 || csn >= 256) {
            Pierce.LOG.log(Level.SEVERE, "csn shoud ge 0 and le 255  ...");
        }
        byte[] res = null;
        final PlanAck ack = new PlanAck();
        final int now = (int)TimeUtils.getTimeInMillis() / 1000;
        final String collectorNum = datalogger + "";
        final String ruleCode = "0105" + Util.strZeroPadding(Integer.toHexString(x0), 2, true);
        ack.setVersion(1).setCtrl1bit3(false).setEncryptType(0).setHasNext(false).setClose(false).setCtrlNumC(csn).setCtrlNumS(ssn).setIsAck(false).setNeedAck(needAck).setFrameType(5).setRuleCode(ruleCode).setResponseRule(RULE.rrMap.get(ruleCode)).setCollectorNumber(collectorNum);
        final JSONObject rule = ack.responseRule;
        ack.responseVal.put(Constants.CACHE_OUTERPARSE_RESPONSE_NOW, (Object)now);
        ack.responseVal.put(Constants.CACHE_OUTERPARSE_RESPONSE_TIMEZONE, (Object)timeZone);
        ack.responseVal.put(Constants.CACHE_OUTERPARSE_RESPONSE_X0, (Object)x0);
        ack.responseVal.put(Constants.CACHE_OUTERPARSE_RESPONSE_DEVICETYPE, (Object)Integer.toHexString(sensor).toUpperCase());
        ack.responseVal.put(Constants.CACHE_OUTERPARSE_RESPONSE_ACK, (Object)"");
        ack.responseVal.put(Constants.CACHE_OUTERPARSE_RESPONSE_DATA, (Object)context);
        if (rule != null) {
            final ByteBuf buf = Util.createAck(ack);
            res = new byte[buf.readableBytes()];
            buf.readBytes(res);
            buf.release();
        }
        return res;
    }

    static {
        LOG = Logger.getLogger(Pierce.class.getName());
    }

@XtheOne how is your progress to implement it? Do you have any plan?

stravinci commented 4 years ago

@ppiwowar how is your progress? Maybe we can work together?

ppiwowar commented 4 years ago

Great idea .... for these days! I think this is all about sending the right magic frame to inverter. In V4 it was simple:

stravinci commented 4 years ago

Can we switch to WhatsApp/Massanger or any another communication way? In my profile you should see my e-mail.

stravinci commented 4 years ago

Regarding to your last post: I think that when we set server address in logger settings then it should ty to connect with our server, but my implementation of simple listener on TCP port does not any packet from logger.

I know that logger response on port 8899, but in my opinion it is wrong way to implement it - solarman page does not send any data at this port to logger, because logger is responsible to open connection between logger-server.

-- Dawid

pawelka commented 4 years ago

FYI: Hi, I found different way to handle v5 protocol communication. I had to decode only one package with sensors data. The result is here https://github.com/pawelka/hassio-addons as an add-on to home-assistant, but it can be used as standalone integration to mqtt in docker.

ppiwowar commented 4 years ago

Hi PawelK, great hint. I have done an ESP32 WiFi sniffer so I can catch these packets while they are traveling to the portal. Your trick is more elegant, though. Please specify, if/how could you redirect these packets to the original portal after catching them. (I am a python beginner). Thanks. Stravinci(David), have you figured out, how/if this java software does send any packet(data) to initialise logger to respond locally. Or maybe it talks to the portal directly? I kindly repeat: We struggle here to find the ability to trigger the logger locally (, as it was before) to get the data. Thanks & regards, Pawel

pawelka commented 4 years ago

Wifi sticker connect to my addon. Addon connect to cloud. Application transfer data from one socket to another and if package has specific size decode it and send result to Mqtt.

The idea with esp32 and wifi sniffer is also interesting ;-)

stravinci commented 4 years ago

@ppiwowar this java code does not send any data - this is only packet parser. @all I think that @pawelka solution is fine, but we need full solution. As example why we need it you can search information about Samil inverters - they shut down servers on October 18, reportedly bankrupt, my father has their inverter, I was able to find a ready solution talking to the inverter, unfortunately our Sofar is harder to hack, but in my opinion we have to protect ourselves.

Mystercoco commented 3 years ago

Hi, I checked on my data logger and I've found a way to make it do a reverse TCP connection to my computer. So here are steps:

  1. Send an UPD packet to the data logger's IP (or broadcast) on port 58899 containing the text: set>server=10.42.0.1:8899; with 10.42.0.1 being the IP of the computer and 8899 the port where you want the data logger to connect to (tcp).

  2. Get the response, the data logger will send an UPD response with the text rsp>server=1; to ack.

  3. Listen on the corresponding TCP port (the one sent to data logger by the first UDP message). The data logger will initiate a TCP connection to it (I struggled because my firewall was blocking it)

  4. Now, the tool is connected to the data logger and can send commands.

To do the testing, I was able to modify the "ScanLoggers.py" to do all steps above. The TCP command P007PIR (19fc0001000eff045e5030303750495249ee380d) looks like giving useful data (grid, power, battery...).

Here under my test code, giving the result (inverter is a 220v, 5kw, 48vc battery hybrid): This script will look for iGEN WiFi Kit loggers from SolarMAN PV List their IPs and S/Ns and connected inverters S/N These loggers are found in Omnik, Hosola, Ginlong, Kstar, Seasun, SolaX, Samil, Sofar, Trannergy and other Solar inverters WiFi kit logger found, IP = ('10.42.0.115', 58899) and S/N = Listing Inverter(s) connected to this WiFi logger Inverter main CPU = 220,00000,00000> Inverter main firmware version: = %s Inverter slave firmware version: = �_�^D0882300,217,2300,500,217,5000,5000,480,460,540,420,564,540,0,30,060,0,0,1,9,0,0,1,0,1,00j� closing socket, scanning done!

code:

SendData = "set>server=10.42.0.1:8899" # Lotto/TM = "AT+YZAPP=214028,READ" try: # Send data to the broadcast address serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) serversocket.bind(("10.42.0.1", 8899)) serversocket.listen(5) sent = sock.sendto(SendData.encode('utf-8'), ('10.42.0.255', 58899))

# Look for responses from all recipients while True: try: data, server = sock.recvfrom(1500) except socket.timeout: break else: if (data == SendData.encode('utf-8')): continue #skip sent data if (data != "rsp>server=1;"): continue #not a data logger #a = data.split(b',') logger_ip = server logger_mac ="" logger_sn = "" sys.stdout.write('WiFi kit logger found, IP = %s and S/N = %s\n' % (logger_ip, logger_sn)) #data = InverterLib.createV4RequestFrame(int(logger_sn)) # Connect the socket to the port where the server is listening #logger_socket.connect((logger_ip, 8899)) serversocket.settimeout(3) (logger_socket, address) = serversocket.accept() logger_socket.send('197d0001000dff045e50303036564657f6e60d'.decode('hex')) sys.stdout.write('Listing Inverter(s) connected to this WiFi logger\n') okflag = False while (not okflag): data = logger_socket.recv(1500) msg = InverterMsg.InverterMsg(data) okflag = True

if (msg.msg)[:9] == 'DATA SEND': logger_socket.close() okflag = True continue

sys.stdout.write('Inverter SN = %s\n' % msg.id) sys.stdout.write('Inverter main firmware version: = %s\n') sys.stdout.write('Inverter slave firmware version: = %s\n' % msg.slave_fwver) #test parsing logger_socket.send('19fc0001000eff045e5030303750495249ee380d'.decode('hex')) data = logger_socket.recv(1500) a = data.split(b',') #grid_rating_voltage,grid_rated_current,ac_output_apparent_power_VA,ac_output_rating_active_power_W,battery_rating_voltage,battery_minimum_voltage,battery_maximum_voltage print (data) logger_socket.close()

MrClever commented 3 years ago

Nice work :)

Just FYI, you can wrap code in triple backticks, ```, on Github like this: ``` multi-line code block ```

You can even specify syntax highlighting like this: ```python Import foo ```

That way, all the indentation and other formatting is retained. :+1: All the details are here.

nichcuta commented 3 years ago

Thanks @Mystercoco for sharing such information. I've been trying to replicate your behaviour without success :(.

Can you perhaps share the complete modified ScanLoggers.py script used and would you need to edit anything on the inverters side? Example changing the "Port Settings" from "TCP" to UDP?

Currently what i get after implementing the code is: "closing socket, scanning done!"

Thanks in advanced for such information.

Mystercoco commented 3 years ago

Thank you @MrClever, I am not working a lot with github. @nichcuta, please find the complete code here under. You need to change the ip 10.42.0.1 with your computer's IP and the IP 10.42.0.255 with your brodcast IP (in my case, the broadcast is for a network mask on 24 bits). If interested, I can also copy packets captured from wireshark.

#!/usr/bin/python
"""Logger search program.

Find an IGEN Wi-Fi kit logger and query the connected inverter(s).
"""
import socket  # Needed for talking to logger
import struct
import sys
import InverterMsg  # Import the Msg handler
import InverterLib  # Import the library

def get_inverter_sn(logger_sn, logger_ip):
    data = InverterLib.createV4RequestFrame(int(logger_sn))
#    print >>sys.stdout, 'DATA = %s' % data
    logger_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    logger_socket.settimeout(3)
    # Connect the socket to the port where the server is listening
    logger_socket.connect((logger_ip, 8899))
    logger_socket.sendall(data)

    print >>sys.stdout, 'Listing Inverter(s) connected to this WiFi logger'
    okflag = False
    while (not okflag):
        data = logger_socket.recv(1500)
        msg = InverterMsg.InverterMsg(data)

        if (msg.msg)[:9] == 'DATA SEND':
            logger_socket.close()
            okflag = True
            continue

        print >>sys.stdout, 'Inverter SN = %s' % msg.id
        print >>sys.stdout, 'Inverter main firmware version: = %s' % msg.main_fwver
        print >>sys.stdout, 'Inverter slave firmware version: = %s' % msg.slave_fwver

    logger_socket.close()

#main
sys.stdout.write('This script will look for iGEN WiFi Kit loggers from SolarMAN PV\n')
sys.stdout.write('List their IPs and S/Ns and connected inverters S/N\n')
sys.stdout.write('These loggers are found in Omnik, Hosola, Ginlong, Kstar, Seasun, SolaX, Samil, Sofar, Trannergy\n')
sys.stdout.write('and other Solar inverters\n')

# Create the datagram socket
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
# Set a timeout so the socket does not block indefinitely when trying to receive data.
sock.settimeout(6)
# Set the time-to-live for messages to 1 so they do not go past the local network segment.
ttl = struct.pack('b', 1)
sock.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_TTL, ttl)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
SendData = "set>server=10.42.0.1:8899" # Lotto/TM = "AT+YZAPP=214028,READ"
try:
    # Send data to the broadcast address
    serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    serversocket.bind(("10.42.0.1", 8899))
    serversocket.listen(5)
    sent = sock.sendto(SendData.encode('utf-8'), ('10.42.0.255', 58899))

    # Look for responses from all recipients
    while True:
        try:
            data, server = sock.recvfrom(1500)
        except socket.timeout:
            break
        else:
            if (data == SendData.encode('utf-8')): continue #skip sent data
            if (data != "rsp>server=1;"): continue #not a data logger
            #a = data.split(b',')
            logger_ip = server
            logger_mac =""
            logger_sn = ""
            sys.stdout.write('WiFi kit logger found, IP = %s and S/N = %s\n' % (logger_ip, logger_sn))
            #data = InverterLib.createV4RequestFrame(int(logger_sn))
            # Connect the socket to the port where the server is listening
            #logger_socket.connect((logger_ip, 8899))
            serversocket.settimeout(3)
            (logger_socket, address) = serversocket.accept()
            logger_socket.send('197d0001000dff045e50303036564657f6e60d'.decode('hex'))
            sys.stdout.write('Listing Inverter(s) connected to this WiFi logger\n')
            okflag = False
            while (not okflag):
                data = logger_socket.recv(1500)
                msg = InverterMsg.InverterMsg(data)
                okflag = True

                if (msg.msg)[:9] == 'DATA SEND':
                    logger_socket.close()
                    okflag = True
                    continue

                sys.stdout.write('Inverter SN = %s\n' % msg.id)
                sys.stdout.write('Inverter main firmware version: = %s\n')
                sys.stdout.write('Inverter slave firmware version: = %s\n' % msg.slave_fwver)
            #test parsing
            logger_socket.send('19fc0001000eff045e5030303750495249ee380d'.decode('hex'))
            data = logger_socket.recv(1500)
            a = data.split(b',')
            #grid_rating_voltage,grid_rated_current,ac_output_apparent_power_VA,ac_output_rating_active_power_W,battery_rating_voltage,battery_minimum_voltage,battery_maximum_voltage
            print (data)
            logger_socket.close()

finally:
    sys.stdout.write('closing socket, scanning done!\n')
    sock.close()
    serversocket.close()
    sys.stdout.flush()
nichcuta commented 3 years ago

@Mystercoco, thanks for the fast reply. Unfortunately nothing happens for me when i try to hit the broadcast address/Invert IP on port 58899. A quick UDP/TCP scan reviews that only TCP ports 80,8899 and UDP 53 are open. Now hitting the inverter with a full TCP/UDP scan perhaps that reviews some open ports.

In the meantime below please find the current inverter Port settings. Is a change here required or? image

PS: Inverter working mode is currently set to "data collection".

Thanks.

XtheOne commented 3 years ago

You are talking about the Voltronic Power protocol, this repo is about the iGEN V4/V5 protocol. Still there is no new help or info found as far as I know. My hopes for now is a firmware update file which covers the V5 protocol and reverse engineer this.

bobbythomas commented 3 years ago

So I am pretty new here, I have a Ginlong Wifi DLS stick ver 2.2, the firmware version is MW_08_512_0501_1.80, I am not sure if this is runnig v4 or v5 protocol, how can we identify that? I was able to capture some packets which are sent to the data1.Solarmanpv.com from my firewall. I have the packet capture in RAW/ASCII/YAML/HEX Dump formats.

Also I managed to host a VM in my network and setup a netcat session on port 10000 output saved on a text file to emulate the solarmanpv traffic, and then I went into the DLS config page http:///config_hide.html and modified the Server A and Optional server settings both poiting to my VM. After the reboot of the DLS, I was able to see a session being established to the inside VM on port 10000 but after couple of attempts DLS disconnects/terminate the TCP session, when I check the file output I see some scrambled text in it along with the IP address assigned to the DLS and the firmware version of the DLS. I believe DLS is trying to authenticate against the server and when it fails it disconnect the session (I could be wrong).

Any idea how we can decode the text?

Below is the output from the netcat session:

¥V A K{+­ú y <xHMW_08_512_0501_1.80 ˜ØcéH.192.168.1.250 ]¥V A K{+·ú ƒ <xHMW_08_512_0501_1.80 ˜ØcéH.192.168.1.250 r¥V A K{+Âú Ž <xHMW_08_512_0501_1.80 ˜ØcéH.192.168.1.250 ‰

My coding skills are very poor but I have some good networking skills. Let me know if any additional information is required.

osman-masood commented 3 years ago

Hi all, I'm also pretty new here, and have a couple LSW-3 loggers. The serial numbers start with 173. Does anyone know how it communicates with the remote server?

I've set up a basic TCP server (on AWS) and have configured it (via its web interface) to talk to this server, but am not seeing any connection or data exchanged. (I'm guessing that the remote server needs to initiate the connection, but how?)

Thanks!

ojeysky commented 3 years ago

Hi all, I'm also pretty new here, and have a couple LSW-3 loggers. The serial numbers start with 173. Does anyone know how it communicates with the remote server?

I've set up a basic TCP server (on AWS) and have configured it (via its web interface) to talk to this server, but am not seeing any connection or data exchanged. (I'm guessing that the remote server needs to initiate the connection, but how?)

Thanks!

I have similar logger, were you able to make some progress with it?

stravinci commented 3 years ago

In my opinion it is wrong direction. Client open communication to server, not the other way around. Our loggers has hardcoded about 3 solarman server addresses, so if you want to sniff it or use custom server you need to give him your local DNS server to resolve solarman domain as your IP.

osman-masood commented 3 years ago

@ojeysky , no progress. Tried a TCP server and it didn't receive any data or even connection at all. Maybe the way the protocol works is, the server initiates the connection to the client? In any case, in order to reverse engineer the protocol we would need to listen in on the connection between the Logger and Solarman's servers (assuming HTTPS/TLS is not used).

@stravinci , You're right, but why should we need to put in a DNS server, if we're configuring the Logger to communicate directly with our server anyway? A DNS server would only be necessary if the forwarding functionality is broken.

stravinci commented 3 years ago

I set it in logger but it does not communicate to server. I read some topic at hm... Danish or Dutch forum where they try to do it in this way (using DNS). Do we have here some java people who can analyze code which I got from V5 loggers support?

bobbythomas commented 3 years ago

How do you guys know the version of the logger protocol? I was able capture packets from the logger to the solarmanpv.com portal but I wasn't able to find anything useful, how can I determine if it's using Version 4 or version 5?

Please see my response above regarding the captures.

nielsvn92 commented 3 years ago

I set it in logger but it does not communicate to server. I read some topic at hm... Danish or Dutch forum where they try to do it in this way (using DNS). Do we have here some java people who can analyze code which I got from V5 loggers support?

I'm a Java Developer, so I should be able to analyze the code. Is this the code you're referring to? If so, it would be good to have the whole JAR instead of one function. https://github.com/XtheOne/Inverter-Data-Logger/issues/3#issuecomment-597001666

stravinci commented 3 years ago

SDK Instruction.zip @nielsvn92 all hope in you ;-)

pawelmuszynski commented 3 years ago

I got a packet with 168 bytes of payload using Wireshark that is sending every 5 minutes, but no idea how to parse it. It contains Inverter S/N, firmware version and some other data.

stravinci commented 3 years ago

Paweł please look at documentation attached by me.

mmaciejow commented 3 years ago

Hi, I have Solis 4G inverter with Ginlong stick wifi with "Device serial number: 4023xxxx". I wonder how can i get data from inverter. How can i check which protocol is it?

I checked this repo and Omnikol-PV-Logger by t3kpunk and the data received is incorrect.

On site IP/config_hide.html i see "Internal server parameters setting". I connected to data logger ip:8899 via Hercules utility (TCP Client) and I founded in internet pdf: "RS 485 Communication Protocol(Version 2) (GCI-1.5K/GCI-2K/GCI-3K/GCI-3.6K/GCI-4.6K/GCI-5K) Sep. 9, 2011"

There is an example: The PC sends data: 7E 02 A1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 A3

And receives data: 7E 02 A1 1C 72 06 21 00 FF 08 18 00 20 01 61 10 00 00 00 00 71 71 01 02 88 13 01 01 3B 0B 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 08

but when I send the data nothing happens. Maybe you know how to send and receive data?

Thanks for help.

jessiesolar commented 3 years ago

I also have an Omnik with the H4.01.51MW.2.01W1.0.64(2018-01-251-D) firmware.

Thanks to the excellent analysis and code from https://www.algodynamic.co.uk/reverse-engineering-the-solisginlong-inverter-protocol.html, I managed to reconstruct the 23-byte server response to the 99-byte 'heartbeat' packet, after which the Omnik inverter starts sending 228-byte data packets.

I have not been able to fully parse the returned data yet, but extracting the main fields like temperature, v_pv1, i_pv1, v_ac1, frequency, e_today, e_total, firmware versions, ... seems to work.

I hope the community here can help complete the analysis!

Steps:

  1. Have the omnik inverter send its traffic to a server your control, either by redirecting the DNS name, or by reconfiguring the IP in http:///config_hide.html.

  2. (Optional) Run https://github.com/planetmarshall/solis-service/blob/main/scripts/intercept.py (change '192.168.10.9', 19042 to IP of your server, 10000). This will listen for any data from the Omnik inverter, forward it to data1.solarmanpv.com, and print out the communication. This will capture the 23-byte answer from data1.solarmanpv.com to the 99-byte heartbeat packet. This helps understanding the exact format of the 23-byte answer, so that we can reconstruct this ourselves in a later step, without involvement of data1.solarmanpv.com. Capturing this traffic using Wireshark might be useful as well. This step is optional, as the 23-byte format can probably be derived from the 99-byte heartbeat, but it helps to have an original response as reference. EDIT: I think I found how to derive the right info directly from the request - see step 4.

  3. Edit https://github.com/planetmarshall/solis-service/blob/main/solis_service/server.py, and change return len(message) == 246 to return len(message) == 228 (the omnik inverter sends 228 bytes instead of 246).

  4. Edit https://github.com/planetmarshall/solis-service/blob/main/solis_service/messaging.py, and change both header = b'\xa5\n\x00\x10' and prefix = b'\x01\xc2\xe8\xd7\xf0\x02\x01' based on your findings in step 2. UPDATE: header seems to be fixed, only prefix needs updating. The actual value can be derived from step 2 as follows:

$ python3
>>> import base64
>>> bytes = base64.b64decode('<23-byte data field of response>')
>>> header = bytes[0:4]
>>> prefix = bytes[6:13]
>>> header
b'\xa5\n\x00\x10'
>>> prefix
b'[..]'

UPDATE2: the prefix can also be derived directly from the 99-byte heartbeat packet:

$ python3
>>> import base64
>>> bytes = base64.b64decode('<99-byte data field of heartbeat>')
>>> prefix = bytes[6:12] + b'\x01'
>>> prefix
b'[..]'
  1. (Optional) Remove the references to the persistence service from https://github.com/planetmarshall/solis-service/blob/main/solis_service/server.py to make it easier to run this in isolation for quick testing. UPDATE: also avoid an ImportError by fixing the reference to messaging:
-from .messaging import (
+from messaging import (
  1. Copy https://github.com/planetmarshall/solis-service/blob/main/config/solis-service.conf into the same directory as server.py, and change:
[service]
hostname = <ip of your server>
port = 10000
  1. Run: python3 server.py. It can take a couple of minutes before you receive a first data message. Output should be similar to:
$ python3 server.py 
2021-04-10 08:17:27,987 - solis_service - INFO - Starting server on <server ip>:10000
2021-04-10 08:18:05,300 - solis_service - DEBUG - Received heartbeat message from ('<inverterip>', port)
2021-04-10 08:18:15,292 - solis_service - DEBUG - Received heartbeat message from ('<inverterip', port)
2021-04-10 08:18:28,341 - solis_service - DEBUG - Received data message from ('<inverterip>', port)
2021-04-10 08:18:28,342 - solis_service - DEBUG - data message: {'inverter_serial_number':
[..]
  1. Tune parse_inverter_message to properly parse out all fields. https://github.com/planetmarshall/solis-service/blob/main/solis_service/messaging.py is a big help here. I am still cleaning up my parsing, but I will add it later. EDIT: these work for me:
def parse_inverter_message(message):
    return {
        "inverter_serial_number":           message[32:48].decode("ascii").rstrip(),
        "inverter_temperature":             0.1 * unpack_from("<H", message, 48)[0] * ureg.centigrade,
        "v_pv1":                            0.1 * unpack_from("<H", message, 50)[0] * ureg.volt,
    "i_pv1":                            0.1 * unpack_from("<H", message, 54)[0] * ureg.amperes,   
        "i_ac1":                            0.1 * unpack_from("<H", message, 58)[0] * ureg.amperes,
        "v_ac1":                            0.1 * unpack_from("<H", message, 64)[0] * ureg.volt,
        "f_ac1":                            0.01 * unpack_from("<H", message, 70)[0] * ureg.hertz,
        "p_ac1":                            unpack_from("<H", message, 72)[0] * ureg.watts,
        "e_today":                          0.01 * unpack_from("<H", message, 76)[0] * ureg.kilowatt_hour,
        "e_total":                          0.1 * float(unpack_from("<I", message, 80)[0]) * ureg.kilowatt_hour,
        "mode":                             unpack_from("<H", message, 88)[0], #1=on, 2=off
        "main_fwver":                       message[106:126].decode("ascii").rstrip(),
        "slave_fwver":                      message[126:146].decode("ascii").rstrip(),
        "other_fwver_1":                    message[146:166].decode("ascii").rstrip(),
        "other_fwver_2":                    message[166:186].decode("ascii").rstrip(),
        "other_fwver_3":                    message[186:206].decode("ascii").rstrip(),
    "other_fwver_4":                    message[206:226].decode("ascii").rstrip()
    }

Enjoy!

silvanverschuur commented 3 years ago

@jessiesolar I would like to help decoding the packets but my knowledge of Python is very limited. Can you explain a little bit more about the header/prefix values in step 4? The result of step 2 is:

serving on ('192.168.1.51', 10000)
{"timestamp": "2021-04-10T11:11:27.605994", "target": ["47.88.8.200", 10000], "data": "pVYAEEEAAYojXCUC9hz7AhQAAAAAAAAABTx4AWQBSDQuMDEuNTFNVy4yLjAxVzEuMC42NCgyMDE4LTAxLTI1MS1EKQAAALxU+fcsqTE5Mi4xNjguMS4xOAAAAAAAAAECAeIV", "length": 99}
{"timestamp": "2021-04-10T11:11:27.755115", "target": ["192.168.1.18", 36486], "data": "pQoAEBEwAYojXCUCAb9rcWCqqgAA3BU=", "length": 23}

How do I convert this result to a header/prefix value?

I'm also having trouble running the python3 server.py command. It complains about imports:

Traceback (most recent call last):
  File "/media/data/solis-service/solis_service/server.py", line 9, in <module>
    from .messaging import (
ImportError: attempted relative import with no known parent package

It would be great if you can help me get it running. Thanks!