jbuehl / solaredge

SolarEdge inverter logging data capture
GNU General Public License v3.0
289 stars 60 forks source link

Monitoring pasively performance data via LAN by duplicating the traffic in the router #27

Open abdullahtahiriyo opened 8 years ago

abdullahtahiriyo commented 8 years ago

Hi!

This is not the description of a problem. It rather provides some insight on an alternative way to retrieve the performance data passively using the LAN.

My SolarEdge inverter is connected with an Ethernet cable directly to the router (I have it on an isolated VLAN that goes directly to the internet, but this is not a requirement for employing this method). My router has the ability to mirror traffic using iptables -j ROUTE -j tee. Many linux enabled routers, such as those using "DD-WRT v24-sp2" have an iptables version new enough (from 2012?) to support this (dd-wrt is not a requirement, iptables is). The idea is to configure the router to make a duplicate of the traffic between the inverter and solaredge and send this to the server where I do the monitoring (Note some router/switches have (ethernet) port mirroring built in, this is another different option, mine has not). So the setup (I hope it renders correctly):

SolarEdge Inverter (192.168.120.100) ==== ROUTER ==== SolarEdge Website ..........Monitoring server (192.168.100.120) ===||

Configuring the router to duplicate traffic

These are the iptables rules enabling duplication of traffic (to be added to the firewall):

iptables -A PREROUTING -t mangle -d 192.168.120.100 -p tcp -j ROUTE --gw 192.168.100.120 --tee iptables -A PREROUTING -t mangle -s 192.168.120.100 -p tcp -j ROUTE --gw 192.168.100.120 --tee iptables -A POSTROUTING -t mangle -d 192.168.120.100 -p tcp -j ROUTE --gw 192.168.100.120 --tee iptables -A POSTROUTING -t mangle -s 192.168.120.100 -p tcp -j ROUTE --gw 192.168.100.120 --tee

Basically says everything that before or after routing has a destination or source from the inverter, make a copy and send the copy to the monitoring server (leave the original unaffected). The trick is that the router changes the destination MAC address to that of the monitoring server (OSI level 2), without changing higher levels, so the IP addresses are unaffected by the duplication of the traffic.

Feeding the result into seamonitor

On the monitoring server, you can do: sudo tcpdump -i eth0 -s 65535 -w - 'tcp and host 192.168.120.100' | python /root/solaredge/semonitor.py -vv | tee output.json | python /root/solaredge/se2state.py -o solar.json

Basically tcpdump captures all the data from the Ethernet interface, filters out so that only tcp traffic to/from the inverter is passed to semonitor, and writes in binary format to the pipe (to semonitor). Semonitor converts the TCP traffic into JSON, and se2state maintains the latest performance data.

For some reason I did not need to indicate the encryption key I recovered yesterday. I do not know if the traffic is currently unencrypted, or if simply semonitor takes it automatically as it is in the same directory. It may be necessary to pass -k with the key to semonitor.

That's all, I just wanted to share with you this way. Someone may find it useful/interesting...

Regards and thanks again for making this tool available.

EDIT: I added a tcp filter to rules.

jbuehl commented 8 years ago

This is very useful, @abdullahtahiriyo. Would it also be possible to do active ethernet monitoring using this method by routing all the TCP and UDP traffic from the inverter to the monitoring server and not sending a copy to SolarEdge? I don't currently have a DD-WRT router, so I can't try it.

abdullahtahiriyo commented 8 years ago

@jbuehl

This is untested, but it should be possible with something like this : iptables -t nat -A PREROUTING -s 192.168.120.100 -p tcp -j DNAT --to-destination 192.168.100.120

So basically, whatever comes from the inverter, sent it to the monitoring server (by changing the IP at OSI level 3).

If you are interested, I can check it for you. Let me know.

jbuehl commented 8 years ago

In that case it looks like it may not be necessary to spoof the DNS resolution of the SolarEdge server. The inverter would want to connect to 217.68.149.126 (prod.solaredge.com) but the router would send that traffic to 192.168.100.120. Would that rule also rewrite the return messages from 192.168.100.120 so that the inverter would think they are coming from 217.68.149.126?

If you want to test it, the command to use on the monitoring server would be

python semonitor.py -t n

abdullahtahiriyo commented 7 years ago

I thank you again. For a week I have been monitoring the inverter and optimizers, sending the information to InfluxDB and plotting it with Grafana. It really works well. Thanks a lot!!

With DNS spoofing, if you have the monitoring server and the inverter on the same network (and you use a switch, which is common this days), you get a direct communication between both machines, without a need for the router to intervene (it just intervenes for DNS spoofing, it the router is the spoofing entity, but does not get/process the communication afterwards). If using the rule, the router receives the communication always, processes it to apply the rule and sends the communication to the monitoring server. So you have more load on the router. It is an alternative solution to the same problem.

As for further lines being needed, you need a FORWARD route between the two networks (the one of the Solaredge inverter and where the monitoring station is). In case of internal separate networks, there is usually such a forward per default. In case of the same network nothing is needed.

There is an important note though, the command above assume that both machines are in different internal networks (192.168.100.0/24 and 192.168.120.0/24).

If they are in the same network, you need in addition to SNAT it (this I do not need because I have them in separate networks, so that the SolarEdge machine does not get to access my other computers, should it be hacked):

iptables -t nat -A POSTROUTING -d 192.168.100.120 -s 192.168.100.121 -p tcp -j SNAT --to 192.168.100.1

Here 192.168.100.1 is the router IP address. 192.168.100.121 is the INVERTER IP. 192.168.100.120 is the monitoring server's IP.

The reason is that the packet will arrive to the router and get DNAT-ed and ultimately arrive to the monitoring server, but the return packet will be sent directly to the INVERTER without passing via the router (they are in the same network after all). The INVERTER would be puzzled by a return packet from the IP address of the monitoring server instead of from the IP of the SolarEdge server (where its communication is directed) and communication will ultimately fail. By SNAT-ing it, you are telling the router: First DNAT the destination address to the one of the monitoring server (PREROUTING), now that the destination address is changed, in POSTROUTING, when the communication is directed to the monitoring station and comes from the inverter, you also change the source IP address to the one of the router, so that the reply goes to the router instead.

Currently I am quite busy and can not test it. If anybody tries this and for any reason does not work, let me know and I will try to help. If it works, well, it would be nice if you could write that it works so that others know...

abdullahtahiriyo commented 7 years ago

Hi! I just wanted to let you know that I have migrated my dd-wrt instalation to openwrt. The tee syntax is slightly different here, so I thought it may be helpful for others if I posted it here:

# mirroring traffic of inverter
iptables -A PREROUTING -t mangle -d 192.168.120.100 -p tcp -j TEE --gateway 192.168.100.125
iptables -A PREROUTING -t mangle -s 192.168.120.100 -p tcp -j TEE --gateway 192.168.100.125
iptables -A POSTROUTING -t mangle -d 192.168.120.100 -p tcp -j TEE --gateway 192.168.100.125
iptables -A POSTROUTING -t mangle -s 192.168.120.100 -p tcp -j TEE --gateway 192.168.100.125

you also need the tee kernel package: opkg install kmod-ipt-tee

@jbuehl Thanks again for your work. It is really great :)

dragoshenron commented 6 years ago

Thanks to you @abdullahtahiriyo for this insight. Indeed this is a very neat way to address the problem, i.e. routing.

Anyway, if I may add, this method is working if you have iptables installed on your modem, i.e. dd-wrt or openwrt. However me (and someone else) is stuck with more simpler modems. E.g. I have a Fritz Box 7360 where it is possible to set static routes ( see https://en.avm.de/service/fritzbox/fritzbox-7360/knowledge-base/publication/show/581_Configuring-a-static-IP-route-in-the-FRITZ-Box/ )

My setup (which I suppose is quite general) is with all devices in a single subnet: IP router: 192.168.178.254 IP Solaredge inverter (HDWave 3000): 192.168.178.46 IP RasPi (python scripts): 192.168.178.100

My idea is to divert all the traffic from my local network (included the inverter) pointing to prod2.solaredge.com to the RasPi (port 22222). The RasPi will duplicate any packet coming from the network interface port 22222. One copy will be sent to prod2.solaredge.com the second will be piped to semonitor.py (and - in my case -to se2state.py and influxdb and domoticz ...). The reply of prod2.solaredge.com will be ideally redirected from the router to the inverter. I'm afraid that this isn't possible so the RasPi will get the answer. This answer should be then sent to the SE inverter.

Make sense? :) I hope so.

In case, @abdullahtahiriyo or anyone else, could you please help me to configure the router and the RasPi to achieve this?

Thanks in advance for all the great work!

Geoff99 commented 6 years ago

Hi there @dragoshenron,

This is not an answer to the question as you pose it, though it is a way of passively monitoring the solaredge internet traffic by duplicating the transmissions that I am using successfully.

The basic approach is described in https://arachnoid.com/wireshark/

It does require an additional piece of hardware - namely a managed switch with a port mirroring facility. If / once you have such a switch, it is relatively painless to set it up.

  1. I feed the LAN traffic from the inverter into (say) port 8 of the switch.
  2. Port 1 (say) of the switch is connected to my router, and passes the traffic onto / back from the internet and the solaredge servers.
  3. Port 7 (say) was set up to mirror any traffic that goes via port 8, by using the standard administrator interface utility to the switch.
  4. After that I connected port 7 to the RasPi via LAN cable.

And that's more or less all there was to it.

Disclaimer - it's a while since I set this up so I've quite possibly forgotten some of the detail and complexities! But I do clearly recall that when I first started using semonitor, this turned out to be the quickest and simplest way of getting passive access to the solaredge internet traffic. It saved me from having to learn a lot more about TCP-IP and routing than I had time or energy for.

Hope that helps

Geoff

abdullahtahiriyo commented 6 years ago

@dragoshenron

I think your idea will not work, because you redirect everything on your network sent to 22222 to the pi, then you use the packet mirroring functionality to duplicate the packet in the pi, and send it from your network to 22222 which will come back to your pi, as your pi is in your network and is trying to go to 22222.

In addition, you must also handle the "way back" from solaredge, to your pi, to duplicate your traffic, to the inverter.

I have heard of the Fritzbox but I have never had one in my hands, so I am not sure what is possible and what not. There should be a way, because duplicating it on the pi is a smart move.

Some ideas:

Hopefully any of those will allow you to avoid having to change your router. If you must change your router, I would really go for an OpenWRT compatible one.

Much luck! abdullah

dragoshenron commented 6 years ago

@Geoff99 Thanks for your feedback. Indeed you can use some extra hardware: an extra router (bare router without modem capabilities), a firewall, configure a RPi as MITM (which is very similar to use a router), or as you said a piece of hardware with mirroring port. I have a managed switch (Mikrotik CSS106-1G-4P-1S) but it hasn't such port :( and I will refrain myself to add extra hardware as - I'm sure about - there must be a solution with the hardware I have (which is on average something that everyone has).

It's going without saying that your solution would be useful for some other user :) Happy new year!

@abdullahtahiriyo Thanks for the feedback. Indeed you spot a glitch in my reasoning. A solution would be to program the RPi as Man-In-The-Middle which - unfortunately - I don't know how to do it. Also - in my particular case - my RPi isn't always on. Indeed it might be beneficial for other user to implement this solution.

Try specific rules per interface if Fritzbox supports them [...]

There are not specific rules per interface so, I think, if you place a static route to forward packets directed to SE server to the RPi, you will encounter the problem you mentioned (packets will come back always to the RPi instead to reach the SE server)

This is provided that the Fritzbox supports tagged vlan, which openwrt supports.

... and which the Fritzbox doesn't. But I can follow your reasoning and indeed it seems a legit approach. Nice as exercise to ICT students and future network administrators ;)

Fritzbox supports a "traffic monitoring"

Wow! You made my day :) I didn't know about this undocumented feature. Indeed there is the possibility to dump all the traffic in Wireshark format. And it seems also quite detailed interface

screen shot 2017-12-31 at 15 06 16 screen shot 2017-12-31 at 15 06 23 screen shot 2017-12-31 at 15 06 30

My SE inverter is wired to eth3 so I'll dump the content and let's see if semonitor can digest that. Also, I need to do this via a command rather then a web-interface. Luckly enough someone did already the job <3 https://github.com/ntop/ntopng/blob/dev/tools/fritzdump.sh

I'll play now with this feature and then I'll give later some feedback. Happy new SOLAR year :)

abdullahtahiriyo commented 6 years ago

@dragoshenron

Happy to be useful. :)

I think it is great to have an issue here with all those different lan passive approaches, so that different users can get their best. At the end we are all together in this (thanks to jbuehl).

Geoff99 commented 6 years ago

@dragoshenron

If you do get your approach working it will be very interesting for anyone using an appropriate Fritzbox with traffic monitoring.

If not, at least you have a fallback to try.

Good luck, Geoff

dragoshenron commented 6 years ago

As promised, I give some feedback on my tests.

The Fritzbox indeed is able to capturing packets going through its ports, in my case eth2 (which is Ethernet port 3 !). The format used is what wireshark calls "modpcap - Modified tcpdump - pcap".

You can obtain a file from the webinterface pointing straight to https://fritz.box/html/capture.html An example of such a file is the following: iad-if-eth2_02.01.18_1842.eth.zip

Of course we need something that could be run straight from the command line. In order to do this I've used this bash script: https://github.com/ntop/ntopng/blob/dev/tools/fritzdump.sh (remember to edit ip, userid, password and iface as needed). As last line of the previous script I'm using wget -qO- http://$FRITZIP/cgi-bin/capture_notimeout?ifaceorminor=$IFACE\&snaplen=\&capture=Start\&sid=$SID

The script gives a "nice" stream via stdin. I'm trying to pipe it to tshark, unhexlify and semonitor as adviced... but there is somewhere a glitch

Using thark to digest the file saved from the fritzbox webinterface gives the following result tshark -r iad-if-eth2_02.01.18_1842.eth -T fields -e data | ./utilities/unhexlify.py | ./semonitor.py -k 73110AAB.key {"inverters": {}, "meters_0x0022": {"73110AAB": {"9_PVProduction": {"TotalE2Grid": 0, "AlwaysZero_off34_int2": 0, "seId": "73110AAB", "TotalEfromGrid": 0, "Flag_off20_hex": "00 80", "devLen": 58, "Flag_off28_hex": "00 80", "E2X": 0, "PfromX": NaN, "P2X": 0.0, "Date": "2018-01-02", "AlwaysZero_off18_int2": 0, "Totaloff22_int4": 0, "Totaloff30_int4": 0, "Interval": 300, "EfromX": 0, "seType": "0x0022", "Time": "18:49:56", "onlyIntervalData": 1, "dateTime": 1514918996, "AlwaysZero_off26_int2": 0, "AlwaysZero_off10_int2": 0, "devType": "meters_0x0022", "Flag_off36_hex": "00 80", "Flag_off12_hex": "00 80", "recType": 9}}}, "events": {}, "optimizers": {}} {"inverters": {}, "meters_0x0022": {"73110AAB": {"9_PVProduction": {"TotalE2Grid": 0, "AlwaysZero_off34_int2": 0, "seId": "73110AAB", "TotalEfromGrid": 0, "Flag_off20_hex": "00 80", "devLen": 58, "Flag_off28_hex": "00 80", "E2X": 0, "PfromX": NaN, "P2X": 0.0, "Date": "2018-01-02", "AlwaysZero_off18_int2": 0, "Totaloff22_int4": 0, "Totaloff30_int4": 0, "Interval": 300, "EfromX": 0, "seType": "0x0022", "Time": "18:54:56", "onlyIntervalData": 1, "dateTime": 1514919296, "AlwaysZero_off26_int2": 0, "AlwaysZero_off10_int2": 0, "devType": "meters_0x0022", "Flag_off36_hex": "00 80", "Flag_off12_hex": "00 80", "recType": 9}}}, "events": {}, "optimizers": {}} {"inverters": {}, "meters_0x0022": {"73110AAB": {"9_PVProduction": {"TotalE2Grid": 0, "AlwaysZero_off34_int2": 0, "seId": "73110AAB", "TotalEfromGrid": 0, "Flag_off20_hex": "00 80", "devLen": 58, "Flag_off28_hex": "00 80", "E2X": 0, "PfromX": NaN, "P2X": 0.0, "Date": "2018-01-02", "AlwaysZero_off18_int2": 0, "Totaloff22_int4": 0, "Totaloff30_int4": 0, "Interval": 300, "EfromX": 0, "seType": "0x0022", "Time": "18:59:56", "onlyIntervalData": 1, "dateTime": 1514919596, "AlwaysZero_off26_int2": 0, "AlwaysZero_off10_int2": 0, "devType": "meters_0x0022", "Flag_off36_hex": "00 80", "Flag_off12_hex": "00 80", "recType": 9}}}, "events": {}, "optimizers": {}} {"inverters": {}, "meters_0x0022": {"73110AAB": {"9_PVProduction": {"TotalE2Grid": 0, "AlwaysZero_off34_int2": 0, "seId": "73110AAB", "TotalEfromGrid": 0, "Flag_off20_hex": "00 80", "devLen": 58, "Flag_off28_hex": "00 80", "E2X": 0, "PfromX": NaN, "P2X": 0.0, "Date": "2018-01-02", "AlwaysZero_off18_int2": 0, "Totaloff22_int4": 0, "Totaloff30_int4": 0, "Interval": 300, "EfromX": 0, "seType": "0x0022", "Time": "19:04:56", "onlyIntervalData": 1, "dateTime": 1514919896, "AlwaysZero_off26_int2": 0, "AlwaysZero_off10_int2": 0, "devType": "meters_0x0022", "Flag_off36_hex": "00 80", "Flag_off12_hex": "00 80", "recType": 9}}}, "events": {}, "optimizers": {}}

I've tried what I thought could be a good pipe ./fritzdump.sh | tshark -i- -T fields -e data | ./utilities/unhexlify.py | ./semonitor.py -k 73110AAB.key Capturing traffic.. Capturing on 'Standard input' tshark: Unrecognized libpcap format or not libpcap data.

The reason being - I think - the following https://wiki.wireshark.org/CaptureSetup/Pipes

There are some limitations that you should be aware of: This only works with the de facto standard libpcap format version 2.4, as described in Development/LibpcapFileFormat, and with the standard pcap-ng format. Capturing from a pipe is inconvenient, because you have to set up the pipe and put a file header into the pipe before you can start the capture. A few patches have been mailed to the development list that could solve this, so if you find the approach inconvenient, try the patches.

and again

While the documentation says that Wireshark should be able to capture from stdin (which is an anonymous pipe), it is not clear whether it always works.

I've also tried to use something along the line of https://serverfault.com/questions/150167/how-do-i-convert-wireshark-capture-files-to-text-files but to no avail.

Is there someone who has some ideas what could be the correct syntax? Cheers.

dragoshenron commented 6 years ago

@Geoff99 @abdullahtahiriyo Hello guys, some ideas? :)

abdullahtahiriyo commented 6 years ago

I am not sure if this can be of help, but it seems to me it is a problem with the stdin pipe, so maybe using a named pipe can help: https://wiki.wireshark.org/CaptureSetup/Pipes#Named_pipes

jshank commented 6 years ago

@abdullahtahiriyo - Did you confirm the key you captured from your HDWave inverter works for decryption? I seem to have gotten mine but it doesn't appear to work. Wondering if the new CPU model moved the key to another memory location.

@jbuehl - How did you determine the memory location for the key in the first place?

jbuehl commented 6 years ago

I wasn't the one who found it. See closed issue #8

abdullahtahiriyo commented 6 years ago

@jshank Not sure what exactly is a HDWave, mine is a SE-3500.

If I remember correctly, it worked for some days without the key. Then it required the key.

I have just checked the script and it passes the -k parameter with the key.

JohnOmernik commented 5 years ago

@jshank HAve you figured out the key issue? I extracted my key, but it won't change to Decryption key not yet available. I have a HDWave like you do, so I am curious if the issue is related to diff mem locations.

jshank commented 5 years ago

@JohnOmernik I haven't. I went down a rabbit hole of trying to install a USB port on the control board so I could get an RS-232 connection. SolarEdge decided to leave the SMD (resistors and capacitor) off of the board itself.

JohnOmernik commented 5 years ago

@jshank For me the issue with the network stuff has to do with the network data in, I was able to get my decryption working using the command @jbuehl mentioned in #115

jshank commented 5 years ago

Thanks @JohnOmernik. I'll give it a try. What command did you use to generate the PCAP file?

JohnOmernik commented 5 years ago

So, for direct to semonitor, while I know you can use tshark, I didn't I used the lightweight tcpdump to create the output, write that to stdout, then read from stdin with tshark... sudo tcpdump -i eno4 -w - -U -s 65535 tcp and host 192.168.225.11|tshark -r - -T fields -e data|./utilities/unhexlify.py |./semonitor.py -k mykey.key

While it's a bit redundant, tcpdump is running with privs, and running under apparmor profiles, tshark can run that way, but I am security nerd, and this works too. This way too, tcpdump does all the heavy lifting with the filtering (tcp and host 192.168.225.11) and tshark only processes the inverter traffic.

jshank commented 5 years ago

I'm still seeing Data length is too big for the message errors. Does this mean semonitor is having problems?

<stdin> --> message: 3677 length: 66
dataLen:    002c
dataLenInv: ffd3
sequence:   c204
source:     fffffffd
dest:       ffffffff
function:   003d
Decrypting message
dataLen:    2204
dataLenInv: 3020
sequence:   58e2
source:     4cf98bb3
dest:       25d8fe38
function:   9ad2
Data length is too big for the message
data:       04 22 20 30 e2 58 b3 8b f9 4c 38 fe d8 25 d2 9a
data:       9c 73
Ignoring this message
<stdin> --> message: 3678 length: 342
dataLen:    0140
dataLenInv: febf
sequence:   0280
source:     73178624
dest:       fffffffe
function:   0500
inverter:      73178624 type: 0010 len: 00b4
jbuehl commented 5 years ago

@jshank It looks like the key you are using isn't valid. An encrypted message is being decoded, but the result is garbage.

jshank commented 5 years ago

Bummer. I tried for days to get the key via the RS-485 connection. I pulled 4 values from the command python semonitor.py -c 12,H239/12,H23a/12,H23b/12,H23c -s 73178624 -m -t 4 -vvvv /dev/ttyUSB0. Maybe I got the values out of order when creating the key or they have moved to different memory addresses. semonitor would crash at various places so I never captured all 4 in the same session.

Throughout the capture attempts, I captured the following 4 values. I Then manually ran sekey and pasted the following lines to get a key. The sequence numbers were edited.

{"data": {"type": 0, "value": 1369828XXX}, "command": 18, "response": 144, "sequence": 82}
{"data": {"type": 0, "value": 940302XXX}, "command": 18, "response": 144, "sequence": 83}
{"data": {"type": 0, "value": 703928XXX}, "command": 18, "response": 144, "sequence": 84}     
{"data": {"type": 0, "value": 1663334XXX}, "command": 18, "response": 144, "sequence": 85}
jbuehl commented 5 years ago

@jshank - I just pushed a change to semonitor.py that may fix the problem of getting a key via RS485, or it may break it. I have no way of testing this myself as I don't currently have an RS485 connection to my inverter, so I would appreciate it if you could try it and let me know what happened. Thanks.

jbuehl commented 5 years ago

@jshank - One more thing. Regardless of whether the change works or doesn't work, can you run it with -vvv and post the output here? Thanks.

jshank commented 5 years ago

I'll give it a try, probably this weekend weather permitting.

mulles commented 4 years ago

@abdullahtahiriyo thx for sharing the iptables solution. I would like to enhance it.

I am looking at duplicating the SE traffic inside my LTE Router and sending it to my remote Server. I hope to get in done, with as little modifications to the router software as possible so it's stability is not affected.

iptables -A PREROUTING -t mangle -d 192.168.120.100 -p tcp -j ROUTE --gw 192.168.100.120 --tee iptables -A PREROUTING -t mangle -s 192.168.120.100 -p tcp -j ROUTE --gw 192.168.100.120 --tee iptables -A POSTROUTING -t mangle -d 192.168.120.100 -p tcp -j ROUTE --gw 192.168.100.120 --tee iptables -A POSTROUTING -t mangle -s 192.168.120.100 -p tcp -j ROUTE --gw 192.168.100.120 --tee

Would it be possible to exchange the LAN IP "192.168.100.120" with the WAN IP of my remote Server? I guess I would need to open a bunch of ports in order to enable the remote server to receive the packets? Thus, it might be better options to tunnel it through ssh right? If that's the case, my first question becomes obsolete as the iptables do not help me to redirect the SE traffic over ssh to my remote server?