Local API with Protocol Buffers

[drobtravels]

I recently had a SE10000H-US installed and noticed when I visit the IP address of the inverter it has a status page with some high level information (although no specifics about each optimizer). I see its using an unsecured API by doing a GET on http:///web/v1/status, with the response in protocol buffers.

I would be interested in reverse engineering this response so I could monitor the status in realtime without setting up the additional hardware involved in the ethernet passive approach. Has anyone done this?

I parsed the response using protoc --decode_raw and have a pretty good idea of what the structure of the data looks like, but am unclear how its actually converted to the proper number. For example it seems likely that 0x481b4600 corresponds to 159 kWh. I have no experience with protocal buffers, so I may be missing something obvious, but am not seeing how to get 159 or 159,000 from what I assume is the hex representation of 481b4600

I apologize if this is out of scope for this particular package, but thought some people here might be interested. I can post more details and examples if people are interested.

[drobtravels]

Ok just realized its a floating point number! This calculator can do the conversion.

Is there any interest in me posting samples or .proto files for working with this API?

[Jerrythafast]

That's quite interesting! Would you mind sharing your firmware version numbers?

[AndyRPH]

I'm interested in this. Just got my SE10000H online yesterday. Need to wait two weeks for meter swap out though before it's producing power. I've got the inverter mirrored to another switch port, and an rPi that just does a file a day of tcpdump.

If the inverters are changing their encryption keys a few times a day, being able to get all the same data your way would be much simpler!

[drobtravels]

I don't see anywhere that explicitly says firmware version, but I'm assuming this is that you're looking for?

I have probably 80% of the API response mapped which includes

• Kwh today
• Kwh this month
• Kwh this year
• Total lifetime kwh
• Current power from production meter
• Total kwh from production meter
• Current power
• Voltage
• Frequency
• number of optimizers communicating

[Jerrythafast]

Yes, that's what I was referring to, thanks! I think that's the first time I see a CPU version number starting with 4. Would be interesting to find out whether it offers more API functions than the one you discovered. More specifically, whether it would offer one that provides optimizer status! If we could get our hands on a recent firmware image we might be able to do some string search on it to find out?

[drobtravels]

Just clicking around in chrome with Dev tools on, I see several different API endpoints being used, but the only optimizer specific information I see displayed in browser is the number online.

I was looking at responses at night when it wasn't producing, so I'll try it again during the day in case the information is just in the API.

The weirdest thing about this dashboard is it's totally unauthenticated and appears to allow you to change any setting you want on the inverter, including grid protection, etc. I haven't tried changing any setting since I don't know what I'm doing. I wonder if the installer enabled it by mistake?

[AndyRPH]

I believe that's by design. My installer used a connection this way to do all the setup and provisioning of the inverter. My versions match yours exactly. Fingers crossed this provides more data than the existing public api.

[jbuehl]

SolarEdge was promoting this capability recently for installers https://marketing.solaredge.com/acton/rif/8801/s-144b-1902/-/l-0045:1133/l-0045/showPreparedMessage?sid=TV2:cfByisFlu.

If this interface can provide information about individual optimizers it is very interesting because the lack of that capability was the original motivation for this project. Hopefully it does and can be available on older inverters with a firmware upgrade.

[drobtravels]

Unfortunately I'm not seeing any details about optimizes (under then number communicating vs paired). Reading through the installer instructions, it has them pair via phone app, then go to the monitoring tool to setup the rest of the optimizer details. That makes me think the local API might not have these details implemented.

On the plus side, I realized that if you leave the dev console open, it actually logs out the parsed API responses as you're browsing the dashboard, so it should be very easy to use the APIs. I'll try to publish details of the APIs over the weekend.

[drobtravels]

I've been digging into this and am pulling out a lot of details. I'll post specifics soon.

I've found some optimizer specific information via the maintenance API

• Module
• Last Telemetry
• Output Voltage
• Input Voltage
• Input Current

Is that useful? I was hoping to see watts / power or something similar.

[jbuehl]

That's very good. Optimizers don't directly report power, but the watts can be calculated by multiplying the input voltage by the input current. The only other thing that would be useful would be the optimizer temperature.

[drobtravels]

Oh great! It does have the temperature in the API actually.

I'm not sure where you want this information posted, so I created my own repo. Happy to merge into your project if you prefer. https://github.com/drobtravels/solaredge-local

I went ahead and mapped what looked to me the most interesting data here

[dragoshenron]

Great job @drobtravels ! Thanks for sharing and for your effort. I made a pull request on your parallel repository with some addition on which inverters (US and European) are supported.

[drobtravels]

Just FYI, here's the information I'm getting back per optimizer

optimizer {
        serialNumber: "xxxx"
        online: true
        lastReport {
          year: 2019
          month: 3
          day: 11
          hour: 10
          minute: 20
          second: 29
        }
        outputV: 23
        inputV: 31
        inputC: 8
        temperature {
          value: 34
        }
      }

[AndyRPH]

Any idea when this might be available? I can pull the data off with your example command but not get it into a csv or such to put into a database nor homeassistant. Very cool to get all the optimizer data on a local api though!

[drobtravels]

I have it working on Home Assistant with a custom component. I'll hopefully be able to submit a PR this weekend.

I don't plan to develop anything which outputs to CSV, as that would require keeping a service running to constantly call the API and store the results. I'm happy to offer guidance if you'ld like to develop that though. You can also leverage the python library.

[AndyRPH]

Awesome! Looking forward to the PR so I can implement it locally until it's merged. I had hoped there was an easy way to convert the protoc output to csv so I could keep the data stored apart from HA, but if it gets into HA than its easy to pull out of HA. Thanks!

[drobtravels]

@AndyRPH There is no issue converting to CSV, but I assume you're looking for historical data in that case? The local API only gives you a snapshot of the current state. Typically CSVs contain lines of data by minute, hour, day, etc. To get historical data you'll need something to poll the API and constantly collect data.

[AndyRPH]

Yeah I just wanted to log it somewhere that was more robust than my rpi running home assistant. The maintenance page seems to include a time stamp in the data. I had hoped there was an easy way to convert the protocol buffer message output with the message definitions as the first row then all the data below, each query to the api being a row, since the "column" numbers should not change unless I change optimizer quantities. At the moment I just threw together a cronjob and bash script that outputs it to a unique log file every 15min. I can always just figure out a regex and clean the file up later on. It looks similar to but not quite json. I didn't see any good documentation on the --decode options for the protoc command line tool.

If it gets reliably into HA I'll just replicate that section of the database to another device to maintain it all.

[drobtravels]

Gotcha. Do you know any python? It should be relatively simple to write a python script using the library I made to create that.

[AndyRPH]

Nah, not a coder by trade but picked up enough knowledge to tinker, as my day job had me learn a relatively language for their hospital software.

Once your stuff is available I'll probably give python a good faith try since I'm better if I can reverse something working as a template to learning.

[jshank]

This API doesn't appear to be available for the LCD-based SE-10000H-US. I opened a ticket with SolarEdge and they pushed out the latest firmware on the same day but I'm not seeing a web interface or any major changes in behavior.

CPU Version: 3.2468.0 (updated per the request) DSP 1 Version: 1.0.751

I may have lost access to the data using semonitor though. All messages are now coming through with Decryption key not yet available.

[ContextPerspective]

@jshank

I think the web interface is only on the non-LCD models. SE marketing failed to rename the models based on this key design change, so it makes it fun with threads like this. ;)

I can personally confirm the same lack of web functionality on LCD Models SE6000H-USRGM AND SE3000H-USRGM. No access to the interface and refused message when attempting to connect.

[dragoshenron]

@jshank Indeed any inverter with display doesn't support these APIs (at least not as of today).

Model number is the same (with and without SetApp) but part number is different (see datasheet): SExxxxH-XXXXBXX4

[oliv3r]

The 4 in the 'cpu version' stands for the new software gen. The LCD only versions use '3' as it's software gen 3, as we see 'portia 3' throughout the codebases and firmwares. The gen 4 is called 'Portia-NG' as in net generation. These guys really need to work on naming and versioning, as the platform is really incomparable. I do wonder how the inside looks like, if they only changed the 'control' PCB. Also I wonder if this solution is cheaper then the LCD (more expensive development for the higher-level software).

Initially, when searching for firmware files for the version 3, I stumbled upon the software for the version 4 and was very pleasantly surprised that it was running Linux (our version 3's are not). As such, I started to take it apart and analyze it.

Sadly, I stumble upon this thread after you spend some painful amount to RE it. The protobuf protocol, and endpoints are available in the firmware quite visibly in python (well pyc, but that was easily resolved). :)

Once I realized however that this firmware was not four our devices, but only for the non-LCD ones, I was sad in that I could not run Linux on my inverter and kinda gave up on the project.

What was peculiar, is that they must have been working on this for a while, as they where using an extremely outdated and slow ARM chip ...

However, all the RE data is still there, with most of my findings. https://gitlab.com/centraluniverse/portia-ng_analysis

P.S. If someone can point me to a new firmware, ideally in chronological order, I don't mind extracting the firmware and adding it to the repo, to nicely track the changes between versions. Btw, I think I'm pretty sure I extracted the firmware image from the android apk, as it's part of the apk image. If I can find an easy way to download older versions (in between) I should be able to get historical data. This assumes that the app gets updated in sync with the inverter software, which is not unreasonable to think, as the app is intended for installers. Edit: This is where I found someone analyzing the App: https://gitlab.com/sehacker/inverter_configuration_tool/tree/setapp

[kingfisher63]

The current SetApp apk (v2.3.0) for Android does not contain any inverter firmware (the apk file is way to small for this anyway). Comparing the proto files from solaredge-local (apparently based on 4.4.67 or later) with the protobuf descriptors in 4.0.58 (from the Portia NG analysis by @oliv3r) shows that SolarEdge did not change essential parts. This may not be true in the future. We will need to find some other way to get to the protocol buffer message descriptors so that we can keep up with SolarEdge developments.

[AndyRPH]

If it's anything like the iOS app, you have to login to Setapp with an installer account and then it pulls new firmwares from their server. I just updated mine yesterday.

[kingfisher63]

Is there any evidence that updating firmware leaves files on the mobile device that can be recovered? (If so, this will probably be easier on Android than on iOS)

[AndyRPH]

The fact that it downloaded firmware to my phone before I connected it to my inverter means that it probably is there before. Maybe not after. I'm not sure how to get a look at what the iOS app may have downloaded since it's not jailbroken. Nor do I have an android device.

[oliv3r]

I've been looking at the code as well; and the URL is reasonably easy, the problem it's behind a login, e.g. you need an authentication token. I didn't spend time yet to see if a) your username needs some special level and b) making it it work :)

As for the file, it is guaranteed to be left on the device, for the simple reason, your inverter can't pull it itself (or you wouldn't use setapp) and thus the app first downloads it, and then uploads it to the device.

So after figuring out the rather simple curl command with the authentication token; it should be easy to get new firmware versions automatically downloaded.

That said, I've extracted the firmware from all 'old' versions (thank apkmirror) and will update the repo with the information where needed :)

[kingfisher63]

Please be advised that access via the local interface no longer works in firmware 4.6.24 as SolarEdge has restricted access to 172.16.0/24 IP addresses (the IP network used by the built-in WiFi access point for SetApp).

[drobtravels]

Yikes!

How do firmware upgrades work? Are they automatic or require you to manually upgrade?

[victorclaessen]

Please be advised that access via the local interface no longer works in firmware 4.6.24 as SolarEdge has restricted access to 172.16.0/24 IP addresses (the IP network used by the built-in WiFi access point for SetApp).

I have two inverters, both running 4.6.24. The SE1500M is still accessible via the LAN http interface, but the SE1000M is not, it only gives {"message": "Internal Server Error"} as an output. Apparently not all devices shut down their LAN web interface.

[scheric]

@drobtravels From what I know they are not updated automatically. The settapp application with an installers account is needed for an upgrade. When the setappp application connects to the inverter it is done automatically. So you can not even chose if you want to update.

So if you want to keep the local interface do not let any installer come near your inverter.

Note: The software can also be updated via an USB port.

[victorclaessen]

Note: The software can also be updated via an USB port.

But has anyone been able to get their hands on the firmware file yet?

[AndyRPH]

@victorclaessen can you browse to their IP address in a web browser? Curious if they closed it totally or just adjusted the protocol such that the python module isn't fully compatible anymore.

[victorclaessen]

Sorry if that was unclear. On both devices the http port is open and a webserver is active on the LAN interface. On the SE1500M I get the 'Commissioning' page from that LAN webserver, the same as you'd get using the SetApp. On the SE1000M I get the aforementioned error. (But that error IS coming over http transport; it's what the webserver reports)

[los7702]

Please be advised that access via the local interface no longer works in firmware 4.6.24 as SolarEdge has restricted access to 172.16.0/24 IP addresses (the IP network used by the built-in WiFi access point for SetApp).

I have two inverters, both running 4.6.24. The SE1500M is still accessible via the LAN http interface, but the SE1000M is not, it only gives {"message": "Internal Server Error"} as an output. Apparently not all devices shut down their LAN web interface.

I have a SE15k I get the {"message": "Internal Server Error"} after upgrade to 4.6.24 firmware

[oliv3r]

I have tried to investigate how to get the firmware file. I have found in the source, that it's a trivial http rest call, but it's authentication protected. I'm not web guy, so not super familiar here. I have installed the android sdk + emulator and installed setapp into an emulator, that works great, however I could not log in with my credentials from the monitoring portal (but i'm not an installer, so there's that).

I tried to create an installer account, but it always fails with 'general error' in a popup and that's that.

What I wanted to do, was intercept the traffic using mitm-proxy and see what calls they are doing for authentication. It should be possible to figure this out from the code alone, but thought that doing it via the app was going to be easier, if not give me some more insight.

Long story short, I haven't figured out the authentication bit yet :) but my guess is, it's a standard authentication header that could be supplied to curl.

If this sounds easy to anybody reading this, hit me up so we can figure this out :) I'll try to find the URL again tonight (have it at home) and replace this part with the example URL of the update file.

[ContextPerspective]

I do not own a SetApp enabled inverter, but love what this project is doing and wanted to add this should it help. Below is a method to get the files on non-jail-broken/non-rooted devices. You'll need an installer account as others have stated. I haven't tested the methods below, but can't see why this wouldn't work as both methods backup the user/private data which should include the FW images. I've used similar methods for other files in the past, so it should work in theory.

iOS - Non Jailbroken Device:

• Login to installer account. Let files update. Close app.
• Perform manual backup using iTunes. Make sure "Encrypt Local Backup" is checked as some protected files will not backup without this. It may not be needed, but just in case. Set a password.
• Using an iTunes Backup Extractor (Google for this, many options - some free, some not) and extract the files from the encrypted iTunes backup. A free option which allows up to 4 files to be downloaded is this . Use Expert Mode and navigate the Applications folder for the setapp data.

For Android - Non Rooted Device

• Login to installer account. Let files update. Close app.
• Perform backup via adb (i.e. adb backup -apk -shared -all -f <filepath>/backup.ab)
• Extract files using any adb backup extractor such as this. Howto instructions here. (i.e.java -jar abe-all.jar unpack backup.ab backup.tar your_unlock_password_unqouted)

[oliv3r]

using the backup is not a bad trick :) Im' sure you did/try this, can you verify/confirm that the update file still has the spff file extension? The files should be named 'Version_4_4_67.spff' for example for firmware version 4.4.67.

The url that probably contains the list of available files should live here: https://api.solaredge.com/solaredge-apigw/api/agileIntegration/spffFiles.json but I haven't figured out the http authentication mechanism. I doubt they use the password as the http-auth token, but who knows. I'll dig more into the source code to figure out how the app is doing it.

[oliv3r]

After further digging, it seems that a CSRF-TOKEN is set as a cookie as part of the http request. okhttp3's Cookie.java is being used for this end. I'm not yet sure how they are setting the value, and how the cookie is initialized (i'm new to CSRF tokens and their implementation); but it seems like you open a http session, you get the initial csrf token data, you use this token with the credentials to request the protected content. I've done something remotly similar once to write a script to keep my sat smart-card up to date; if curl can do this, it should be possible somehow.

[AndyRPH]

Best tool to explore a firmware file once you have one? Nevermind, I'm in over my head, who wants the file? ;)

[Whizkidzz]

The best tool to view the firmware file is Ida pro

[oliv3r]

Hey @Whizkidzz

In this case, no, it's useless. What IDA pro IS usefull for, is to look at core_app, I tried it with retdec back then, but that didn't produce anything useful. I think core_app is similar to what we have on the atmel in portia-3 (LCD variant)

What is core_app? I analyzed the firmware last year already, but nobody cared back then :) https://gitlab.com/centraluniverse/portia-ng_analysis

The interesting bits are (if still valid) are the root password and such :) I am in the process in writing a script to generate these datapoints from the spff file automatically (shouldn't be to hard). But first, I need to figure out how to automatically download the files (and old files while at it).

sniffing the setapp tool gives us a lot of information, going over the java code gave me a lot as well. I just need an installer account and some web guy that knows about CTRF-TOKENS and session cookies and such :) Progress is there, just slow.

P.S. can you answer me at some point about https://github.com/Whizkidzz/Solaredge/issues/3 :)

[oliv3r]

Best tool to explore a firmware file once you have one? Nevermind, I'm in over my head, who wants the file? ;)

I do want the file of course :) I only have the ones I was able to extract from the APK atm.

[AndyRPH]

Are the files from your apk extraction any different than: (i didn't list them all, since i don't have the other inverter models.

-rwxr-xr-x 1 me staff 128K Sep 20 2019 EVSE_2_1_12.spff -rwxr-xr-x 1 me staff 640K Sep 20 2019 JupiterMngr_1_13_1541.spff -rwxr-xr-x 1 me staff 331B Sep 20 2019 SE10K-RWHDCBNN4.ECO-17569.SW-PN-01068.spff -rwxr-xr-x 1 me staff 24M Sep 20 2019 Version_4_6_27_88490.spff -rwxr-xr-x 1 me staff 24M Sep 20 2019 Version_4_7_17_89106.spff

[oliv3r]

Yeah, the only interesting bits are the Version4 files. I haven't figured out what the other SPFF files are for, but don't really 'care' either :) I think this is configuration data or similar. It's not directly relevant. The Version_4 files contain everything that gets flashed. Interesting that they changed the file-naming slightly, they added some SVN sequence number. Nice to know :)

It may be though that the DPS spff's are in binary form here too, but haven't looked at that yet.

[AndyRPH]

Also worth nothing that version 4.7 isn't out yet.... at least not per their website.

[MarcHagen]

Found a dropbox url in the Dutch Tweakers website: https://gathering.tweakers.net/forum/view_message/60013242

I'm not going to post the link here because that may cause some trouble? So just referring to the post.

And yes it is indeed the firmware file