Library broken because api new signature was added by update

[bnc-rainman]

2022-10-16 AlphaCloud V5.0.0.2 New Signature verification and anti-crawl data functionality are added in all APIs of AlphaCloud.

---- New Request headers--- authsignature: al8e4s2c6c8e91af3e3......... authtimestamp: 1665929281

[bnc-rainman]

https://cloud.alphaess.com/static/js/app.4545d660.js

e.headers.authtimestamp = parseInt((new Date).getTime() / 1e3), e.headers.authsignature = "al8e4s" + Object(m.sha512)("LSZYDA95JVFQKV7PQNODZRDZIS4EDS0EED8BCWSS" + parseInt((new Date).getTime() / 1e3)) + "ui893ed"

I am afraid because of the constants. I can imagine that they will change from time to time or will differ per user.

[SorX14]

I've modified the library in other ways so can't do a PR, but here are some bits that I added and its working again.

Basically its possible to workaround this issue but your point about the constant changing is a concern.

        import hashlib
        ...
        self.authconstant = "LSZYDA95JVFQKV7PQNODZRDZIS4EDS0EED8BCWSS"
        ...
        date = datetime.utcnow()
        utc_time = calendar.timegm(date.utctimetuple())

        self.authtimestamp = str(utc_time)
        constant_with_timestamp = self.authconstant + self.authtimestamp
        self.authsignature = "al8e4s" + hashlib.sha512(constant_with_timestamp.encode('ascii')).hexdigest() + "ui893ed"

        header_with_signature = HEADER
        header_with_signature["authsignature"] = self.authsignature
        header_with_signature["authtimestamp"] = self.authtimestamp

        ...
        headers=header_with_signature
        ...
        session.headers.update({'authsignature': self.authsignature})
        session.headers.update({'authtimestamp': self.authtimestamp})
        session.headers.update({'Authorization': f'Bearer {self.accesstoken}'})

[crowdstrike-steve]

this is awesome thank you. for those that use java / scala for there own custom applications

    import com.google.common.base.Splitter
    import com.google.common.hash.Hashing
    import com.google.gson.Gson

    val authconstant = "LSZYDA95JVFQKV7PQNODZRDZIS4EDS0EED8BCWSS"
    val authtimestamp = Splitter.fixedLength(10).split(Instant.now().toEpochMilli().toString).iterator().next()
    post.setHeader("authtimestamp", authtimestamp)
    val constant_with_timestamp = authconstant + authtimestamp
    val authsignature = "al8e4s" + Hashing.sha512().hashString(constant_with_timestamp,StandardCharsets.UTF_8) + "ui893ed"
    post.setHeader("authsignature", authsignature)

[matt-oneill]

Thanks @bnc-rainman @SorX14 for the code which was used for the fix. Hopefully they don’t change too often. The constants were the same for me and the Alpha demo account so assumed they are static at least for now.

[SorX14]

While this issue can be closed now, it seems like the best way to have the discussion, I'll raise another issue if preferred.

Its clear that Alpha are aware that the data is being 'scraped' likely by clients similar to this one. If I was on their dev team, I would be using this repo as the recipe for how to counteract it. We're on the backfoot as our code is out in the open.

I believe we're all on the same page that we believe our data is our property, and shouldn't have to use their dashboard to access it if we wish to provide a different integration or display. In my case, scraping the data reduces the load for them. I would just set up a screen reader if they entirely disabled the JSON API so not sure their end goal.

Anyway, perhaps it would be good to offer an olive-branch by adding some documentation or enforced limits in this library to not poll their service excessively. Feels like being polite at the very least.

Their dashboard polls api/ESS/GetLastPowerDataBySN every 5 seconds, and my local use of this library changes to every 10 seconds. I would therefore suggest that 10 seconds should be the minimum time. Doesn't have to be a super secure limit, just something that most people would accept as default and can fork if they find it restrictive.

[bnc-rainman]

I have the same concerns as @SorX14 . In my opionion the getdata() creates too much load on the servers. Especially the history data may need expensive database operations.

Therefore, I have already enhanced the method on my side as follows, because I need only the power data for my purpose.

  async def getdata(self, skips=[]) -> Optional(list):
        """Retrieve ESS list by serial number from Alpha ESS"""

        try:
            alldata = []
            units = await self.__get_data("Account/GetCustomMenuESSList")
            logger.debug(alldata)

            for unit in units:
                if "sys_sn" in unit:
                    serial = unit["sys_sn"]
                    logger.debug(f"Retreiving energy statistics for Alpha ESS unit {serial}")

                    if 'statistics' not in skips:
                        unit['statistics'] = await self.__daily_statistics(serial)

                    if 'system_statistics' not in skips:
                        unit['system_statistics'] = await self.__system_statistics(serial)

                    if 'powerdata' not in skips:
                        unit['powerdata'] = await self.__powerdata(serial)

                    if 'settings' not in skips:
                        system = await self.__system_id_for_sn(serial)
                        unit['settings'] = await self.__settings(system)

                    alldata.append(unit)
            return alldata

I wanted to write the next days the changes again cleanly and suggest it as PR Parameter as "Option In" list and not as Skip list) but now the discussion has caught up with me

[SorX14]

Interesting, thats the same approach that I took except my hacking was to comment out the bits that were not relevant to me.

I should really start using the main branch instead of a fork so I can contribute, but my thoughts:

__ess_list should cache for the lifetime of the script. The tradeoff of having to restart the script if you add an additional unit to your account is more than acceptable.

The list of statistics should definitely be opt-in. Default to just powerdata if a default is needed - most interactive result set (changes often) and least expensive from Alpha's side judging by their dashboard setup. Might be better to have separate functions for each (getpowerdata, getsystemstatistics etc) so the reply format is the same instead of having to check if a property is available in the consumer code.

Add some examples (or inline function comments) on how to responsibly use the library (minimum polling interval, relative expense of query etc.). I know when I first started using this script that I wouldn't have delved into a discussion like this to understand the impact of using it.

I'll (hopefully) work on documentation when I get more time.

[CharlesGillanders]

While this issue can be closed now, it seems like the best way to have the discussion, I'll raise another issue if preferred.

Its clear that Alpha are aware that the data is being 'scraped' likely by clients similar to this one. If I was on their dev team, I would be using this repo as the recipe for how to counteract it. We're on the backfoot as our code is out in the open.

I believe we're all on the same page that we believe our data is our property, and shouldn't have to use their dashboard to access it if we wish to provide a different integration or display. In my case, scraping the data reduces the load for them. I would just set up a screen reader if they entirely disabled the JSON API so not sure their end goal.

Anyway, perhaps it would be good to offer an olive-branch by adding some documentation or enforced limits in this library to not poll their service excessively. Feels like being polite at the very least.

Their dashboard polls api/ESS/GetLastPowerDataBySN every 5 seconds, and my local use of this library changes to every 10 seconds. I would therefore suggest that 10 seconds should be the minimum time. Doesn't have to be a super secure limit, just something that most people would accept as default and can fork if they find it restrictive.

I initially wrote the library to be used within my HomeAssistant custom component. That component is locked to only update once per minute. I personally consider that to be sufficiently frequent to scrape data.....

[SorX14]

I initially wrote the library to be used within my HomeAssistant custom component. That component is locked to only update once per minute. I personally consider that to be sufficiently frequent to scrape data.....

😐 Might be my fault why they're limiting access then... Still less than half of their dashboard if I were to use that instead.

I use it to store in a local database, then display near-live results to several in-home displays to know when its best to use high-energy devices.

[bnc-rainman]

I don't think anyone from this thread has anything to do with that. The website itself polls the energy data every 5 seconds and the statistics about every 5 minutes. So I don't think either a full poll every minute or a poll of the energy data every 5 seconds will leave a noticeable footprint.

Nevertheless, I think the proposed improvements make sense, because probably many more will use this library and not everyone will have the know how to handle the resources of Alphaess responsibly.

What already comes out here is that our use cases coincide. I too sync the data into data points of my home automation Rasperrymatic using a Docker container. The displays with their ESP32 chips then serve themselves from there. Therefore, I assume that many other users have the same need.

(Apologies for the automatic translation using deepl).

[crowdstrike-steve]

give me access to the inverter itself please alphaess :)

[bnc-rainman]

@crowdstrike-steve https://www.alpha-ess.de/images/downloads/handbuecher/AlphaESS-Handbuch_SMILET30_ModBus_RTU_V123-DE.pdf

Query via the cloud is more convenient ;-)

[iaincoulter]

I notice Alpha are changing API again

AlphaCloud V5.0.0.3 2022-10-29 Add

1. Platform security has been improved further, and multiple anti-crawling mechanisms have been added. After this update, please clear the browser cache and log into the system again if there is message that "Network anomaly, please try again later " .

[bnc-rainman]

It´s seems they really want screen parsers instead.

[SorX14]

It´s seems they really want screen parsers instead.

Which is really odd, this library adds less load than screen parsing.

Think I might start work on one anyway, I don't want to have to update my setup everytime Alpha updates

[CharlesGillanders]

For now it seems that the API library continues to work. We obviously have no way of knowing if that will continue to be the case.

[bnc-rainman]

The update is announced for 10/29/2022. We'll know more in two days.

[matt-oneill]

@bnc-rainman @SorX14 it looks like on the website the app.****.js has been updated already around authtimestamp and authsignature if you want a head start. I don’t have time to look at it for a few days but you may be able to see what the change is before they enforce it on the weekend

[SorX14]

Too tired to figure it out now, but looks like its a morass of webpack obfuscated code to come up with the authsignature result. With a little effort it can be reverse engineered.

I would suggest to everyone subscribed to this issue to use the 'Product Suggestion' feature of the Alpha dashboard to ask for sanctioned API access. This cat-and-mouse game to prevent access to data from our own equipment isn't customer friendly and annoying to work around.

Hobbyists work on the weekend, so they've wasted a week working on this 😄

[matt-oneill]

Looks like the update has changed some 'POST's to 'GET's. The authentication still works and it doesn't seem to be checking for authtimestamp and authsignature (unless they haven't finished breaking it for today)

[CharlesGillanders]

Thanks all, looks like the latest changes are not yet going to break anything else.

[Ralf12358]

Hi All.

Unfortunately it breaks for me since some hours:

pub_alphalog    | 2022-10-31 07:12:08,707|ERROR   |__main__                                          |200, message='Network exception, please try again later!', url=URL('https://cloud.alphaess.com/api/Account/Login')

I hate these cloud stuff. Connecting locally would be the best option. If their is a power outage and it runs on backup there is probably no internet, too if you have no strarlink. This means basically zero info about the system.

I also don't understand why they implement anti scrapping stuff. What do they gain?

In case someone really implements a screen reader for it, please be so kind and make it public.

[bnc-rainman]

Hi All.

Unfortunately it breaks for me since some hours:

pub_alphalog    | 2022-10-31 07:12:08,707|ERROR   |__main__                                          |200, message='Network exception, please try again later!', url=URL('https://cloud.alphaess.com/api/Account/Login')

I hate these cloud stuff. Connecting locally would be the best option. If their is a power outage and it runs on backup there is probably no internet, too if you have no strarlink. This means basically zero info about the system.

I also don't understand why they implement anti scrapping stuff. What do they gain?

In case someone really implements a screen reader for it, please be so kind and make it public.

It is possible to connect locally with ModBus RTU. Look above, I linked the manual.

Nobody here understand that anit scrapping stuff. May be this is the result of small minds to use open source, but be restrictive themself. May be a cultural thing that chinese have an other point of view. At the end of the day it doesn´t matter.

For myself I will start to work on a screen parser solution with more footprint. If they want them, they will get them.

By the way: The login request header contains now an authsignature and authtime entry too.

[Ralf12358]

It is possible to connect locally with ModBus RTU. Look above, I linked the manual. Thanks! While this is of course nice, I thought of a local web server like you can find it in a consumer router. Anyway it is what it is, I got this system, because nothing else was deliverable. It actually looks nice! But it is more a system for people who do not care and just want if running without doing anything unusual with it.

I would like to control the heating of the pool und such stuff by reading actual power values. In addition I like to control the maximum discharging and by using a weather/solar forecast to ensure there is remaining battery capacity in case of a power outage, without unnecessarily restricting it too much if it is know there will be enough sun to charge the battery again, soon. All such thing become unnecessarily complex, because of such stupid things.

By the way: The login request header contains now an authsignature and authtime entry too. Yes and it worked until today early in the morning.

[SorX14]

Been looking around, and this guy has made an unbelievably well documented repo on how to use the ModBus port. I'll be doing that once the parts arrive :) https://github.com/dxoverdy/Alpha2MQTT

There is also this project that captures the outbound packets via proxy or firewall forwarding: https://github.com/230delphi/alphaess-to-mqtt/

I would image that the proxy method would fail if they decide to encrypt the comms and might fail if your home internet connection is offline?

I think a RaspberryPi with a RS428 adapter is the easiest sanctioned way to do this.

FWIW, we're the absolute minority. Yes there are some turbo nerds that want local data all the time (like me!), but most users are more than happy with the app. Documenting, developing and supporting an API - especially locally - is a cost they are not willing to bare for the 12 people who want it. I don't even know if the inverter has enough local processing power to run a webserver. I believe it just blindly fires unencrypted data at an IP address (which you can see/change in the settings 52.230.104.147), so its not too clever.

Yes and it worked until today early in the morning. Most likely you got logged out, and the refresh token authentication process didn't work as expected. My usage of this library will completely clear the session and reauth from scratch. Been running fine since the last update

[Ralf12358]

Been looking around, and this guy has made an unbelievably well documented repo on how to use the ModBus port. I'll be doing that once the parts arrive :) https://github.com/dxoverdy/Alpha2MQTT

There is also this project that captures the outbound packets via proxy or firewall forwarding: https://github.com/230delphi/alphaess-to-mqtt/

Wow thanks! This might help.

[matt-oneill]

Looks like Alpha have updated again this morning, now getting this api response logging in

{
    "code": 9006,
    "info": "Network exception, please try again later!",
    "exMessageInfo": "Please use web browser to view the data"
}

[SorX14]

Yet they still haven't fixed their typo...

I just logged in with a browser, grabbed the authsignature and authtimestamp, then ran it through our implementation. It no longer matches. They've changed something with that part

[dehsgr]

Auth Base Key changes from LSZYDA95JVFQKV7PQNODZRDZIS4EDS0EED8BCWSS to LS885ZYDA95JVFQKUIUUUV7PQNODZRDZIS4ERREDS0EED8BCWSS. See https://github.com/dehsgr/node-red-contrib-alphaess/commit/6a14ff07d5e7edc3706950ce7fda421b28781ac8

greetz

[SorX14]

@dehsgr How did you derive that from their code? I need your skills :D

[dehsgr]

@SorX14 prettify their app.js and search for the Base Key. That should direct you in the right place. ;-)

[SorX14]

oh lol, I was trying to reverse engineer all of this mess:

 return e[A("0x93")][A("0x8e")] = A("0x64") + Object(h.sha512)(t + parseInt((new Date)[A("0x8f")]() / 1e3)) + A("0x24"),
            e[A("0x93")].authtimestamp = parseInt((new Date)[A("0x8f")]() / 1e3),

but the code is just listed in plain text above 🤦

[SorX14]

Excuse my constant posting, but just mirrored the Alphas port and wiresharked the output.

I believe it just blindly fires unencrypted data at an IP address (which you can see/change in the settings 52.230.104.147), so its not too clever.

This is exactly what it does. Sends an unecrypted JSON of the current state to the IP address on port 7777 every 10 seconds. There doesn't appear to be any auth either. If you have a switch with port mirroring, this is definitely a way to get the data locally.

e.g.

{"Time":"2022/10/31 14:41:54","SN":"ALxxx","Ppv1":"57","Ppv2":"57","PrealL1":"112","PrealL2":"0.0","PrealL3":"0.0","PmeterL1":"3270","PmeterL2":"0","PmeterL3":"0","PmeterDC":"0","VmeterL1":"-3270","VmeterL2":"0","VmeterL3":"0","Fmeter":"4.99","Pbat":"0.0000","Sva":"0","VarAC":"-464","VarDC":"0","SOC":"0.0"}

[CharlesGillanders]

There’s another project on GitHub which runs a clone server to grab the output from the unit at home. If you use that project you lose the ability to get updates for your alpha.

The modbus interface seems likely to be the best long term option unfortunately.

[Ralf12358]

There’s another project on GitHub which runs a clone server to grab the output from the unit at home. If you use that project you lose the ability to get updates for your alpha.

Do you have a url? Actually, I would prefer to not get every update automatically, and maybe just stay with the current version.

[CharlesGillanders]

There’s another project on GitHub which runs a clone server to grab the output from the unit at home. If you use that project you lose the ability to get updates for your alpha.

Do you have a url? Actually, I would prefer to not get every update automatically, and maybe just stay with the current version.

https://github.com/tidalvirus/alphaess-server

[SorX14]

There is also https://github.com/230delphi/alphaess-to-mqtt/ which uses a proxy instead so it'll still communicate with Alpha for AlphaCloud and presumably any updates pushed.

The more I look into this, the more inept the Alpha implementation looks. I didn't see this in my capture, but if this is true - https://github.com/tidalvirus/alphaess-server/blob/main/1-1-4.json - sending your address unencrypted is ridiculous. At least the destination is Singapore (52.230.104.147) which appears to be covered by GDPR. Anyone want to try a right of access request to them? :D

My system remains connected only due to the warranty period being reduced otherwise.

[bnc-rainman]

Consumption data are personal data according to DSGVO. I cannot imagine with the best will in the world that an unencrypted transfer of such sensitive data to non-European countries is legal. (compare, for example, Article 32 DSVGO from Germany).

This also explains their anti-crawling nonsense. They do not know what they are doing.

(Deepl translation)

[SorX14]

Might be worth mentioning this to their DE office - https://www.alpha-ess.de - they have an email listed in the header. I am not a native German speaker, and don't know enough about the relevant rules though.

I could come up with fan-fiction of how this data could be used inappropriately (albeit unlikely). If you can monitor all traffic going to that IP, simply plot on Google Maps each address, and then check for abnormally flat usage. That house is very likely to be empty (occupants on holiday, out of town etc.). Having a link between an address and its usage is a bad idea ™️ . There was a similar issue with the first attempt at smart meters which would simply broadcast to any listener.

This also explains their anti-crawling nonsense. They do not know what they are doing.

Seems like they are more interesting in protecting 'their' data instead of their customers. Remember the data is ours to begin with 🤦 I'll get off my soapbox now...

[CharlesGillanders]

As problematic as the unencrypted transfer is I suspect there's likely a good argument that this is not a GDPR issue. Just sending an address without connecting that address to an individual is probably ok. As far as I can tell there's no indication of a customer identity or name in the data payload?

[SorX14]

not a GDPR issue

I don't know so can't argue either way. Some articles I've read mark address as an identifier which would be covered but I'm searching for resources to back my case rather than fight it.

GDPR is a mountain-out-of-a-molehill territory, but I'm definitely uncomfortable with an insecure HTTP request in 2022. Chrome stopped trusting it in 2018 - even a TTL connection without a trusted cert would be an easy first step. If Alpha can update devices remotely, they should be able to renew a cert etc. Makes you wonder how the data is stored at rest on their servers.

IoT has always been security-last, but I don't know of many devices that chat about its installed location so openly. Judging by the the lack-lustre comms, the upgrade/patching process is probably equally a free-for-all.

Can't wait for the first DDOS from a botnet of solar inverters 😂

[bnc-rainman]

As problematic as the unencrypted transfer is I suspect there's likely a good argument that this is not a GDPR issue. Just sending an address without connecting that address to an individual is probably ok. As far as I can tell there's no indication of a customer identity or name in the data payload?

In my opinion, one can argue about how much the IP address is suitable for identifying people. The European Court of Justice has said it is personal. However, it is undisputed that the serial number of the system contained in the data and transmitted in plain text can be assigned to natural persons. (DSVGO Art 4 No.1)

The consumption data, which is also transmitted in plain text, allows profiling according to DSVGO Art 4 No. 4.

Thus, the data may only be processed if one of the reasons mentioned in DSVGO Art 6(1) applies. The only reason that could apply in my opinion is the consent and thus we are in DSVGO Art 7. I am not aware at the Installlation of the solar system to have given such consent.

And that in my opinion the security of the processing according to DSVGO Art 32 number 1 is not fulfilled, I have already written above.

If there are really arguments for it, they must be very very good.

-- edit-- P.S. I mention the problem with the possible lack of consent, because this gives the lack in the security of processing again another dimension. Honestly, we are here in the forum so that we want to share the cloud solution data for our IOT projects.

Translated with www.DeepL.com/Translator (free version)

[CharlesGillanders]

@bnc-rainman - this is not the right place to argue about GDPR and what it does and does not apply to. If you have submitted a data subject access request or a GDPR complaint to Alpha then I wish you the best of luck.

[SorX14]

@bnc-rainman Thank you for your insight, Charles is right that this isn't the place for this conversation but I enjoyed reading your response. FWIW, I reviewed the terms on sign-up for the cloud account which mentions PRC a lot and that everything is shared with your installer. Not sure if explicit consent is given but 🤷

Out of interest, you mention being translated and it seems very good - what is your native language?

Anyway, I've been busy setting up the modbus connection, and it would be cheap enough for most to afford. Use the menu on the device to enable modbus in slave mode - I have a SMILE5. A Raspberry Pi (I'm using a 2B v1.1, but a zero W would work) and a RS485 module (£6 https://shop.pimoroni.com/products/m5stamp-rs485-module?variant=40171638751315). Snip the end off a ethernet cable, blue to B, blue/white to A, RX to GPIO 15, TX to GPIO 14. Set serial port up, and then use https://minimalmodbus.readthedocs.io/. Pop the ethernet cable into the CAN port.

You can get statistics with surprising resolution, about ~0.2s. I can tell when a phone charger is plugged in! You can also force the unit to charge/discharge the battery etc. Basically full control. Will document further once I've matured my POC but this is definitely the way to go if you want reliable comms with your Alpha equipment.

[CharlesGillanders]

I've honestly no idea where the translated text came from. I'm a native English speaker. I thought it was your replies that are translated?

[SorX14]

I've honestly no idea where the translated text came from. I'm a native English speaker. I thought it was your replies that are translated?

Not yourself, but @bnc-rainman where it says 'Translated with www.DeepL.com/Translator (free version)' at the end of their comments. I'm a native English speaker too.

[CharlesGillanders]

I've honestly no idea where the translated text came from. I'm a native English speaker. I thought it was your replies that are translated?

Not yourself, but @bnc-rainman where it says 'Translated with www.DeepL.com/Translator (free version)' at the end of their comments. I'm a native English speaker too.

Homer Simpson moment - Doh! I need to pay more attention to who's commenting.

[bnc-rainman]

@SorX14 I'm from Germany and a native German speaker.

Thanks for your link above with the MQTT repository and your comments on MODBUS. I'm going to put a Homeassistant Docker on my NAS this weekend and test the solution.

[dehsgr]

Next update is arriving tomorrow…

[matt-oneill]

Next update is arriving tomorrow…

a week tomorrow (19-nov), lets hope for a week of stability before it changes again! doesn't mention the dreaded anti-crawling mechanisms this time though