Request Header Signature

[TonyM1958]

Fox has made a change the cloud API that requires a signature to be added to the request header. This stops 3rd party code from accessing the services for the time being.

Signatures are used for the Fox Open API, documented here: https://www.foxesscloud.com/public/i18n/en/OpenApiDocument.html

The example code shows how to generate a signature.

This has been shown to work but only for the very limited Open API end points listed in the document linked above.

We hypothesis that the full cloud API used by the Fox App and web site might use the same signing mechanism. However, at present it appears we are unable to correctly sign a login request to obtain an access token that may allow full API access.

Further updates as / when we make any progress.

[TonyM1958]

Updated code has been produced that adds a signature to each request. We are waiting on a meeting with Fox to work out why our requests do not currently work.

[TonyM1958]

Contrary to initial views, it does not look like Fox has implemented HMAC between client and server. They are adding a signature to their API requests as a means to maintain their integrity.

I found this helpful in providing context: https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-message-signatures

[canton7]

I've been on hold for half an hour, so decided to have a look....

Signing is handled by signature.js, invoked by G.GetSignature in app.js.

It takes as inputs:

• URL
• A token, which seems to be empty? This is also set as the "token" header.
• The language (e.g. 'en'). Set as the "lang" header.
• The timestamp, also set as the "timestamp" header.

Not sure whether it'll also include the body when doing a POST. One would hope so...

The actual signature algorithm is defined in webassembly, in signature.wasm. See begin_signature and end_signature.

It looks like the hashing algorithm they use is actually embedded in the webassembly, which makes it a bit annoying to decompile. A hacky workaround would just be to invoke it however....

[ants-greyton]

The generated API key allows you to make 20 calls per day so not really sure this is going to be an option unless this changes.

[canton7]

Looking at some of the magic numbers in begin_signature strongly suggests this is MD5. The bit of the signature before the . is 128 bits, which supports this.

[canton7]

I'm looking at the signing used by foxesscloud, not their rate-limited API. I think that's what @TonyM1958 is looking at too.

[TonyM1958]

thanks @canton7 - that may be the missing part of the puzzle - I need to update some code to test it.

In the Open Api sample code, they use path, token and timestamp from the header using an md5 hash and hex digest to create the signature. Sounds from what you say like the full api adds Lang to this. The sample code passes Lang, but doesnot use it when generating the signature...

Here's the Open API signature generation code:

    def get_signature(self, _token, _url, _lang='en'):
        timestamp = round(time.time() * 1000)
        signature = fr'{_url}\r\n{_token}\r\n{timestamp}'

        result = {
            'token': _token,
            'lang': _lang,
            'timestamp': str(timestamp),
            'signature': self.md5c(text=signature),
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36'
        }
        return result

[TonyM1958]

It doesn't just work so a bit more digging is needed. I think I need to take the header data from a valid call and see if I can re-create the signature from it...

[alpriest]

lang and timezone are required fields on the newer /generic API fields used for the scheduler too.

It looks to be the md5 hash of urlPath\r\ntoken\r\ntimestamp.something. The something though is puzzling becuase it changes per request for the app but not the website. For the website I get 5245784 - is this consistent with anyone else?

[TonyM1958]

@alpriest As @canton7 mentioned... the login is a post with the username and password in the body of the message. I wonder if they are adding the body into the signature generation string for a post? Its worse than looking for a needle in a haystack though pasting together different parts of the request to see if we stumble upon the correct hash...

[alpriest]

I've not tried the login signature, but ES logs in via the POST already.

To verify the signature though you can login with the website, grab the token from the response, then use that in the signature. However that doesn't verify even with the website's consistent appendage.

As you say, needle in a haystack for now.

[stevetrease]

The something though is puzzling becuase it changes per request for the app but not the website. For the website I get 5245784 - is this consistent with anyone else?

For the website I am also seeing 5245784 in the . part of the signature header.

[TonyM1958]

Thanks for the input - I now have some working code that gets the API data I expect for most of the API. My test suite runs through all the API calls ok but fails with operation timed out when getting battery settings at present.

[stevetrease]

Let me know if I can test anything. I was using you libraries to set forceDiuscharge schedules and for a nightly upload to PVOutput.

[mhzawadi]

Thanks for the input - I now have some working code that gets the API data I expect for most of the API. My test suite runs through all the API calls ok but fails with operation timed out when getting battery settings at present.

Hey, do you have any pointer on how they build the signature?

[TonyM1958]

The Fox web site is also timing out getting the charge times and min soc for the battery status - so this may be a problem at their end rather than in my code?

[ants-greyton]

I get a response using the API key I generated. Its an error but at least I get a proper response instead of just the 406 Not Acceptable error so I guess the format of the request is correct.

{ "errno": 41201, "msg": "Server exception", "result": null }

[TonyM1958]

I've updated and tested as much as my code as I can for now - get_charge() and get_min() fail with operation timed out - but this is the same for the Fox App v1 and the Fox web site. This does limit how much I have been able to test charge_needed() though.

All other calls to the Fox ESS Cloud API are working in my tests and getting forecasts and uploading data to pvoutput is also working.

The code for v0.9.9 is now on github and the foxesscloud module on PyPi has been updated.

This release removes the need to load the random_user_agent module but adds the need to specify your time_zone. The default setting is Europe/London. If you are running in Jupyter Lab and the code fails getting a token, delete the file token.txt that is used to cache your login token and it will get updated with a new token and the time zone.

In addition to other things, you can now get_cell_temps() and get_cell_volts(). These return a list of the cell temperatures and voltages as you might expect.

[stevetrease]

@TonyM1958 That works for me! Thanks.

[ian1182]

Tony, you might want to remove your personally identifiable data from "run_tests.ipynb" in the 0.9.9 download

[TonyM1958]

thanks @ian1182 - I did know I'd done that to get it out there. It's updated now. Nothing too serious unless you decide you want to spam my phone!

GitGuardian also just flagged my commit as exposing secrets as well. Good to know they are watching.

[MartinBlackburn]

or the website I am also seeing 5245784 in the . part of the signature header.

When I was having a dig around, the signature.js was using a function like this to generate those numbers:

var ret = 0;
if (str !== null && str !== undefined && str !== 0) {
  var len = (str.length << 2) + 1;
  ret = stackAlloc(len);
  stringToUTF8(str, ret, len)
}
return ret

stackAlloc is embedded in the webassembly, in the signature.wasm - that's as far as i got.

But if they return based on the length of the string, it might be possible to reverse engineer

[canton7]

The 5245784 is calculated from $func20 in the webassembly I think. It seems to do a lot, but it's fed a constant value (1076) as its input, so I haven't quite figured out what's going on there...

@MartinBlackburn That code is just part of the boilerplate to call webassembly from javascript.