Integration faild to load in HA - UserAgent

benek984 commented 2 years ago

It starts today morning. Sensors values didn't load. So rebooted HA no help.

Below is what I found in the logs:

This error originated from a custom integration.

Logger: custom_components.foxess.sensor
Source: custom_components/foxess/sensor.py:130
Integration: foxess (documentation, issues)
First occurred: 10:32:47 (1 occurrences)
Last logged: 10:32:47

FoxESS Cloud initializaction failed, fix error and restar ha

This error originated from a custom integration.

Logger: custom_components.foxess.sensor
Source: custom_components/foxess/sensor.py:153
Integration: foxess (documentation, issues)
First occurred: 10:32:47 (1 occurrences)
Last logged: 10:32:47

Unexpected error fetching FoxESS data: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 187, in _async_refresh
    self.data = await self._async_update_data()
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 147, in _async_update_data
    return await self.update_method()
  File "/config/custom_components/foxess/sensor.py", line 105, in async_update_data
    token = await authAndgetToken(hass, username, hashedPassword)
  File "/config/custom_components/foxess/sensor.py", line 153, in authAndgetToken
    response = json.loads(restAuth.data)
  File "/usr/local/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Captain0Kangaroo commented 2 years ago

I have the same. No data

I hope this is not the case but Fox have been doing a lot of maintenance on their cloud platform over the last week (its been on & off a number of times. I just hope they haven't pulled the API abilities !?

williamjeccles commented 2 years ago

same here @Captain0Kangaroo

Daibutt commented 2 years ago

@williamjeccles @macxq This does look different than the outages earlier in the week. As Will says, the WebUI is still functional, and it is not "logging out" or losing the token as it normally does when foxess-ha does it's 60 second update/API call, it's like foxess-ha isn't even making a successful connection. I do hope FoxESS haven't chosen to restrict access to the API outside of the WebUI, similar to their restriction on MQTT access. BTW, my logs show the same as @benek984 above.

BigManDave commented 2 years ago

I think they've tried to kick us out. I've had a fiddle in Postman and they're now filtering on the 'User-Agent' Request Header. Continuing to look for a workaround....

BigManDave commented 2 years ago

Just created a PR to resolve the issue. Already got it working in my HA instance. If anyone wants to sort it before the PR is merged and the new release is ready instructions are below:

Edit custom_components\foxess\Sensor.py
Find and edit the headersData = { to read {"token": token, "User-Agent": "Chrome"}
Find and edit the headersAuth = { to read {"Content-Type": "application/json;charset=UTF-8", "Accept": "application/json, text/plain, /", "lang": "en", "User-Agent": "Chrome"}
Save changes to the file and restart HA

benek984 commented 2 years ago

Can confirm it works. @BigManDave thank you for this.

williamjeccles commented 2 years ago

Thanks @BigManDave

macxq commented 2 years ago

Awesome work @BigManDave 💪 Thank you for fixing this issue 🙇

I think they've tried to kick us out.

They can try, but there's almost no chance to secure the web/ui api so that we could not use it - of course it can required a huge effort from our side, or they can change it so often that we will be tired of adapting to changes :/

PS. I hope that they are not reading this 😅

macxq commented 2 years ago

New release has been pushed to HACS

Captain0Kangaroo commented 2 years ago

Nice one. Thought it was all over then.

warmfusion commented 2 years ago

Looks like this has been blocked again;

UserAgent: Chrome gets a 404 from nginx

❯ curl 'https://www.foxesscloud.com/c/v0/device/history/report' -X POST  -H 'Accept: application/json, text/plain, */*' -H 'Accept-Language: en-US,en;q=0.5' -H 'Content-Type: application/json;charset=utf-8' --data-raw '{"deviceID":"REDACTED","reportType":"day","variables":["feedin","generation","gridConsumption","chargeEnergyToTal","dischargeEnergyToTal"],"queryDate":{"year":2022,"month":2,"day":1,"hour":null}}' -H 'token: REDACTED' -H 'User-Agent: Chrome'

<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

whereas UserAgent: Chrome 91 gets a valid response

❯ curl 'https://www.foxesscloud.com/c/v0/device/history/report' -X POST  -H 'Accept: application/json, text/plain, */*' -H 'Accept-Language: en-US,en;q=0.5' -H 'Content-Type: application/json;charset=utf-8' --data-raw '{"deviceID":"REDACTED","reportType":"day","variables":["feedin","generation","gridConsumption","chargeEnergyToTal","dischargeEnergyToTal"],"queryDate":{"year":2022,"month":2,"day":1,"hour":null}}' -H 'token: REDACTED' -H 'User-Agent: Chrome 91'

{"errno":0,"result":[{"variable":"feedin","data"...

macxq commented 2 years ago

doh 😬

new version released

Thank you @warmfusion for catching this out

macxq commented 2 years ago

Wondering for how long this will work 😭 Maybe random user agent generator will help for longer 🤓 https://pypi.org/project/random-user-agent/

www.foxesscloud.com if you are reading, this pleas do not block our access 🙇

warmfusion commented 2 years ago

My own quick patch did this f"Chrome {datetime.now().minute}" to generate a psuedo random user agent ;)

BigManDave commented 2 years ago

@macxq Thanks for the quick release. Had just started looking into it myself :D

Random user agent generator looks like an excellent solution if you can implement it. I've decided to copy the full User-Agent from my browser of choice.

I'd imagine FoxESS aren't happy with a bunch of us fetching data from the API every minute of the day and don't want to simply upgrade their infrastructure to support it. Bastards!

warmfusion commented 2 years ago

Its particularly frustrating given the existence of an MQTT brokering API and formal clients they issue to installers but seem reluctant to issue to end-users.

I assume the goal is to reduce the volume of support queries from uninformed users, relying on installers as middle-men to handle first line support queries.

Give us a proper API endpoint, set the rate limits on the tokens, and ideally have an API that lets us get the data we want rather than the data the website wants and i bet it'll be better for everyone concerned. Just means Fox have to write the API which is the core issue - who pays for that?

I'm considering methods to bypass the foxcloud endpoint entierly and tap into the RS232 serial dongle or the wifi stream itself somehow as that'd be a handy long term solution; my electrical engineering is a bit rusty thou....

benek984 commented 2 years ago

@warmfusion "Its particularly frustrating given the existence of an MQTT brokering API and formal clients they issue to installers but seem reluctant to issue to end-users." . Not sure where you live but, I live in the UK and I can assure you that 98% of the installers in the UK don't know what is the MQTT or API. Before @macxq has wrote this integration I was trying to get some API or RS485 information from Fox support and then from my installer. Fox support said they don't provide this info to end user and my installer as I said before didn't know what I am talking about.

100% agree that some local pulling using RS485 or Ethernet (I have socket on my inverter) would be the best and the most reliable options but for that we need some technical information.

@macxq BIG THANK YOU FOR SUCH QUICK ACTION

Captain0Kangaroo commented 2 years ago

I think we're offline again?

Daibutt commented 2 years ago

Same here. HA data not updating since about 02:30. Mobile app also not working.

Edit: Web UI also not working. Looks like a total foxesscloud crash.....

Captain0Kangaroo commented 2 years ago

Actually it looks like it might be a Fox problem as I can't load data in their app...

warmfusion commented 2 years ago

I think they've started blocking by origin ip now.

Using this app on my mobile data works but not on my wifi.

No bypassing that one with a quick code patch.

How frustrating

On Wed, 2 Feb 2022, 07:20 Captain0Kangaroo, @.***> wrote:

Actually it looks like it might be a Fox problem as I can't load data in their app...

— Reply to this email directly, view it on GitHub https://github.com/macxq/foxess-ha/issues/28#issuecomment-1027652071, or unsubscribe https://github.com/notifications/unsubscribe-auth/AARQGKESGPURACLNNOXKWZTUZDLMTANCNFSM5MA2ZQNA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

BigManDave commented 2 years ago

Have we been IP blocked? Can't load foxesscloud.com on WiFi but can on mobile data but my inverter is offline.

Disclaimer: I'm not out of bed yet 😆

BigManDave commented 2 years ago

Yeah pretty sure we're right. Don't get a ping on wifi but do on 4G. Guess im out of bed now as my Solar installer is about to get reamed

benek984 commented 2 years ago

Just checked and all works as expected in the mobile app and home assistant.

blocking IP wouldn’t make sense as most und users are on dynamic IP so that would be a lot of manual work for them. Also are they able to determine who ( meaning HA or Mobile App or webpage) make the API call? We just want to access our OWN data so if I am sitting on the mobile app all day long I should be able access my data.

Local way of retrieving data is something Must Have rather than nice to have.

BigManDave commented 2 years ago

Just rerouted my traffic to foxesscloud through my VPNs and all is working. Will have to do until my installer answers the phone lol

Captain0Kangaroo commented 2 years ago

Still not working for me on HA (over 4G Cloud) or using the Fox app ? Not sure what's going on.

Do you think they have blocked my home IP?

BigManDave commented 2 years ago

Yeah it looks like they've blocked the IP for everyone that was running this integration. The block has even stopped us sending usage! Insane really.

Captain0Kangaroo commented 2 years ago

I have a number for Ben (FoxESS UK rep) if anyone wants it to chat to him about it. Maybe he and speak with head office? Totally ridiculous given they have an API documented.

nigelspowage commented 2 years ago

It doesn't look like it's IP blocked me, so possibly not quite everyone was blocked?

warmfusion commented 2 years ago

I've emailed Ben and support.uk but i'm not particularly optimistic as my recent comms have explicitly stated they have no API.

I've only had this gear 2 months and part of my decision was being able to get this data into homeassistant. :-(

BigManDave commented 2 years ago

I guess it isn't exactly an api we're using, we're just executing a select few of the requests that the web dashboard makes and then extracting the json. What we're doing is therefore less intensive than setting up the web dashboard to auto refresh every minute, not exactly unreasonable in my opinion. But it is absolutely unreasonable to now IP block us from even sending our data. For them to even have the MQTT broker (although not let anyone use it) they must have at least anticipated the demand for realtime data, how could they possible justify blocking us instead!

warmfusion commented 2 years ago

I'll need to check the code over this evening, but I'm thinking what about this plugin in particular would be considered unusual compared to the main website that justifies a block like this.

Runs every minute - Could we run every half hour and pull back a batch of data?
Performs a full authentication cycle each minute - Could the response auth token be cached and refreshed periodically?
Pulls raw data for the whole day - Can this be further filtered to reduce the data set being grabbed (eg last hour?)?

Setting the user-agent to HomeAssistant::foxess-ha v1.2.3 - Submit issues to https://github.com/macxq/foxess-ha/issues might also be a friendly thing to do too

BigManDave commented 2 years ago

I think just the fact we're all fetching every minute whereas the majority of their installations are quite possibly daily at most.

I've just changed mine to fetch every 5 minutes, since that's how often my inverter sends data anyway. I think if we extend this to once or twice an hour the only downside is for those of us using the reimann sum for more accurate Solar energy figures, as they will be significantly less accurate?
I didn't realise we were authenticating for each call. Looking at my testing in Postman my token hasn't actually changed in a very long time, so definitely should be caching it.
We might be able to switch from using total_Increasing sensors to just total? Not sure

macxq commented 2 years ago

I'm out of home for a few days so it is hard for me to take any actions. My inverter still sends data to the cloud and I can see them in the mobile app but the HA integrations stopped working.

@warmfusion @BigManDave thx for dining into it:

5 min period wold be fine and sufficient ot take some actions (automation based on solar production) in my opinion 30 is to big
yes token could be cached
the api a have discover does not suport filtering, it is returning whole day

Maybe we will need to find a way to bypass ip blocking, by using free proxies 🐈 🐁

The other solution I see is to change the dns records for foxesscloud.com to route the measurements by our proxy serwer that will capture them. EDIT. yes I know - there can be a problem with certificates

BigManDave commented 2 years ago

@macxq Interesting idea with the DNS change... Just had a look on my firewall and it might not be using TLS to push the data. Doing some packet capturing now to dig deeper.

Captain0Kangaroo commented 2 years ago

They really don't like me 🤣 I've got no data at all within the app or FoxCloud - just says offline.

macxq commented 2 years ago

@Captain0Kangaroo is foxesscloud.com loading for you?

Captain0Kangaroo commented 2 years ago

@Captain0Kangaroo is foxesscloud.com loading for you?

No nothing mate. No app or HA. I've totally been chopped off. Internet works at home so not a broadband issue.

warmfusion commented 2 years ago

Same. Works on mobile data, so it's an IP level block at their end.

I emailed fox support but no reply yet. I've also chopped out the plug-in from my installation so if it's a temporary block I won't be immediately rebanned.

Hopefully there will be a reply to my support query that can help us find a mutual approach to getting access to our own data from equipment we bought and paid for.

On Wed, 2 Feb 2022, 14:53 Captain0Kangaroo, @.***> wrote:

@Captain0Kangaroo https://github.com/Captain0Kangaroo is foxesscloud.com loading for you?

No nothing mate. No app or HA. I've totally been chopped off. Internet works at home so not a broadband issue.

— Reply to this email directly, view it on GitHub https://github.com/macxq/foxess-ha/issues/28#issuecomment-1028020165, or unsubscribe https://github.com/notifications/unsubscribe-auth/AARQGKARG4V5GZ4DCZQL6RLUZFAOJANCNFSM5MA2ZQNA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

williamjeccles commented 2 years ago

Every time i've asked a question I get a generic answer back about if I'm an installer or end-user.

macxq commented 2 years ago

Fingers crossed that this wold be just a temporary block 🤞

I'm recommending everyone to disable the integration until the most webui looking request will be implemented with all the proper header params:

 -H 'Connection: keep-alive' \
  -H 'sec-ch-ua: " Not;A Brand";v="99", "Google Chrome";v="97", "Chromium";v="97"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36' \
  -H 'Accept: application/json, text/plain, */*' \
  -H 'X-Requested-With: XMLHttpRequest' \
  -H 'lang: en' \
  -H 'sec-ch-ua-platform: "macOS"' \
  -H 'Sec-Fetch-Site: same-origin' \
  -H 'Sec-Fetch-Mode: cors' \
  -H 'Sec-Fetch-Dest: empty' \
  -H 'Referer: https://www.foxesscloud.com/bus/device/inverterDetail?id=xyz&flowType=1&status=1&hasPV=true&hasBattery=false' \
  -H 'Accept-Language: pl,en-US;q=0.9,en;q=0.8,de;q=0.7,nl;q=0.6' \
  --compressed

Captain0Kangaroo commented 2 years ago

I've removed it from HACS. Is that's all that's needed for now?

benek984 commented 2 years ago

All still works for me. Looks like there was two short drops in the morning but except that all is fine.

BigManDave commented 2 years ago

@benek984 That's curious. How long have you been running the integration? Do you know if you have an dynamic IP address?

benek984 commented 2 years ago

I have integration running from 31 Dec 2021. I am on dynamic IP but looking on myip sensor in HA my IP haven't changed since that time.

nigelspowage commented 2 years ago

Mine was working earlier today but the foxesscloud would intermittently report me as offline so I removed the plugin. I haven't noticed it going offline since. I'm only had this installed since Monday and my IP hasn't changed in that time.

BigManDave commented 2 years ago

Thanks @benek984. Makes me think they've only blocked the top so many of us that've been running the integration for a while in that case.

Daibutt commented 2 years ago

Seems I'm screwed! Removed foxess-ha via HACS, rebooted hub, rebooted HA and still can't get access via WebUI or phone app (on wifi or 4G) ... not a happy bunny !!

macxq commented 2 years ago

@Daibutt give it some time, usually such locks are not removed immediately

warmfusion commented 2 years ago

in my case, its a flat out connection drop.

www.foxesscloud.com resolves to 8.209.116.72 (and does so globally; https://dnschecker.org/#A/www.foxesscloud.com )

A curl request from my laptop connected to my home ISP results in interesting behaviour.

The TCP Connection establishes, (Connected to ..) and begins a HTTPS SNI handshake but just hangs till the timeout is reached;

> curl --connect-time 2 --verbose --trace-time https://www.foxesscloud.com 

20:03:02.031019 *   Trying 8.209.116.72...
20:03:02.032078 * TCP_NODELAY set
20:03:02.034453 * Connected to www.foxesscloud.com (8.209.116.72) port 443 (#0)
20:03:02.034621 * ALPN, offering h2
20:03:02.034650 * ALPN, offering http/1.1
20:03:02.043143 * successfully set certificate verify locations:
20:03:02.043181 *   CAfile: /etc/ssl/cert.pem
  CApath: none
20:03:02.043275 * TLSv1.2 (OUT), TLS handshake, Client hello (1):
20:03:04.030017 * Operation timed out after 2000 milliseconds with 0 out of 0 bytes received
20:03:04.030057 * Closing connection 0
curl: (28) Operation timed out after 2000 milliseconds with 0 out of 0 bytes received

This happens on http (port:80) as well so it certainly feels like some sort of web application firewall dropping my connection packets after their edge has permitted the connection.

The whois for 8.209.116.72 shows its an Alibabacloud.com hosted ip, which aligns with Fox ESS being a chinese based company (And why I'm so keen to have my own data locally hosted...)

TLDR; In my case; my IP is blocked by the Fox ESS system preventing their own equipment reporting data to them, while also preventing this plugin to get data back again.

I have contacted fox support, who gave some first-line suggestions of 'restart the wifi dongle' and I have continued to clarify the position.

Lets see if the block is permanant or temporary...

macxq / foxess-ha

Integration faild to load in HA - UserAgent #28