Integration faild to load in HA - UserAgent

[benek984]

It starts today morning. Sensors values didn't load. So rebooted HA no help.

Below is what I found in the logs:

This error originated from a custom integration.

Logger: custom_components.foxess.sensor
Source: custom_components/foxess/sensor.py:130
Integration: foxess (documentation, issues)
First occurred: 10:32:47 (1 occurrences)
Last logged: 10:32:47

FoxESS Cloud initializaction failed, fix error and restar ha

This error originated from a custom integration.

Logger: custom_components.foxess.sensor
Source: custom_components/foxess/sensor.py:153
Integration: foxess (documentation, issues)
First occurred: 10:32:47 (1 occurrences)
Last logged: 10:32:47

Unexpected error fetching FoxESS data: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 187, in _async_refresh
    self.data = await self._async_update_data()
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 147, in _async_update_data
    return await self.update_method()
  File "/config/custom_components/foxess/sensor.py", line 105, in async_update_data
    token = await authAndgetToken(hass, username, hashedPassword)
  File "/config/custom_components/foxess/sensor.py", line 153, in authAndgetToken
    response = json.loads(restAuth.data)
  File "/usr/local/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[Captain0Kangaroo]

I have contacted fox support, who gave some first-line suggestions of 'restart the wifi dongle' and I have continued to clarify the position.

I contacted Ben the UK rep and told him the situation but he had no idea what I was on about and I could have been talking another language when I mentioned HA and the integration. He basically came back with "have you got an internet connection and is the dongle plugged in" as they have no reported problems 🙄. I tried to explain again but..... it was useless.

🤞 as I'm totally in the dark with no access to their cloud app (which was useless anyway which is why we're all using this great integration 😊).

[BigManDave]

@williamjeccles Were you blocked yesterday, and then again today on a new IP? Have you tried changing the frequency from 1 minute to 5? I've not been blocked yet on my VPN, although I am load-sharing between two VPNs so should effectively be fetching data from Fox once every 10 minutes on 2 IPs.

[warmfusion]

I removed this plugin from my installation yesterday afternoon, the inverter is trying to reach fox and i occasionaly check from my browser/fox-app but i remain blocked after 30 hours or so.

[williamjeccles]

@BigManDave was blocked yesterday, reset my router around 2pm UK time and got a new ip issues and everything was back, but was blocked again around 2am this morning. I did not change my refresh feq, could you confirm what I change to do this?

@warmfusion assume you use a staic ip address so are banned for life on that IP or have you just not rebooted your router to be issued a new dynamic ip?

[BigManDave]

@williamjeccles in sensor.py have a look for SCAN_INTERVAL , it's quite near the top :)

[BigManDave]

I believe my ISP, although not officially allocating me a static IP, assigns me more of a 'sticky IP'. Might be able to contact them and ask to be given a new one if you're not contractually given a static. Might have to make up more of an excuse than some not-so-nice Chinese folk have blocked you from your solar platform. Perhaps some unsavoury characters have gained your IP and are making threats or something.

[williamjeccles]

Ok, i've updated my scan_interval so i'll monitor over the next 24hs and see if that fixes it. I'd be happy to even go to 15mins if this works to keep their platform happy. I don't want them to keep blocking me as it's creates reporting gaps and i'm trying to use my HA system for some real automation and proper reporting. Thanks for the help.

[BigManDave]

@Captain0Kangaroo That's the system complaining that you've added configuration for FoxESS without it knowing the integration exists on your install (since you haven't loaded it yet). Either force a system restart using docker or however you're hosting HA, or comment out your FoxESS config, restart, uncomment and then restart again 👍

[Captain0Kangaroo]

@Captain0Kangaroo That's the system complaining that you've added configuration for FoxESS without it knowing the integration exists on your install (since you haven't loaded it yet).

Either force a system restart using docker or however you're hosting HA, or comment out your FoxESS config, restart, uncomment and then restart again 👍

Thanks I got it working with a forced reboot. 👍

[Daibutt]

Interesting..... I managed to get reconnected by turning off router for long enough that it got a new IP. Then, when checking inverter, the WiFi push time had been put back up to 300secs (default) ... presumably by FoxESS?! I had previously changed it to 60 secs. Guess they don't like the high frequency of API requests, even from their own kit!

[macxq]

didn't foxess blocked those who have modified WiFi push interval ?

[williamjeccles]

@macxq they don't block it based on wifi push any more as instead, they're updating the value to 300 automatically every few hours it's reverting on it's own for the last few weeks which is what @Daibutt is seeing. I noticed it a while back.

[williamjeccles]

Started to see this, which I can only assume means they have an issue?

[williamjeccles]

this is when trying to access the webui via foxesscloud.com

[BigManDave]

Same here. Jesus christ they suck!

[williamjeccles]

They can't blame us for this one as most people have suspended their HA usage while things settle down. Just rubbish api capacity I'm sure. they're running it on an Alibaba micro instance

[williamjeccles]

@BigManDave it's back!

[Captain0Kangaroo]

So I put my HA back online after getting a new IP overnight. I set the call time to 10 minutes which worked for about 2/3 hours and I've just been blocked again!

This error originated from a custom integration.

Logger: custom_components.foxess.sensor Source: components/rest/data.py:60 Integration: foxess (documentation, issues) First occurred: 12:50:44 (31 occurrences) Last logged: 14:02:51

Unexpected error fetching FoxESS data: 'bool' object has no attribute 'encode' Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 187, in _async_refresh self.data = await self._async_update_data() File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 147, in _async_update_data return await self.update_method() File "/config/custom_components/foxess/sensor.py", line 108, in async_update_data await getErnings(hass, headersData, allData, deviceID) File "/config/custom_components/foxess/sensor.py", line 172, in getErnings await restEarnings.async_update() File "/usr/src/homeassistant/homeassistant/components/rest/data.py", line 60, in async_update response = await self._async_client.request( File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1500, in request request = self.build_request( File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 340, in build_request headers = self._merge_headers(headers) File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 407, in _merge_headers merged_headers.update(headers) File "/usr/local/lib/python3.9/site-packages/httpx/_models.py", line 966, in update headers = Headers(headers) File "/usr/local/lib/python3.9/site-packages/httpx/_models.py", line 837, in init self._list = [ File "/usr/local/lib/python3.9/site-packages/httpx/_models.py", line 841, in normalize_header_value(v, encoding), File "/usr/local/lib/python3.9/site-packages/httpx/_utils.py", line 54, in normalize_header_value return value.encode(encoding or "ascii") AttributeError: 'bool' object has no attribute 'encode'

[BigManDave]

Still working here. (Again I am sharing my foxess connections through multiple VPNs) I'm also using a user-agent that I took directly from my own browser, if that helps

[williamjeccles]

@BigManDave what does your user agent look like in the sensor.py so I can copy and paste yours to test it?

[BigManDave]

Just so long as FoxESS aren't lurking here... ;) Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:96.0) Gecko/20100101 Firefox/96.0

[BigManDave]

@williamjeccles Yikes! Bet they'll block it soon... Might be worth grabbing a random one from here so we're not all using the same: https://developers.whatismybrowser.com/useragents/explore/

[BigManDave]

@williamjeccles I can do you one better! Just implemented the random-user-agent generator that @macxq mentioned. See #39 if you want to give it a try

[dajmirycyny]

I removed integration after being blocked twice same day (had to get new ip). Will wait until wiser minds will work it out, however, if you need anything from me to test some stuff, let me know. I guess in the end I will have to end up anyway with proper energy meter in my home. Shame that I paid for the foxess Wi-Fi dongle when they are not giving good support for it:/

[macxq]

@BigManDave released 🚀 hope this will help 🤞

[williamjeccles]

thank you @BigManDave & @macxq

[williamjeccles]

@macxq / @BigManDave should we make it standard config that it's set to 10min refreshes to prevent us upsetting foxess again?

[Alllfff]

@BigManDave does increasing the update period reduce the chance of being blocked? Thanks

[williamjeccles]

@Alllfff it appears so but we won't know until we can run it for a full day or so

[BigManDave]

@williamjeccles I'd say leave the default at 5 for now, since that's the default for pushing data from the inverters. It reduces the load that they weren't happy with by 80%, so let's see if they're ok with that. But also not a bad idea to leave yours at 10 if you want to be more cautious.

[williamjeccles]

@BigManDave I'm not getting data after updating to the latest version. Restarting to make sure that my random user agent isn't on the blacklist.

[benek984]

We need to open support case with fox but ALL of as at the same time with exactly the same description, if that won't help we need to start to make bad publicity so more people can be at least aware that this is not good brand for inverters. We are them customers and all we want is to access our DATA. Now when we are blocked our inverters cant report correctly so we will be missing on correct readings.

I rebooted my router twice but getting the same IP address. Will have to turn it off for several minute but can't do it now as working from home.

[BigManDave]

@williamjeccles Sorry, I ballsed that one up when I copied from HA. Could you try editing manifest.json and move "random_user_agent" from dependencies to requirements?

[dajmirycyny]

My inverter took about 3 Hours after getting New IP to start posting data back to cloud. If only they would Just give US MQTT access :/

[williamjeccles]

@BigManDave works fine after that tweak

[BigManDave]

Phew! Thanks for testing @williamjeccles

[Captain0Kangaroo]

Is the latest version confirmed as working?

[benek984]

yes, it is working as long as your IP is not blocked. Let see for how long. I bet they are reading this git conversation. @Foxess please do understand we paid for this equipment and data generated belong to each od us. If you have better way of giving us access to our data just let us know.

[warmfusion]

I would recommend caution at this time - Fox are clearly implementing rules to block access to the data via the mechanism used by the plugin, and you may find your home IP gets blocked again, or perhaps they escalate further and begin to remove the API endpoint itself by implementing more complex request requirements.

I continue to liase with support on this matter, attempting to convince them there is an IP block in place and its not just an issue with the inverters wifi dongle. My IP remains blocked, with no information from my inverter being sent, and no access to foxesscloud.com.

[williamjeccles]

@warmfusion I've made a copy of their product promotion content from their website and documentation. If they want to play hardball about accessing a product that we paid for then we can set up a class action against them using a no win no fee company and force them to provide us the service we paid for. If they have any sense and are reading this they should be supporting us to integrate with their solution as it will lead to more sales and more revenue when they show they're pro developer as lots of us want this data in our smart homes.

Sadly right now they're clearly fire fighting an issue behind the scenes and have been using a hammer to resolve the issue. I'm also not convinced they're targetting us specifically but more likely misidentifying us as some kind of dos attack. These platform stability issues have existed long before this plugin was developed. I've used foxess hardware for a while now and these outages have been common. Regardless, the latest update and slowing down the polling seems to keep them happy.

@BigManDave The latest update and having the refresh level set to 5 minutes seems to satisfy their undocumented requirements. I will continue to monitor but i'm happy to poll less often if it keeps them happy. We might want to look into cache and reuse of tokens to keep the number of calls as low as possible.

[BigManDave]

@williamjeccles Yeah I'm still able to access too, I just hope those with IP bans find that they expire soon. Agreed. The next best thing for us to do is sort that token. I can give it a look but I think it might be out of my skillset.

[Captain0Kangaroo]

@BigManDave I was just IP blocked :( moving to 15mins to see if that works any better.

I'm not sure its a block as i can't event get their website to load whereas before it loaded and logged in but didn't show any data.

This could just be a Fox Outage?

NO your right.. its blocked...

[macxq]

@williamjeccles there was an error in code and the SCAN_INTERVAL was not taken into consideration while setting scan interval, new release has been shipped 14.4

[macxq]

I was also blocked - no data since 10:12 and device is offline 😭

[BigManDave]

thanks @macxq I didn't spot that! Yeah they've blocked 3 of my VPNs all at once. Guess I'm spinning up another... lol So stupid they're blocking us from sending data. When I did a packet capture we send on port 10001 so fuck knows why they've not just blocked us on 443 and left sending open. They're just digging a hole at this point.

[benek984]

They blocked me around an hour ago. So no data from my batteries. Thanks god for todays wether in the UK as not missing much on energy production. lol

I have emailed support. Let's see what they reply with. (if anything)

[Daibutt]

I only reinstalled foxess-ha about 2 hrs ago (so my IP hadn't been polling for over 24hrs). I've now got my WiFi push interval set at 10 mins and my HA scan interval set to 15 mins (some data is better than no data!), so I should be accessing API less than if I just had inverter WiFi push set at 5 mins. So far I'm still getting data in HA and in the WebUI. Will update if this changes!! p.s. I've also got latest 14.4 release

[Captain0Kangaroo]

I only reinstalled foxess-ha about 2 hrs ago (so my IP hadn't been polling for over 24hrs).

I've now got my WiFi push interval set at 10 mins and my HA scan interval set to 15 mins (some data is better than no data!), so I should be accessing API less than if I just had inverter WiFi push set at 5 mins.

So far I'm still getting data in HA and in the WebUI.

Will update if this changes!!

p.s. I've also got latest 14.4 release

When I updated last night (9pm) it ran through until about 9:30 this morning. At that point I thought it was going to last but seems not 🙄

[BigManDave]

Well this is interesting. Just started up a new VPN connection that I've never used before and it isn't working either. Makes me think they've now blocked a chunk of known VPN IP addresses. Oh dear, oh dear.

[nigelspowage]

I'm not using the homeassistant extension anymore but I'm watching my usage on foxesscloud.com. I can confirm that sometimes it reports the device status as offline even though I'm not blocked.

Basically I believe their infrastructure isn't handling all the devices 100% of the time. This means you'll see some transient outages with the HA extension. Obviously they're doing some blocking as well, so you'll see a combination of the two issues.