search

macxq / foxess-ha

Home Assistant & FoxESS integration. Monitor you photovoltaic installation directly from HA ☀️ ⚡️
119 stars 37 forks source link

Integration faild to load in HA - UserAgent #28

Closed benek984 closed 2 years ago

benek984 commented 2 years ago

It starts today morning. Sensors values didn't load. So rebooted HA no help.

Below is what I found in the logs:

This error originated from a custom integration.

Logger: custom_components.foxess.sensor
Source: custom_components/foxess/sensor.py:130
Integration: foxess (documentation, issues)
First occurred: 10:32:47 (1 occurrences)
Last logged: 10:32:47

FoxESS Cloud initializaction failed, fix error and restar ha
This error originated from a custom integration.

Logger: custom_components.foxess.sensor
Source: custom_components/foxess/sensor.py:153
Integration: foxess (documentation, issues)
First occurred: 10:32:47 (1 occurrences)
Last logged: 10:32:47

Unexpected error fetching FoxESS data: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 187, in _async_refresh
    self.data = await self._async_update_data()
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 147, in _async_update_data
    return await self.update_method()
  File "/config/custom_components/foxess/sensor.py", line 105, in async_update_data
    token = await authAndgetToken(hass, username, hashedPassword)
  File "/config/custom_components/foxess/sensor.py", line 153, in authAndgetToken
    response = json.loads(restAuth.data)
  File "/usr/local/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Alllfff commented 2 years ago

@BigManDave I was just IP blocked :( moving to 15mins to see if that works any better.

I was blocked this morning even with 15 minutes interval set, but this was before v14.4 was released.

BigManDave commented 2 years ago

@warmfusion Do you have any direct contact info for Fox support? I called the UK contact number yesterday but got told someone would get back to me -_-

BigManDave commented 2 years ago

@williamjeccles I've just thrown togther a temporary workaround for the authentication. https://github.com/BigManDave/foxess-ha/commit/2554254144a02335e7f63cd08c2b3bc0b5159511 Instead of using username and password at all, you need to replace those with your FoxESS token in your HA config. image You can get your token from logging in to FoxESS in a browser and inspecting the Request Headers in the browser dev-tools (F12). Feel free to give it a try, although it is ENTIRELY untested since all my VPNs seem to be blocked :(

BigManDave commented 2 years ago

Scratch that last one. Looks like it might have been even easier to properly fix token not being cached. Got a different branch to test, just a one-liner this time! https://github.com/BigManDave/foxess-ha/commit/503ddf797dc38c431c9535eeadd58e0b6bb08299 So I've tried moving the fetch-token API call from the update_data job to the setup_platform job. To me that sounds like it should now only run on integration initialisation.

Daibutt commented 2 years ago

@BigManDave With this mod does that mean that accessing the Foxesscloud webui or phone app would cause HA integration to "lose" the token & need to be reinitialised?

BigManDave commented 2 years ago

@Daibutt It shouldn't result in that. Unfortunately I'm unable to test it since I'm completely blocked out. But I've had the same token now for many, many weeks so I can't see how it'd need re-fetching.

Daibutt commented 2 years ago

@BigManDave OK, cool :)

dajmirycyny commented 2 years ago

By the way, have you noticed that besides foxess cloud app there is also from the same developer PVHub app which works with same credentials and looks almost identically but has some extra features like trend zooming? Wonder why it’s split into 2 different apps

BigManDave commented 2 years ago

Just tried that app (on mobile data) and couldn't log in. Maybe I don't have access with my end-user account?

nigelspowage commented 2 years ago

foxesscloud has gone down for me on two different IPs and I'm not using the homeassistant plugin, so it's not a block. It's probably just down.

warmfusion commented 2 years ago

Hi everyone,

I've been working with the support folks at Fox UK to try and understand whats going on.

The simplest summary would be that their servers are not designed to accommodate the kinds of requests this plugin is making and the additional load its putting on their platform is resulting in much of the instability we've seen in the past, including today taking down their servers entirely.

My opinion of the situation, and not from Fox, would be that the website and app is designed to make infrequent (perhaps even just one) request to their data feeds and to then rely on the local client for display filtering etc. This architectural decision means the backend is expecting comparatively few, but bulky requests from end users. The HA plugin however tips this architecture by instead performing many (every few minutes), equally bulky requests.

I have volunteered some of my time to see how I could help improve matters for Fox on this regard - Scaling API services is a very challenging problem to solve, and realistically Fox are a hardware company with a data-frontend for hardware engineers (installers) so its not unreasonable that they would have challenges in delivering a scalable metrics API service.

I'd recommend that in the meantime, you consider disabling your plugin so that data is at least being sent to Fox without causing blocks to your IP space and further antagonising their technical teams.

Furthermore, Fox have suggested contacting them for 'authorised' access to the data feeds, a process that I am attempting to clarify so we can have a nice process outlined in the readme of this project and a healthy long-term relationship with Fox rather than a cat-and-mouse game.

I'm absolutely as frustrated as everyone on this thread, my own equipment is 3 months old and not having access to my data in this way is certainly not my idea of a good customer experience.

I'll keep this issue updated if I find out any more information, and I really do hope that I can work with some of Fox'es technical team to find a workable approach for everyone involved.

To repeat - Fox have asked that we disable the plugin to allow their servers to recover; Not an ideal situation of course, but I'd prefer to at least have my data being stored somewhere to view it vs being frequently blocked.

Cheers

benek984 commented 2 years ago

Is hashing the config enough to disable the plugin?

Screenshot_04_02_2022__18_45
macxq commented 2 years ago

@benek984 yes, hashing the config is enough to disable the plugin

@warmfusion thx for reaching out Foxess and sharing the update with us 👍 I also open to cooperate with the on the api design, we do not need the whole historical data that we currently fetching but this is the only way to get some of the data that we are interested in.

Do you know what the process of being unblocked ? will this happen automatically after disabling the plugin after some time ?

Captain0Kangaroo commented 2 years ago

Can anyone confirm if the Fox Cloud is currently down?

I disabled HA this morning and obtained a new IP and the cloud was working fine until mid afternoon and it all went off line.

I don't think I'm IP Blocked as I literally turned off my Raspberry Pi so no data requests from HA.

benek984 commented 2 years ago

I am pretty much in the same situation. Integration DISABLED Had my router rebooted got new IP, can get to fox cloud portal but can't see any data from my inverter. I am thinking of rebooting INVERTER or at least WiFi dongle.
Any ideas?

warmfusion commented 2 years ago

The easiest quick way to check is with https://downforeveryoneorjustme.com/foxesscloud.com

That website makes a quick connect test to the website but otherwise it's quite limited.

It currently says it's working, and I can confirm the website works when on my mobile so any local issues are likely related to your own connection.

My home IP remains blocked.

On Fri, 4 Feb 2022, 19:55 benek984, @.***> wrote:

I am pretty much in the same situation. Integration DISABLED Had my router rebooted got new IP, can get to fox cloud portal but can't see any data from my inverter. I am thinking of rebooting INVERTER or at least WiFi dongle. Any ideas?

— Reply to this email directly, view it on GitHub https://github.com/macxq/foxess-ha/issues/28#issuecomment-1030303942, or unsubscribe https://github.com/notifications/unsubscribe-auth/AARQGKERXUKCGX57IEIW4VDUZQVLTANCNFSM5MA2ZQNA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

dajmirycyny commented 2 years ago

I can login From app. Disabled plugin 2 days ago

williamjeccles commented 2 years ago

Hi everyone,

I've been working with the support folks at Fox UK to try and understand whats going on.

The simplest summary would be that their servers are not designed to accommodate the kinds of requests this plugin is making and the additional load its putting on their platform is resulting in much of the instability we've seen in the past, including today taking down their servers entirely.

My opinion of the situation, and not from Fox, would be that the website and app is designed to make infrequent (perhaps even just one) request to their data feeds and to then rely on the local client for display filtering etc. This architectural decision means the backend is expecting comparatively few, but bulky requests from end users. The HA plugin however tips this architecture by instead performing many (every few minutes), equally bulky requests.

I have volunteered some of my time to see how I could help improve matters for Fox on this regard - Scaling API services is a very challenging problem to solve, and realistically Fox are a hardware company with a data-frontend for hardware engineers (installers) so its not unreasonable that they would have challenges in delivering a scalable metrics API service.

I'd recommend that in the meantime, you consider disabling your plugin so that data is at least being sent to Fox without causing blocks to your IP space and further antagonising their technical teams.

Furthermore, Fox have suggested contacting them for 'authorised' access to the data feeds, a process that I am attempting to clarify so we can have a nice process outlined in the readme of this project and a healthy long-term relationship with Fox rather than a cat-and-mouse game.

I'm absolutely as frustrated as everyone on this thread, my own equipment is 3 months old and not having access to my data in this way is certainly not my idea of a good customer experience.

I'll keep this issue updated if I find out any more information, and I really do hope that I can work with some of Fox'es technical team to find a workable approach for everyone involved.

To repeat - Fox have asked that we disable the plugin to allow their servers to recover; Not an ideal situation of course, but I'd prefer to at least have my data being stored somewhere to view it vs being frequently blocked.

Cheers

I asked them 9+ months ago, long before this plugin was a thing for official access and sent them questions around how to integrate with them or access my data locally or via an api and they just ignored me or replied with generic answers that didn't even seem like they had even read what I was asking. Very disappointing to read this update and the deflection that Craig from Foxess has sent me which is similar to your update.

I will stress that they had uptime issues long before this integration existed, while it may not have helped, it is not the cause of their platform capacity issues. They need to scale their systems to handle demand, plain and simple. hundreds of inverts are being installed each month in the UK alone and these guys are a global company with access to billions in resources, so I don't believe them, they're doing us a disservice by not properly developing and scaling their software offering, This will only get worse with every week that goes by as more and more inverters go live streaming data. I'd love a call with their technical folks to discuss why scaling a cloud based api is such an impossible task because I do it every day.

More than happy to disable my integration on the promise that they're working on providing proper access, with regular updates from them on progress/development otherwise sadly I'll be forced to login at least once a day via HA to simply capture my solar totals because this isn't available in their poor excuse of an app. I don't really need realtime data but it has been handy, but a data point even every hour would be perfect in the mean time until they fix the bigger issue.

I think we're all on the same page and it needs to be made clear to Foxess that we're more than happy to work with them and co-operate with them on a solution, we're not trying to cause them a problem, simply access the data that we own and generate. If they supported us correctly this would only lead to them being more successfull.

BigManDave commented 2 years ago

I've received a similar response from Craig. 'server down due to people requesting data without permission... We only support the fox app' OK, but the issue isn't how we're requesting the data, it's that we're doing it more than the once per day that they thought everyone would check the app.

Quick wins would be for them to give us a proper solar kwh figure so we don't have to fetch pvpower frequently for an accurate reimann sum. I reckon we can then fetch data like, once an hour to satisfy the HA dashboard and couple that with fixing caching of the token in the integration. It would be perfectly reasonable that they allow/support us to fetch once per hour.

According to their advice there's nothing stopping me setting up an auto refresh of the website, because that's supported... 🤦‍♂️

williamjeccles commented 2 years ago

Reading the email from Craig at UK Foxess support they have shut down their cloud platform service currently, inverters show online but all the data is frozen in time and not updating.

Once and for all, I've asked them for a meeting with their technical and product teams to meet with anyone here who is interested in being part of the solution with them. Instead of multiple threads going between us and them I think we should all get on one call with them to discuss this.

We just want access to the inverter data, even once per hour would be fine but real time should be possible in all honestly, it's 2022 after all.

@BigManDave if we all disabled our plugin and just used the web app or web gui they would still have downtime because they're not setup to scale to their customer base wanting to monitor their investment. Plain and simple, they are the ones who need to fix this, we're not in the wrong for using our hardware.

Their own web gui exposes a WebSocket that is streaming messages every second, you'd think that was more resource-intensive than a poll every x minutes.

BigManDave commented 2 years ago

Just spotted I've been unbanned. Guessing that's because I emailed them my IP. Looks like the inverter is online and uploading however they've disabled the data API calls so the app is barely functioning. That said, they've left the websocket open so we can observe live data at a glance at least.

warmfusion commented 2 years ago

Same; Back online for me, but i'm keeping the plugin offline for the time being.

On Sat, Feb 5, 2022 at 10:12 AM BigManDave @.***> wrote:

Just spotted I've been unbanned. Guessing that's because I emailed them my IP. Looks like the inverter is online and uploading however they've disabled the data API calls so the app is barely functioning. That said, they've left the websocket open so we can observe live data at a glance at least.

— Reply to this email directly, view it on GitHub https://github.com/macxq/foxess-ha/issues/28#issuecomment-1030594210, or unsubscribe https://github.com/notifications/unsubscribe-auth/AARQGKH2HX6EAKFDQYJQKBDUZTZYZANCNFSM5MA2ZQNA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

-- Toby Jackson warmfusion.co.uk

macxq commented 2 years ago

I'm have the integration running since yesterday, with not released yet version from main branch that

My traffic is routed over external server (in case of block) using sshuttle

sshuttle -r *****@*** --verbose --python python3.6 8.209.116.72:443

williamjeccles commented 2 years ago

@macxq also happy to test this out if you have a subbranch or version I can pull

macxq commented 2 years ago

@williamjeccles v0.15-beta released, to be able to download it in HASC you need to select beta checkbox

Captain0Kangaroo commented 2 years ago

@williamjeccles v0.15-beta released, to be able to download it in HASC you need to select beta checkbox

Trying it now... I'll see how long it takes them to boot me off :-)

macxq commented 2 years ago

@Captain0Kangaroo 🤞 I'm still online ✌️

williamjeccles commented 2 years ago

@macxq testing 0.15 now!

benek984 commented 2 years ago

@macxq also testing 0.15 as some data are better than no data ;)

williamjeccles commented 2 years ago

I've tested 1 hr data update rates and 30min update rates over the last day without being blocked. So at least I can get some data. 5-minute resolution would be perfect because I can run some automations off that kinda data, like if I'm exporting and the battery is full then turn on the water heater, or charge the car.

Captain0Kangaroo commented 2 years ago

I've got mine set to 5mins and currently been running about 12 hours and no ban. 🤞

williamjeccles commented 2 years ago

@Captain0Kangaroo we'll see how things go once the other 20 or so plugin users reenable again. Hopefully all the optimizations are going to mean a lot less load for FoxESS

macxq commented 2 years ago

Foxess is down, probably some maintenance window, wondering what they prepared for us🧨

nigelspowage commented 2 years ago

I can't load foxesscloud.com, it appears a library the page requires is failing to load from the CDN they use. https://unpkg.com/vue-i18n@9.1.9/dist/vue-i18n.js 404 not found. I was talking with their support about something else so I've let them know (if they didn't already).

williamjeccles commented 2 years ago

@nigelspowage how do you get hold of them? They never get back to my calls or reply to my emails? I also noticed their broken webgui this evening.

williamjeccles commented 2 years ago

Foxess is down, probably some maintenance window, wondering what they prepared for us🧨

@macxq I'm living one day at a time right now waiting for their surprise anti-consumer update.

macxq commented 2 years ago

Is it just me being blocked?

This time symptoms are different, Inverter is continue to send the data but web and mobile app is not loading when i'm using my wifi. Routing traffic through different server make the service available for me.

btw. I did disabled the integration two days ago so they had to take some old ip list of bandits 🏴‍☠️

BigManDave commented 2 years ago

I'm blocked too as of around lunchtime. My integration was still active though. Happy to see they've not blocked the inverters this time at least.

Captain0Kangaroo commented 2 years ago

Their Cloud went off line for a while this morning and now i'm blocked.

All works in the cloud and data is showing but nothing in HA.

Seem their on it!

spmaid commented 2 years ago

I've only had my FoxESS inverter system for a week and already regretting it, their systems have been down I think 3 or 4 times already, how can they be this bad!

spmaid commented 2 years ago

I've only had my FoxESS inverter system for a week and already regretting it, their systems have been down I think 3 or 4 times already, how can they be this bad!

Oh, and why they wont allow access to the MODBUS data/interface I do not know, they discuss it in the user manual so why wont they allow use :-(

Daibutt commented 2 years ago

@spmaid Totally agree. Even if they just issued the register addresses and said accessing via modbus is at your own risk & if you brick your inverter tough luck, at least it would give us some choice.

dajmirycyny commented 2 years ago

Didn’t know there is modbus interface. Just found polish product which does utilize that, so in fact, if we could simply read modbus then finding out what numbers under modbus addresses mean shouldn’t be an issue. Although, this still requires some additional hardware then source: https://pvmonitor.pl/pvmt_instrukcja.php

Daibutt commented 2 years ago

@dajmirycyny Yes if you connect a cable to the ethernet port and poll normal modbus port 502 you get a response, but I so far haven't been able to read any registers. No idea of master/slave ID but have tried 0/1/2/3 and tried to read holding & input registers but with no success, just "0x02 - ILLEGAL DATA ADDRESS"

dajmirycyny commented 2 years ago

Interesting, thanks for the tip, will try to sniff. Although the thing I found above was using rs485 modbus

spmaid commented 2 years ago

I have an RS485 adapter connected too, and it 'sees' signals but I can't yet get anything out of it, and as Daibutt says, I am 'connected' to the LAN IP on port 502 with a number of different bits of software, but I don't see anything, just get errors. Seems wrong to me they can be like this.

williamjeccles commented 2 years ago

@macxq just had this from the product guy at Foxess, you appear to be in contact with him.

Good morning William, I have this message passed from Craig in our service team.

I am trying to get an understanding of the community of customers using alternative applications to report data on their FoxESS systems. I am not sure if you are using FoxESS Cloud or portal, or an alternative.

Are you able to give me more information please?

Somebody else reached out to me today via LinkedIn – they seem to be a developer of Home Assistant, so we will engage with them directly for co-operation.

Is this what we are talking about that affects you?

https://github.com/macxq/foxess-ha - a plugin that connects https://www.foxesscloud.com/ and Home Assistant(https://home-assistant.io/).

Kind regards

Justin

macxq commented 2 years ago

@williamjeccles yes, it's me who contacted with Justin over the LinkedIn. I'm looking positively into cooperation with them ;)

Captain0Kangaroo commented 2 years ago

@williamjeccles yes, it's me who contacted with Justin over the LinkedIn.

I'm looking positively into cooperation with them ;) Sounds positive 🙏

spmaid commented 2 years ago

After having an awful morning trying to get in to FoxESS Cloud, I now see a the below banner:

FOXESS is improving the portal stability from 14th Feb. to 8th Mar. 2022, during this period, users may not be able to log in timely, which will not influence the products performance or data uploading. FOXESS team apologizes for the inconvenience!