Error during setup of component wemportal

[dm82m]

currently not working and wemportal.com is reachable and responding.

Logger: homeassistant.setup
Source: helpers/update_coordinator.py:178
First occurred: 11:02:10 (1 occurrences)
Last logged: 11:02:10

Error during setup of component wemportal
Traceback (most recent call last):
  File "/config/custom_components/wemportal/coordinator.py", line 29, in _async_update_data
    return await self.hass.async_add_executor_job(self.api.fetch_data)
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 205, in _async_refresh
    self.data = await self._async_update_data()
  File "/config/custom_components/wemportal/coordinator.py", line 28, in _async_update_data
    async with async_timeout.timeout(DEFAULT_TIMEOUT):
  File "/usr/local/lib/python3.10/site-packages/async_timeout/__init__.py", line 129, in __aexit__
    self._do_exit(exc_type)
  File "/usr/local/lib/python3.10/site-packages/async_timeout/__init__.py", line 212, in _do_exit
    raise asyncio.TimeoutError
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/setup.py", line 235, in _async_setup_component
    result = await task
  File "/config/custom_components/wemportal/__init__.py", line 65, in async_setup
    await coordinator.async_config_entry_first_refresh()
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 178, in async_config_entry_first_refresh
    raise ex
homeassistant.exceptions.ConfigEntryNotReady

[erikkastelec]

I will raise the timeout of the component setup. When you reported the issue, I tried accessing the WEM app and it took quite a long time to connect. This would explain the timeout issue. I will also try to add setup retries for events when Wemportal isn't available or slow to respond.

[iridium2001]

I faced the same issue after upgrading to the latest version. Increasing the timeouts in Erik's python code to 5 and 10 seconds helped.

[erikkastelec]

@iridium2001 in which file did you make the changes? I can only find the 120 sec timeout in const.py

[iridium2001]

I modified the wemportalapi.py > Line 550 (expertpage timeout) to 5 seconds and Line 603 (scrape pages) to 5 seconds.

And the "timedelta" in line 56 to 10 seconds.

[erikkastelec]

I increased the timeout to 3 seconds for now. If the issue persists, let me know.

[PioSkot]

Two bugs after upgrading :

1. Logger: custom_components.wemportal Source: custom_components/wemportal/wemportalapi.py:71 Integration: Weishaupt WEM Portal (documentation, issues) First occurred: 11:07:43 (1 occurrences) Last logged: 11:07:43

There was a problem with getting data from WEM Portal. If this problem persists, open an issue at https://github.com/erikkastelec/hass-WEM-Portal/issues Traceback (most recent call last): File "/config/custom_components/wemportal/wemportalapi.py", line 71, in fetch_webscraping_data self.data = processor.run([wemportal_job])[0] IndexError: list index out of range

2. Logger: homeassistant.setup Source: helpers/update_coordinator.py:178 First occurred: 11:09:42 (1 occurrences) Last logged: 11:09:42

Error during setup of component wemportal Traceback (most recent call last): File "/config/custom_components/wemportal/coordinator.py", line 29, in _async_update_data return await self.hass.async_add_executor_job(self.api.fetch_data) asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 205, in _async_refresh self.data = await self._async_update_data() File "/config/custom_components/wemportal/coordinator.py", line 28, in _async_update_data async with async_timeout.timeout(DEFAULT_TIMEOUT): File "/usr/local/lib/python3.10/site-packages/async_timeout/init.py", line 129, in aexit self._do_exit(exc_type) File "/usr/local/lib/python3.10/site-packages/async_timeout/init.py", line 212, in _do_exit raise asyncio.TimeoutError asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/setup.py", line 235, in _async_setup_component result = await task File "/config/custom_components/wemportal/init.py", line 65, in async_setup await coordinator.async_config_entry_first_refresh() File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 178, in async_config_entry_first_refresh raise ex homeassistant.exceptions.ConfigEntryNotReady

[erikkastelec]

@PioSkot

1. What kind of Home Assistant installation do you have? (container, supervised, ...)
2. Did you do a complete restart of Home Assistant?
3. What was the version of this component before upgrading?

I am asking because the first error is caused by outdated version of Scrapy library, which is a requirement that changed in the previous version of this component (1.3.9) #37

[PioSkot]

1. I do not understand this question. I do not distinguish between types of installations. How to check?
2. Of course.
3. I do not know. It worked so I didn't focus on the numbers. I installed the latest today. How to check? How to get back to the omitted one?

[erikkastelec]

@PioSkot There are 4 installation methods listed in the following link. I hope this helps you determine yours. It would also be beneficial to know the version of your installation. You can view it in Home Assistant settings in about section.

[PioSkot]

Thank you. Installation is on Home Assistant Operating System, v. 2022.10.05

[erikkastelec]

I assume you restarted Home Assistant by going to Settings > System > Reboot (top right, menu)?

If this is so I suggest you try to do a system reboot, if possible. You can reboot the device that is hosting Home Assistant by going to Settings > System > Hardware. Then look under the three vertical dot menu in the top right.

[SKis274]

Hey there. I got the first bug (https://github.com/erikkastelec/hass-WEM-Portal/issues/36#issuecomment-1301891286) after updating to the newest version. I'm running Homeassistent as HAOS with version 2022.11.1 but the error exist also in version 2022.11.0.

Rebooting the complete host doesn't resolve the problem. Some sensors like Vorlauf, Anlagendruck, Warmwasser and Außentemperatur work fine.

[erikkastelec]

I am working on a solution. As this isn't an isolated case and does not relate to the original bug from this issue, let's move the discussion to #39 .

[erikkastelec]

I increased the timeout from 3 sec to 5 sec as suggested by @iridium2001 . @PioSkot Let me know if this helps. In any case this is a temporary fix and I will investigate further the underlying problem.

[dm82m]

Wemportal.com is currently very very slow, takes about 15 seconds for login and so on. Thus leads to timeout problems. I have the same issues here:

Logger: homeassistant.setup
Source: helpers/update_coordinator.py:178
First occurred: 20:03:22 (1 occurrences)
Last logged: 20:03:22

Error during setup of component wemportal
Traceback (most recent call last):
  File "/config/custom_components/wemportal/coordinator.py", line 29, in _async_update_data
    return await self.hass.async_add_executor_job(self.api.fetch_data)
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 205, in _async_refresh
    self.data = await self._async_update_data()
  File "/config/custom_components/wemportal/coordinator.py", line 28, in _async_update_data
    async with async_timeout.timeout(DEFAULT_TIMEOUT):
  File "/usr/local/lib/python3.10/site-packages/async_timeout/__init__.py", line 129, in __aexit__
    self._do_exit(exc_type)
  File "/usr/local/lib/python3.10/site-packages/async_timeout/__init__.py", line 212, in _do_exit
    raise asyncio.TimeoutError
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/setup.py", line 235, in _async_setup_component
    result = await task
  File "/config/custom_components/wemportal/__init__.py", line 65, in async_setup
    await coordinator.async_config_entry_first_refresh()
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 178, in async_config_entry_first_refresh
    raise ex
homeassistant.exceptions.ConfigEntryNotReady

I guess the only way is to dramatically increase the timeouts and retry on occuring errors or wait for the wemportal to be back more stable...

[PioSkot]

@erikkastelec, unfortunately not

[erikkastelec]

I released a new version with increased timeout from 120 sec to 360 sec, which should mitigate this. I am working on a better solution, but this should suffice for now.

[dm82m]

Works out for me. I already informed weishaupt about the problems with their service. I guess they will clean up the memory tomorrow and all will be good for 3 weeks 😅

[iridium2001]

Hope that they don't consider blocking us from fetching the data automatically. I was already wondering if we all should run the script with longer intervals to not stress the server too much. But normally they should be able to cope with one request per user every ~5 minutes (or even less).

[dm82m]

They only block if we try to fetch the data too frequently. Thus there is a limit in the component. This is at least the fact for the wemportal.com - I guess there is no blocking for the mobile API.

To be honest: it seems they made really bad software architecture decisions. I am using wemportal since 2019 and wrote about 50 mails with a contact person of Weishaupt due to the problems we all are having. As I said: after writing a mail they do some "magic cleanup" and it runs for a couple of weeks till the timing problems are back. This is very sad to be honest.

[erikkastelec]

They also block requests if you make too many API calls. It doesn't produce any errors, but you get old data and the values do not change. I think the limit is reset every week.

Probably some sort of memory leak nobody wants to spend time fixing. They should just write a script that does a cleanup/restart once a week.

[dm82m]

If the service is free I would accept that. But we pay for a service with a bad quality. Not what I expected back then when I bought a weishaupt heating.

[tennet]

dm82m

If the service is free I would accept that. But we pay for a service with a bad quality. Not what I expected back then when I bought a weishaupt heating.

we pay to have the app I remember you at least in Italy. @erikkastelec thanks for your work

[dm82m]

just to give you an update: I got answers of weishaupt telling me that they are blocking all requests that are too often polling the data. I asked them about their minimal acceptable polling time for wemportal and mobile api but got no answer. the portal is again unreachable for this component on my side since yesterday 22:20. will keep you posted if I get an reply.

[iridium2001]

Same for me. Lost connection at 22:20 and recovered at 9:40. Also their Android app didn't allow to log in.

[erikkastelec]

I implemented config flow (Ability to add integration inside Home Assistant GUI) and improved handling of timeouts/failed requests. This should solve this and will be released after testing is done.

If anyone is interested in testing:

• Put all the files from /custom_conponents/wemportal/ that are in dev branch into homeassistat_config/custom_components/wemportal folder on your server.
• Add new integration inside Home Assistant frontend (yaml support is deprecated)

[dilki001]

Since I did the upgrade to the last version I have this error message: Can somebody help me ? The web part is working when I put in the config file web but if I use API or both I have this . I was using the API since sometimes now without any problem.

I tried to increase the time out but it didn't help.

thanks

Error during setup of component wemportal Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 971, in json return complexjson.loads(self.text, **kwargs) File "/usr/local/lib/python3.10/site-packages/simplejson/init.py", line 525, in loads return _default_decoder.decode(s) File "/usr/local/lib/python3.10/site-packages/simplejson/decoder.py", line 372, in decode obj, end = self.raw_decode(s) File "/usr/local/lib/python3.10/site-packages/simplejson/decoder.py", line 402, in raw_decode return self.scan_once(s, idx=_w(s, idx).end()) simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 239, in _async_refresh self.data = await self._async_update_data() File "/config/custom_components/wemportal/coordinator.py", line 29, in _async_update_data return await self.hass.async_add_executor_job(self.api.fetch_data) File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/config/custom_components/wemportal/wemportalapi.py", line 60, in fetch_data self.fetch_api_data() File "/config/custom_components/wemportal/wemportalapi.py", line 94, in fetch_api_data self.get_data() File "/config/custom_components/wemportal/wemportalapi.py", line 265, in get_data ).json() File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 975, in json raise RequestsJSONDecodeError(e.msg, e.doc, e.pos) requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/setup.py", line 256, in _async_setup_component result = await task File "/config/custom_components/wemportal/init.py", line 65, in async_setup await coordinator.async_config_entry_first_refresh() File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 211, in async_config_entry_first_refresh raise ex homeassistant.exceptions.ConfigEntryNotReady: Expecting value: line 1 column 1 (char 0)

[erikkastelec]

@dilki001 I created a new issue for you as this is not related to the problems we discussed here. I will get back to you with the fix or let you know if I need any additional information.

[dilki001]

Thanks

[erikkastelec]

Closing as this is fixed now. If this is still an issue let me know.