problem with getting data from WEM Portal

[juergen88400]

2023-02-22 16:33:13.482 ERROR (SyncWorker_6) [custom_components.wemportal] There was a problem with getting data from WEM Portal. If this problem persists, open an issue at https://github.com/erikkastelec/hass-WEM-Portal/issues Traceback (most recent call last): File "/config/custom_components/wemportal/wemportalapi.py", line 71, in fetch_webscraping_data self.data = processor.run([wemportal_job])[0] IndexError: list index out of range 2023-02-22 16:33:17.803 WARNING (MainThread) [homeassistant.helpers.frame] Detected integration that called async_setup_platforms instead of awaiting async_forward_entry_setups; this will fail in version 2023.3. Please report issue to the custom integration author for hacs using this method at custom_components/hacs/init.py, line 171: hass.config_entries.async_setup_platforms( 2023-02-22 16:33:20.448 WARNING (MainThread) [homeassistant.setup] Setup of wemportal is taking over 10 seconds.

[juergen88400]

I am running Home Assistant 2023.2.5 Supervisor 2023.01.1 Operating System 9.5 Frontend 20230202.0 - latest

[erikkastelec]

Did this happen at startup and are entities still unavailable? If so, the only solution for now is to manually restart Home Assistant. I am working on a solution as this is a known issue. Let me know if this is not the case.

[juergen88400]

I realized this afternoon that data is not being updated. After realizing this I rebooted HA raspi and found that a portion of the entities are missing or not being updated. "Diese Entität wird nicht mehr von der Integration wemportal bereitgestellt. " Eg. outside temp is there, solar pump is missing.

[iridium2001]

Maybe unrelated, but I also have no updates from my WEM entities since yesterday, 19:00. No change in HA since then (I did not touch anything, also not restarted yet) and WEM App and Website seem to work normal.

[mathiaslehmann]

Hello, I have the same problem. It's funny that since yesterday shortly after 7 p.m. no more data has come via web scraping. The API values ​​seem to work.

[asdt1803]

Same issue here. Wemportal.de works normally when accessed with a browser. Tried with wget from the machine the script is executed on. Works only when a "valid" User-Agent is specified. Changing the User-Agent string in wemportalapi.py unfortunately did not help.

[erikkastelec]

Will look into this and let you know what I find. API (values from mobile app) work fine for me, but wemportal values stopped updating at 19:00 for me too.

[erikkastelec]

This is fixed in the new version. @asdt1803 It was indeed related to the User-Agent. Thanks @juergen88400 for reporting this and to all others for feedback.

[juergen88400]

wow - thanks for fixing that quick

[ghenzler]

@erikkastelec I ran into the same problem yesterday and added the user agent in the same way as 2687702efa, but that also seemed to not reliably work... I ended up with alternating user agents (but even now, it's not 100% working)... I keep monitoring it (but maybe other's have that problem as well)

[erikkastelec]

I checked my instance and everything seems to work. It is possible that they implemented some rate limiting. What is scan_interval (how often web scraping is done) set to? I have is set to default 30 min

[ghenzler]

@erikkastelec I'm currently running it every three minutes (and that might well be the reason)... see #53 why I would love to have the option to get the data recorded more often into HA

[erikkastelec]

I think the rate limit resets at midnight, but I'm not sure. I had problems when I set it to an interval lower than 15 minutes. This will have to do before I implement #53 .

I'm closing this as it's not related to the original issue