Keep session also for web scraping and just use the reload button to minimise requests needed

[ghenzler]

First of all: Great extension, thanks for maintaining this @erikkastelec !

Currently the documentation suggests to set the crawling interval at most to 15min: https://github.com/erikkastelec/hass-WEM-Portal#configuration

Now there are some values in the web portal, that I would ideally have scraped in 30sec intervals or at least every minute (especially the "Leistungsanforderung" or "Stellung Umschaltventil" value from the web portal).

With an interval like 15min, the graphs in home assistant have a lot of delay (e.g. I notice that the power consumption as reported by the Fronius inverter has gone up a lot because of hot water generation, but the this integration only shows it delayed... sometimes the hot water generation is not recorded at all because it "only took 10min" and it happened between two scans.

So maybe the above can be fairly easily achieved by keeping the web session of the scraper and only press the "reload button" here:

[erikkastelec]

The easiest way to accomplish this is probably to use generic module #50 . This approach would allow us to avoid scraping altogether and keep the same session.

Moving to generic module is a priority right now, but I can't promise an exact timeline.

[ghenzler]

So https://github.com/floek/wemportal/blob/main/wemportal/wem_portal_web.py is also using the web login, to me it looks like it currently only retrieves the statistics (mostly graphs)... I'm more interested in instant values (and have them often sampled) because once instant samples are taken over to HA, the graphs/reports can be generated a lot better in HA. But maybe "Leistungsanforderung" or "Stellung Umschaltventil" is also available via the regular API somehow, then it would be even better.

[erikkastelec]

I decided to implement this before migrating to generic module. Code is in dev branch and you can manually put it into homeassistat_config/custom_components/wemportal. I hope this helps with scraping frequency. Let me know if you encounter any problems.

[dm82m]

In const.py an import is missing:

from enum import Enum

[erikkastelec]

Fixed, thanks for letting me know.

[iridium2001]

Any feedback regarding the new scraping method? Is it 'safe' to update?

[ghenzler]

@erikkastelec Thanks for fix! And sorry for taking that long to upgrade, I had some local changes to use a random user agent that I wanted to save to a git branch before upgrading - I have the latest version currently running the static user agent from https://github.com/erikkastelec/hass-WEM-Portal/blob/3577f2995baed20d33e03529d51ee097e3bf51fd/custom_components/wemportal/wemportalapi.py#L570 and it seems to work fine though