Use of HACS 'Tapo Controller' Integration will cause Login WebAPI (mein-senec.de) failed

[GregorEstebanCardosoSchiller]

Hallo, ich weiß noch nicht genau, was ich falsch mache, da das Issue offenbar bei keinem anderen besteht, aber die Logindaten habe ich inzwischen x mal überprüft und auch mehrfach händisch bei HA als auch bei senec direkt eingegeben. Bei Senec läufts, im HA aber nicht (siehe Screenshot) Ergo: Was mache ich falsch?

Meine Version ist 3.0.6 - die normale Integration läuft problemlos. Habe jetzt auch händisch den Code nochmal aktualisiert (von hier: https://github.com/marq24/ha-senec-v3/releases/tag/3.0.6) und HA neu gestartet. - Keine Änderung.

[marq24]

Hi,

Du kannst versuchen a) Deine HA version auf den aktuellsten Stand zu bringen (2023.11.1) [wenn das nicht sowieso schon der Fall ist] oder b) ggf. enthält Dein Passwort ein Zeichen, das von der Integration nicht verarbeitet werden kann [dann würde ich einmal versuchen bei mein-senec das PWD zu ändern und es dann nochmals zu versuchen]...

Ansonsten ist mit aktiviertem DebugLog die mögliche Fehlersuche immer einfacher...

Grüße Matthias

[GregorEstebanCardosoSchiller]

Hi, PW mit simpelsten Sonderzeichen am Ende geändert. + und - HA auf aktuellem Stand, Debug gibt nicht mehr aus als:

Logger: custom_components.senec.config_flow Source: custom_components/senec/config_flow.py:376

[marq24]

you might like to try the latest beta version - but my guess goes into the direction, that the cookie jar patching does not work in your specific setup/environment [no clue why this can be the case]...

Would you be interested in digging deeper into the possible root cause? But this requires some manual efforts - like reading directly the log output from the console (and not via the gui)...

[GregorEstebanCardosoSchiller]

Anything for the cause =) Tell me what to do.

HASS OS, Intel Nuc

[marq24]

Might be, that HASS OS could be already the source of the issue? [Might be that this use a different aiohttp lib?] - So for the start please update to the beta channel, enable the debug log output for the integration [I guess this has to happen in the configuration.yaml] and try to get access to the actual log file of HA [I use here a docker setup which makes it very easy for me to access the complete log output - including all stack traces (and that is what I need - a stack trace)]

[GregorEstebanCardosoSchiller]

Good morning, updated to beta (3.0.8-rc5) debug protocoll:

2023-11-11 07:59:50.334 DEBUG (MainThread) [custom_components.senec] Finished fetching senec data in 0.262 seconds (success: True) 2023-11-11 07:59:51.336 DEBUG (MainThread) [custom_components.senec] Finished fetching senec data in 0.265 seconds (success: True) 2023-11-11 07:59:52.332 DEBUG (MainThread) [custom_components.senec] Finished fetching senec data in 0.261 seconds (success: True) 2023-11-11 07:59:52.931 INFO (MainThread) [custom_components.senec.pysenec_ha] restarting MySenecWebPortal... for user: 'verratichnicht@gmx.de' with options: None 2023-11-11 07:59:52.931 INFO (MainThread) [custom_components.senec.pysenec_ha] * authenticate(self) **** 2023-11-11 07:59:53.368 DEBUG (MainThread) [custom_components.senec] Finished fetching senec data in 0.296 seconds (success: True) 2023-11-11 07:59:53.450 WARNING (MainThread) [custom_components.senec.config_flow] Could not connect to mein-senec.de with 'verratichnicht@gmx.de', check credentials 2023-11-11 07:59:53.450 ERROR (MainThread) [custom_components.senec.config_flow] Could not connect to mein-senec.de with User 'verratichnicht@gmx.de', check credentials 2023-11-11 07:59:54.369 DEBUG (MainThread) [custom_components.senec] Finished fetching senec data in 0.298 seconds (success: True) 2023-11-11 07:59:55.333 DEBUG (MainThread) [custom_components.senec] Finished fetching senec data in 0.261 seconds (success: True) 2023-11-11 07:59:56.330 DEBUG (MainThread) [custom_components.senec] Finished fetching senec data in 0.259 seconds (success: True) 2023-11-11 07:59:57.339 DEBUG (MainThread) [custom_components.senec] Finished fetching senec data in 0.268 seconds (success: True)

working on trace log...

Also the homeassistant log on debug does not log anything else. "HASS OS stack trace log" with google always leads to standard logger....still looking.

Anything I can trace via cli?

[marq24]

... ok looks like that all traces are catched... but since in the console output we read a DEBUG, you ave successfully managed to enable the debug log (and have access to it)...

I hope you have also access to the filesystem where the custom_components folder is located... In this folder you have the senec folder (which is this integration)... I am not sure if you have experience with python - but the most important thing is, that you have to keep the intents...

What I am asking you to do is to alter the integration code locally - so that we can finally see the exception trace...

please open/edit the following file custom_components/senec/pysenec_ha/__init__.py file and search for:

    async def authenticate(self, doUpdate: bool, throw401: bool):
        _LOGGER.info("***** authenticate(self) ********")
        self.checkCookieJarType()
        auth_payload = {
            "username": self._SENEC_USERNAME,
            "password": self._SENEC_PASSWORD
        }
        async with self.websession.post(self._SENEC_AUTH_URL, data=auth_payload, max_redirects=20) as res:
            try:
                res.raise_for_status()
                if res.status == 200:
                    # be gentle reading the complete response...
                    r_json = await res.text()
                    self._isAuthenticated = True
                    _LOGGER.info("Login successful")
                    if doUpdate:
                        await self.update()
                else:
                    _LOGGER.error("Login failed with Code " + str(res.status))
                    self.purgeSenecCookies()
            except ClientResponseError as exc:
                #_LOGGER.error(str(exc))
                if throw401:
                    raise exc
                else:
                    if exc.status == 401:
                        self.purgeSenecCookies()
                        self._isAuthenticated = False
                    else:
                        _LOGGER.error("Login exception with Code " + str(exc.status))
                        self.purgeSenecCookies()

and remove the # in front of the _LOGGER.error(str(exc)) - or better replace this line with:

_LOGGER.error(f"ClientResponseError while authenticate {exc}", exc)

so the complete method should look like:

    async def authenticate(self, doUpdate: bool, throw401: bool):
        _LOGGER.info("***** authenticate(self) ********")
        self.checkCookieJarType()
        auth_payload = {
            "username": self._SENEC_USERNAME,
            "password": self._SENEC_PASSWORD
        }
        async with self.websession.post(self._SENEC_AUTH_URL, data=auth_payload, max_redirects=20) as res:
            try:
                res.raise_for_status()
                if res.status == 200:
                    # be gentle reading the complete response...
                    r_json = await res.text()
                    self._isAuthenticated = True
                    _LOGGER.info("Login successful")
                    if doUpdate:
                        await self.update()
                else:
                    _LOGGER.error("Login failed with Code " + str(res.status))
                    self.purgeSenecCookies()
            except ClientResponseError as exc:
                _LOGGER.error(f"ClientResponseError while authenticate {exc}", exc)
                if throw401:
                    raise exc
                else:
                    if exc.status == 401:
                        self.purgeSenecCookies()
                        self._isAuthenticated = False
                    else:
                        _LOGGER.error("Login exception with Code " + str(exc.status))
                        self.purgeSenecCookies()

after you have altered the code, you need to restart your HA, and then please start with setting up the integration again - and check the console log output - TIA

[GregorEstebanCardosoSchiller]

Hi, you can throw anything at me ;-) (ich programmiere seit 20 Jahren)

Output:

2023-11-13 08:39:32.063 DEBUG (MainThread) [custom_components.senec] Finished fetching senec data in 0.261 seconds (success: True) 2023-11-13 08:39:32.639 INFO (MainThread) [custom_components.senec.pysenec_ha] restarting MySenecWebPortal... for user: 'lalaland@xxx.de' with options: None 2023-11-13 08:39:32.640 INFO (MainThread) [custom_components.senec.pysenec_ha] * authenticate(self) **** 2023-11-13 08:39:33.061 DEBUG (MainThread) [custom_components.senec] Finished fetching senec data in 0.262 seconds (success: True) 2023-11-13 08:39:33.157 WARNING (MainThread) [custom_components.senec.config_flow] Could not connect to mein-senec.de with 'lalaland@xxx.de', check credentials 2023-11-13 08:39:33.157 ERROR (MainThread) [custom_components.senec.config_flow] Could not connect to mein-senec.de with User 'lalaland@xxx.de', check credentials

same?

[marq24]

this is a bit strange - I would expect more output from 'pysenec_ha'

2023-11-13 08:39:32.640 INFO (MainThread) [custom_components.senec.pysenec_ha] ***** authenticate(self) ********

either one of this three lines should had generated some output in the log ?!

• _LOGGER.info("Login successful")
• _LOGGER.error("Login failed with Code " + str(res.status))
• _LOGGER.error(f"ClientResponseError while authenticate {exc}", exc)

Are you sure, that your modified code was executed? [you might need to restart HA (one again)?!]

[GregorEstebanCardosoSchiller]

Strange indeed, but the modified file is live. Tested with "* authenticate(selftest) ****" instead of (self), just to be sure.

2023-11-13 10:44:37.282 INFO (MainThread) [custom_components.senec.pysenec_ha] * authenticate(selftest) ****

still no further output from pysenec_ha.

WTF....I just tried to debug the file myself (inserted another logger within the try...("Inside try") and now the whole config did work? Waaaaaas zur Hölle...das ergibt doch keinen Sinn...

2023-11-13 10:48:56.780 INFO (MainThread) [custom_components.senec.pysenec_ha] Inside try 2023-11-13 10:48:56.999 INFO (MainThread) [custom_components.senec.pysenec_ha] * authenticate(selftest) **** 2023-11-13 10:48:59.407 INFO (MainThread) [custom_components.senec.pysenec_ha] Inside try 2023-11-13 10:48:59.408 INFO (MainThread) [custom_components.senec.pysenec_ha] Login successful 2023-11-13 10:48:59.408 INFO (MainThread) [custom_components.senec.pysenec_ha] * update(self) **** 2023-11-13 10:48:59.408 DEBUG (MainThread) [custom_components.senec.pysenec_ha] * update_now_kW_stats(self) **** 2023-11-13 10:49:01.330 INFO (MainThread) [custom_components.senec.pysenec_ha] * authenticate(selftest) **** 2023-11-13 10:49:03.597 INFO (MainThread) [custom_components.senec.pysenec_ha] Inside try 2023-11-13 10:49:03.820 INFO (MainThread) [custom_components.senec.pysenec_ha] * authenticate(selftest) **** 2023-11-13 10:49:04.300 INFO (MainThread) [custom_components.senec.pysenec_ha] Inside try 2023-11-13 10:49:04.521 INFO (MainThread) [custom_components.senec.pysenec_ha] * authenticate(selftest) **** 2023-11-13 10:49:05.972 INFO (MainThread) [custom_components.senec.pysenec_ha] Inside try 2023-11-13 10:49:06.185 INFO (MainThread) [custom_components.senec.pysenec_ha] * authenticate(selftest) **** 2023-11-13 10:49:07.760 INFO (MainThread) [custom_components.senec.pysenec_ha] Inside try 2023-11-13 10:49:07.981 INFO (MainThread) [custom_components.senec.pysenec_ha] * authenticate(selftest) **** 2023-11-13 10:49:09.451 INFO (MainThread) [custom_components.senec.pysenec_ha] Inside try 2023-11-13 10:49:09.678 INFO (MainThread) [custom_components.senec.pysenec_ha] * authenticate(selftest) **** 2023-11-13 10:49:10.157 INFO (MainThread) [custom_components.senec.pysenec_ha] Inside try 2023-11-13 10:49:10.383 INFO (MainThread) [custom_components.senec.pysenec_ha] * authenticate(selftest) **** 2023-11-13 10:49:11.798 INFO (MainThread) [custom_components.senec.pysenec_ha] Inside try 2023-11-13 10:49:12.065 INFO (MainThread) [custom_components.senec.pysenec_ha] * authenticate(selftest) **** 2023-11-13 10:49:12.538 INFO (MainThread) [custom_components.senec.pysenec_ha] Inside try 2023-11-13 10:49:12.546 DEBUG (MainThread) [custom_components.senec.pysenec_ha] * update_context(self) **** 2023-11-13 10:49:12.546 INFO (MainThread) [custom_components.senec.pysenec_ha] * authenticate(selftest) **** 2023-11-13 10:49:13.933 INFO (MainThread) [custom_components.senec.pysenec_ha] Inside try 2023-11-13 10:49:13.946 INFO (MainThread) [custom_components.senec.binary_sensor] No binary_sensors for WebPortal... 2023-11-13 10:49:13.946 INFO (MainThread) [custom_components.senec.switch] No switches for WebPortal...

The module now collects the data...No explanation why this is now suddenly working...

[marq24]

a true "software developer WTF moment" - I assume somebody @ senec side have made an adjustment (your login attempts had been monitored and they realized an issue on their end with your account)...

[GregorEstebanCardosoSchiller]

Zu früh gefreut. Die Statistiken haben sich nicht weiter geupdatet. Kriege nun folgende Nachricht aus dem Log:

Bad logger message: ClientResponseError while authenticate 401, message='', url=URL('https://mein-senec.de/auth/oauth2/authorize?response_type=code&client_id=login-portal&scope=user&state=jD26m_byP9x5KCsP3CwQpU8786ffDL58U3-JWdm-huQ%3D&redirect_uri=https://mein-senec.de/login/oauth2/code/login-portal') ((ClientResponseError(RequestInfo(url=URL('https://mein-senec.de/auth/oauth2/authorize?response_type=code&client_id=login-portal&scope=user&state=jD26m_byP9x5KCsP3CwQpU8786ffDL58U3-JWdm-huQ%3D&redirect_uri=https://mein-senec.de/login/oauth2/code/login-portal'), method='GET', headers=<CIMultiDictProxy('Host': 'mein-senec.de', 'User-Agent': 'HomeAssistant/2023.11.1 aiohttp/3.8.5 Python/3.11', 'Accept': '/', 'Accept-Encoding': 'gzip, deflate', 'Cookie': 'JSESSIONID=A5C0D7398E8001F38A332A2FF530532F')>, real_url=URL('https://mein-senec.de/auth/oauth2/authorize?response_type=code&client_id=login-portal&scope=user&state=jD26m_byP9x5KCsP3CwQpU8786ffDL58U3-JWdm-huQ%3D&redirect_uri=https://mein-senec.de/login/oauth2/code/login-portal')), (<ClientResponse(https://mein-senec.de/auth/login) [302 ]> <CIMultiDictProxy('Vary': 'Origin', 'Vary': 'Access-Control-Request-Method', 'Vary': 'Access-Control-Request-Headers', 'Set-Cookie': 'JSESSIONID=268B847AD3C11E46D8EAA498D28CEDE0; Path=/auth; HttpOnly', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'Cache-Control': 'no-cache, no-store, max-age=0, must-revalidate', 'Pragma': 'no-cache', 'Expires': '0', 'x-frame-options': 'DENY', 'Location': '/auth/', 'Content-Length': '0', 'Date': 'Mon, 13 Nov 2023 13:34:54 GMT', 'x-server': 'node2')> , <ClientResponse(https://mein-senec.de/auth/) [302 ]> <CIMultiDictProxy('Vary': 'Origin', 'Vary': 'Access-Control-Request-Method', 'Vary': 'Access-Control-Request-Headers', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'Cache-Control': 'no-cache, no-store, max-age=0, must-revalidate', 'Pragma': 'no-cache', 'Expires': '0', 'x-frame-options': 'DENY', 'Location': 'https://mein-senec.de', 'Content-Language': 'en', 'Content-Length': '0', 'Date': 'Mon, 13 Nov 2023 13:34:54 GMT', 'x-server': 'node2')> , <ClientResponse(https://mein-senec.de) [302 ]> <CIMultiDictProxy('Set-Cookie': 'JSESSIONID=A5C0D7398E8001F38A332A2FF530532F; Path=/; HttpOnly', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'Cache-Control': 'no-cache, no-store, max-age=0, must-revalidate', 'Pragma': 'no-cache', 'Expires': '0', 'x-frame-options': 'SAMEORIGIN', 'Location': 'http://mein-senec.de/oauth2/authorization/login-portal', 'Content-Length': '0', 'Date': 'Mon, 13 Nov 2023 13:34:55 GMT', 'x-server': 'node2')> , <ClientResponse(http://mein-senec.de/oauth2/authorization/login-portal) [301 Moved Permanently]> <CIMultiDictProxy('Content-Length': '0', 'Location': 'https://mein-senec.de/oauth2/authorization/login-portal')> , <ClientResponse(https://mein-senec.de/oauth2/authorization/login-portal) [302 ]> <CIMultiDictProxy('x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'Cache-Control': 'no-cache, no-store, max-age=0, must-revalidate', 'Pragma': 'no-cache', 'Expires': '0', 'x-frame-options': 'SAMEORIGIN', 'Location': 'https://mein-senec.de/auth/oauth2/authorize?response_type=code&client_id=login-portal&scope=user&state=jD26m_byP9x5KCsP3CwQpU8786ffDL58U3-JWdm-huQ%3D&redirect_uri=https://mein-senec.de/login/oauth2/code/login-portal', 'Content-Length': '0', 'Date': 'Mon, 13 Nov 2023 13:34:55 GMT', 'x-server': 'node2')> ), status=401, headers=<CIMultiDictProxy('Set-Cookie': 'JSESSIONID=CD4160A50074CAE62C2354FFB1EDFEC8; Path=/auth; HttpOnly', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'Cache-Control': 'no-cache, no-store, max-age=0, must-revalidate', 'Pragma': 'no-cache', 'Expires': '0', 'x-frame-options': 'DENY', 'Transfer-Encoding': 'chunked', 'Date': 'Mon, 13 Nov 2023 13:34:55 GMT', 'x-server': 'node2')>),))

[marq24]

um mal einwenig ORDNUNG in diese Matsche reinzubringen...

DER request geht auf die Bretter 'https://mein-senec.de/auth/oauth2/authorize':

((ClientResponseError(RequestInfo(
URL('https://mein-senec.de/auth/oauth2/authorize?response_type=code&client_id=login-portal&scope=user&state=jD26m_byP9x5KCsP3CwQpU8786ffDL58U3-JWdm-huQ%3D&redirect_uri=https://mein-senec.de/login/oauth2/code/login-portal'),
method='GET',
headers=<CIMultiDictProxy('Host': 'mein-senec.de', 'User-Agent': 'HomeAssistant/2023.11.1 aiohttp/3.8.5 Python/3.11', 'Accept': '/', 'Accept-Encoding': 'gzip, deflate',
'Cookie': 'JSESSIONID=A5C0D7398E8001F38A332A2FF530532F')>,
URL('https://mein-senec.de/auth/oauth2/authorize?response_type=code&client_id=login-portal&scope=user&state=jD26m_byP9x5KCsP3CwQpU8786ffDL58U3-JWdm-huQ%3D&redirect_uri=https://mein-senec.de/login/oauth2/code/login-portal')),

Es wird die JSESSIONID=A5C0D7398E8001F38A332A2FF530532F verwendet... wenn man sich aber den flow ansieht, dann ist das die session id von der base URL 'https://mein-senec.de'... für die URL 'https://mein-senec.de/__oauth2/__' wird aber eine andere JSESSIONID gesetzt (nämlich die CD4160A50074CAE62C2354FFB1EDFEC8)

--- FLOW from HERE ---

(<ClientResponse(https://mein-senec.de/auth/login) [302 ]>
<CIMultiDictProxy('Vary': 'Origin', 'Vary': 'Access-Control-Request-Method', 'Vary': 'Access-Control-Request-Headers',
'Set-Cookie': 'JSESSIONID=268B847AD3C11E46D8EAA498D28CEDE0; Path=/auth; HttpOnly', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'Cache-Control': 'no-cache, no-store, max-age=0, must-revalidate', 'Pragma': 'no-cache', 'Expires': '0', 'x-frame-options': 'DENY', 
'Location': '/auth/', 'Content-Length': '0', 'Date': 'Mon, 13 Nov 2023 13:34:54 GMT', 'x-server': 'node2')> ,

<ClientResponse(https://mein-senec.de/auth/) [302 ]>
<CIMultiDictProxy('Vary': 'Origin', 'Vary': 'Access-Control-Request-Method', 'Vary': 'Access-Control-Request-Headers', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'Cache-Control': 'no-cache, no-store, max-age=0, must-revalidate', 'Pragma': 'no-cache', 'Expires': '0', 'x-frame-options': 'DENY', 
'Location': 'https://mein-senec.de', 'Content-Language': 'en', 'Content-Length': '0', 'Date': 'Mon, 13 Nov 2023 13:34:54 GMT', 'x-server': 'node2')> ,

<ClientResponse(https://mein-senec.de) [302 ]>
<CIMultiDictProxy(
'Set-Cookie': 'JSESSIONID=A5C0D7398E8001F38A332A2FF530532F; Path=/; HttpOnly', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'Cache-Control': 'no-cache, no-store, max-age=0, must-revalidate', 'Pragma': 'no-cache', 'Expires': '0', 'x-frame-options': 'SAMEORIGIN', 
'Location': 'http://mein-senec.de/oauth2/authorization/login-portal', 'Content-Length': '0', 'Date': 'Mon, 13 Nov 2023 13:34:55 GMT', 'x-server': 'node2')> ,

<ClientResponse(http://mein-senec.de/oauth2/authorization/login-portal) [301 Moved Permanently]>
<CIMultiDictProxy('Content-Length': '0',
'Location': 'https://mein-senec.de/oauth2/authorization/login-portal')> ,

<ClientResponse(https://mein-senec.de/oauth2/authorization/login-portal) [302 ]>
<CIMultiDictProxy('x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'Cache-Control': 'no-cache, no-store, max-age=0, must-revalidate', 'Pragma': 'no-cache', 'Expires': '0', 'x-frame-options': 'SAMEORIGIN',
'Location': 'https://mein-senec.de/auth/oauth2/authorize?response_type=code&client_id=login-portal&scope=user&state=jD26m_byP9x5KCsP3CwQpU8786ffDL58U3-JWdm-huQ%3D&redirect_uri=https://mein-senec.de/login/oauth2/code/login-portal', 'Content-Length': '0', 'Date': 'Mon, 13 Nov 2023 13:34:55 GMT', 'x-server': 'node2')> ), status=401, headers=<CIMultiDictProxy(
'Set-Cookie': 'JSESSIONID=CD4160A50074CAE62C2354FFB1EDFEC8; Path=/auth; HttpOnly', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'Cache-Control': 'no-cache, no-store, max-age=0, must-revalidate', 'Pragma': 'no-cache', 'Expires': '0', 'x-frame-options': 'DENY', 'Transfer-Encoding': 'chunked', 'Date': 'Mon, 13 Nov 2023 13:34:55 GMT', 'x-server': 'node2')>),))

Das ist genau der BUG in der aiohttp lib, den ich fixen musste, dammit der Zugriff auf das mein-senec.de portal überhaupt mit aiohttp funktioniert - leider wird der fix erst ab der version 3.9 (meine ich zumindst) ausgerollt - die gibt es aber noch nicht - deswegen enthält meine Integration einen PATCH, der versucht ein alternatives CookieHandling für die aiohttp sessions zu forcen... Das ganze passiert im Code ab hier: https://github.com/marq24/ha-senec-v3/blob/849a897bcb80049f486c3e7d1cd1069e5ad19d3a/custom_components/senec/pysenec_ha/init.py#L1347

Dieser Austausch des CookieJar's scheint bei Deiner speziellen Konstellation meistens NICHT zu funktionieren (oder Du hast eine weitere Integration am start, die "meine" CookieJar Implementierung wieder auf die default Implementierung zurücksetzt...

Du kannst bei Dir lokal mal das python script laufen lassen (das habe ich verwendet um den bug in der aiohttp lib zu fixen)...

import aiohttp
import asyncio
import logging

_LOGGER = logging.getLogger(__name__)
logging.basicConfig(level=logging.DEBUG)

import custom_components.senec.pysenec_ha
import custom_components.senec.pysenec_ha.util

async def on_request_redirect(session, trace_config_ctx, params):
    #_LOGGER.info('--- on_request_redirect ---')
    _LOGGER.debug(f"Response from: {params.url} [{params.response.status}] > {params.response.headers['Location']}")
    if 'Set-Cookie' in params.response.headers:
        _LOGGER.debug(f"Set-Cookie: {params.response.headers['Set-Cookie']}")

async def on_request_headers_sent(session, trace_config_ctx, params):
    #_LOGGER.info('--- on_request_headers_sent ---')
    _LOGGER.debug(f"Requesting: {params.url}")
    if 'Cookie' in params.headers:
        _LOGGER.debug(f"using cookie: {params.headers['Cookie']}")

async def main():
    trace_config = aiohttp.TraceConfig()
    trace_config.on_request_redirect.append(on_request_redirect)
    trace_config.on_request_headers_sent.append(on_request_headers_sent)
    async with aiohttp.ClientSession(trace_configs=[trace_config]) as session:
        sen = custom_components.senec.pysenec_ha.MySenecWebPortal(user="[YOUR_USER_HERE]", pwd="[YOUR_PWD_HERE]", websession=session)
        await sen.authenticate(do_update=False, throw401=True)
        await sen.update_context()
        print('AnlagenNummer: '+ str(sen._master_plant_number))

        await sen.update()
        print(sen._is_authenticated)
        print(sen._energy_entities)
        print(sen._power_entities)
        print(sen._battery_entities)

asyncio.run(main())

[GregorEstebanCardosoSchiller]

Nach etwas Gefriemel hab ichs hinbekommen (ich habe versucht aus diesem Terminal etwas zu kopieren....selbst markieren hab ich nicht (zuverlässig) hinbekommen, daher nur Screens: -gibt allerdings nen Error:

[marq24]

sorry - that your reply slippend though my attention - that was a busy week... the error on the second screenshot is (IMHO) fine... that's cause the test called some additional stiff that is not present in your environment - the most important thing is, that the code is able to login (with the patched CookieJar) - so this is indicating, that in your HA environment something is "overwriting" the cookie handler (that will be set by "my" integration)... so we have a race condition here... (that's my interpretation)...

Not sure if you are able to disable other integrations one by one - till you find the other integration that seams to be in conflict (with the Senec)...

[marq24]

BUT the code line

if self._QUERY_SPARE_CAPACITY:

does not exist in the BETA releases - it should be

if hasattr(self, '_QUERY_SPARE_CAPACITY') and self._QUERY_SPARE_CAPACITY:

so I am asking myself, if you might simply just need to update your integration?! to 3.0.9-rc1

[GregorEstebanCardosoSchiller]

Just updated to new version, ran script again:

Also tried to integrate webAPI again, new error, just as in screen:

2023-11-28 11:14:12.841 INFO (MainThread) [custom_components.senec.pysenec_ha] CookieJar patch required cause aiohttp version is below 3.9.0 (current lib version is: 3.8.5)

aiohttp manually updated to 3.9.1 got some HA dependency issus:

ran script again:

Tried to add WebAPI:

2023-11-28 11:25:30.191 DEBUG (MainThread) [custom_components.senec] Finished fetching senec data in 0.256 seconds (success: True) 2023-11-28 11:25:30.884 INFO (MainThread) [custom_components.senec.pysenec_ha] restarting MySenecWebPortal... for user: '*' with options: None 2023-11-28 11:25:30.884 INFO (MainThread) [custom_components.senec.pysenec_ha] *** authenticate(self) **** 2023-11-28 11:25:31.193 DEBUG (MainThread) [custom_components.senec] Finished fetching senec data in 0.259 seconds (success: True) 2023-11-28 11:25:31.330 WARNING (MainThread) [custom_components.senec.config_flow] Could not connect to mein-senec.de with '', check credentials 2023-11-28 11:25:31.331 ERROR (MainThread) [custom_components.senec.config_flow] Could not connect to mein-senec.de with User '', check credentials 2023-11-28 11:25:32.231 DEBUG (MainThread) [custom_components.senec] Finished fetching senec data in 0.296 seconds (success: True) 2023-11-28 11:25:33.224 DEBUG (MainThread) [custom_components.senec] Finished fetching senec data in 0.290 seconds (success: True)

[marq24]

ok - first of all with the new version the script need to be adjusted slightly:

search for: await sen.authenticate(doUpdate=False, throw401=True)

and replace with: await sen.authenticate(do_update=False, throw401=True)

and I can not recommend to update aiohttp lib manually in your ha-installation - I just saw in the ha code, that when HA is running with python 3.12 that then aiohttp 3.9 is used

[GregorEstebanCardosoSchiller]

Siehe hier:

[GregorEstebanCardosoSchiller]

update

[marq24]

sigh - another update of the test script (cause of my refactoring)...

print(sen._isAuthenticated) must be print(sen._is_authenticated)

but you don't have to run the script again - I can see from the log, that this code is working (as designed) - so "standalone" the MySenecPortal code is working fine... it's the connection with other integrations that "might" cause this issue... (at least that's my thesis)...

[GregorEstebanCardosoSchiller]

So it seems, that as you said, the code is working:

But I have no idea how i should check which integrations might interfere...shall I look for aiohttp?

Suggestions? ^^

[marq24]

just as "proof" - can you please update to 3.1.1, enable DEBUG Log for the senec integration and then (try) to reinstall the WebAPI? - TIA [I hope the messages are now better to understand]

[GregorEstebanCardosoSchiller]

Updated to 3.1.1, restarted:

2023-11-28 13:53:11.287 DEBUG (MainThread) [custom_components.senec] Finished fetching senec data in 0.269 seconds (success: True) 2023-11-28 13:53:12.110 INFO (MainThread) [custom_components.senec.pysenec_ha] restarting MySenecWebPortal... for user: 'xxx' with options: None 2023-11-28 13:53:12.110 INFO (MainThread) [custom_components.senec.pysenec_ha] * authenticate(self) **** 2023-11-28 13:53:12.246 DEBUG (MainThread) [custom_components.senec] Finished fetching senec data in 0.260 seconds (success: True) 2023-11-28 13:53:13.118 WARNING (MainThread) [custom_components.senec.config_flow] Could not connect to mein-senec.de with 'xxx', check credentials (exception) 2023-11-28 13:53:13.119 ERROR (MainThread) [custom_components.senec.config_flow] Could not connect to mein-senec.de with User 'xxx', check credentials 2023-11-28 13:53:13.295 DEBUG (MainThread) [custom_components.senec] Finished fetching senec data in 0.308 seconds (success: True) 2023-11-28 13:53:14.249 DEBUG (MainThread) [custom_components.senec] Finished fetching senec data in 0.263 seconds (success: True) 2023-11-28 13:53:15.275 DEBUG (MainThread) [custom_components.senec] Finished fetching senec data in 0.289 seconds (success: True)

[marq24]

I miss in the log output from the 'pysenec_ha'... after the restarting MySenecWebPortal... for user: 'xxx' with options: None - I "miss" some lines about the used aiohttp lib... have you now forced the 3.9.x version ?!

[GregorEstebanCardosoSchiller]

Strange, strange, I updated the aiohttp again, restarted HA (think I updated the HA Version after the fix), BUT in your log it still says 3.8.5 instead of 3.9.1 (but at least, the log has more output now):

2023-11-28 14:04:33.011 INFO (MainThread) [custom_components.senec.pysenec_ha] restarting MySenecWebPortal... for user: 'xxx' with options: None 2023-11-28 14:04:33.012 INFO (MainThread) [custom_components.senec.pysenec_ha] aiohttp version is below 3.9.0 (current version is: 3.8.5) - CookieJar need to be patched 2023-11-28 14:04:33.013 INFO (MainThread) [custom_components.senec.pysenec_ha] CookieJar patch applied... 2023-11-28 14:04:33.013 INFO (MainThread) [custom_components.senec.pysenec_ha] * authenticate(self) **** 2023-11-28 14:04:33.020 DEBUG (MainThread) [custom_components.senec] Finished fetching senec data in 0.266 seconds (success: True) 2023-11-28 14:04:33.422 WARNING (MainThread) [homeassistant.helpers.entity_component] Forced update failed. Entity sensor.myenergi_hub_11357557_power_charging not found. 2023-11-28 14:04:34.007 DEBUG (MainThread) [custom_components.senec] Finished fetching senec data in 0.260 seconds (success: True) 2023-11-28 14:04:34.425 WARNING (MainThread) [homeassistant.helpers.entity_component] Forced update failed. Entity sensor.myenergi_hub_11357557_power_charging not found. 2023-11-28 14:04:34.837 INFO (MainThread) [custom_components.senec.pysenec_ha] Login successful 2023-11-28 14:04:34.837 DEBUG (MainThread) [custom_components.senec.pysenec_ha] * update_context(self) **** 2023-11-28 14:04:34.837 DEBUG (MainThread) [custom_components.senec.pysenec_ha] * update_get_customer(self) **** 2023-11-28 14:04:35.002 WARNING (MainThread) [custom_components.senec.config_flow] Could not connect to mein-senec.de with 'xxx', check credentials (exception) 2023-11-28 14:04:35.002 ERROR (MainThread) [custom_components.senec.config_flow] Could not connect to mein-senec.de with User 'xxx', check credentials

[GregorEstebanCardosoSchiller]

But maybe the last sentence may result in different behaviour within HA...

Found this in the web: "I assume you use Home Assistant OS? You can’t install libraries. The OS doesn’t support it. Libraries come preinstalled with the integrations"

[marq24]

please do not try to manually update the aiohttp lib...

... with the additional output of your log it looks like, that the code can login at mein-senec.de - but then at a certain point the content of the CookieJar get wiped... and so the following requests to the portal are failing...

For me it's impossible to judge, which of the integrations you are running might be the root cause... I would deactivate one by one in order to see, when the code start to work... and I will provide you with a list of integrations I have here running myself (so they should be fine)...

I have also running:

• AVM Fritz!Box Tool
• DLNA Digital Media Renderer
• HACS
• Mobile App
• Shelly
• Reolink IP NVR/Camera
• Sonne
• Radio Browser
• Local Kalender
• Meteorologisk institutt (Met.no)
• Version

[GregorEstebanCardosoSchiller]

Interesting...I tried to deactivate the ones with cloud support...It is the tapo integration...Have 14 of the devices to for energymonitoring...fffffff...It´s also a hacs...

It´s from https://github.com/petretiandrea/home-assistant-tapo-p100 - grrrrr I´ll try to look into it.

[marq24]

This lines are the root cause of the issue...

https://github.com/search?q=owner%3Apetretiandrea%20cookie_jar&type=code

This code is purging all existing cookies of a ha web session (and HA share sessions across integrations) - so as soon as this tapo-integration is making a post the previous fetched cookies are purged from the JAR - causing the "login error"...

When calling clear without any additional predicate the aiohttp lib will purge all data from the cookie store... https://github.com/aio-libs/aiohttp/blob/213d1b22d42ce8efc54d7858b490e920fcdb4f0a/aiohttp/cookiejar.py#L114

So IMHO nobody should call this method when you are not in control of the websession... The developer should always use clear_domain(...)

https://github.com/petretiandrea/home-assistant-tapo-p100/issues/642

I close this issue here - since there is nothing that I can do - except hoping that plugp100 get's fixed

[marq24]

after I wrote 'there is nothing I could do'... I had an idea, that I COULD DO something... please update to https://github.com/marq24/ha-senec-v3/releases/tag/3.1.2 and let me know, if this solves the situation for you...

[GregorEstebanCardosoSchiller]

First great appreciation for your dedication and persistence ;-) A Spam for every second of the tapos triggering the cookie purge:

2023-11-29 13:05:42.580 WARNING (MainThread) [homeassistant.helpers.entity_component] Forced update failed. Entity sensor.myenergi_hub_11357557_power_charging not found. 2023-11-29 13:05:42.583 WARNING (MainThread) [custom_components.senec.pysenec_ha] CookieJar.clear() have been called without ANY predicates! This would PURGE all cookies (IMHO nobody should do that inside a home assistant integration) - We will keep at least the mein.senec.de cookies! but any other integration might break cause of this call! 2023-11-29 13:05:42.585 WARNING (MainThread) [custom_components.senec.pysenec_ha] CookieJar.clear() have been called without ANY predicates! This would PURGE all cookies (IMHO nobody should do that inside a home assistant integration) - We will keep at least the mein.senec.de cookies! but any other integration might break cause of this call! 2023-11-29 13:05:42.587 WARNING (MainThread) [custom_components.senec.pysenec_ha] CookieJar.clear() have been called without ANY predicates! This would PURGE all cookies (IMHO nobody should do that inside a home assistant integration) - We will keep at least the mein.senec.de cookies! but any other integration might break cause of this call! 2023-11-29 13:05:42.589 WARNING (MainThread) [custom_components.senec.pysenec_ha] CookieJar.clear() have been called without ANY predicates! This would PURGE all cookies (IMHO nobody should do that inside a home assistant integration) - We will keep at least the mein.senec.de cookies! but any other integration might break cause of this call! 2023-11-29 13:05:42.591 WARNING (MainThread) [custom_components.senec.pysenec_ha] CookieJar.clear() have been called without ANY predicates! This would PURGE all cookies (IMHO nobody should do that inside a home assistant integration) - We will keep at least the mein.senec.de cookies! but any other integration might break cause of this call! 2023-11-29 13:05:42.594 WARNING (MainThread) [custom_components.senec.pysenec_ha] CookieJar.clear() have been called without ANY predicates! This would PURGE all cookies (IMHO nobody should do that inside a home assistant integration) - We will keep at least the mein.senec.de cookies! but any other integration might break cause of this call! 2023-11-29 13:05:42.596 WARNING (MainThread) [custom_components.senec.pysenec_ha] CookieJar.clear() have been called without ANY predicates! This would PURGE all cookies (IMHO nobody should do that inside a home assistant integration) - We will keep at least the mein.senec.de cookies! but any other integration might break cause of this call! 2023-11-29 13:05:42.598 WARNING (MainThread) [custom_components.senec.pysenec_ha] CookieJar.clear() have been called without ANY predicates! This would PURGE all cookies (IMHO nobody should do that inside a home assistant integration) - We will keep at least the mein.senec.de cookies! but any other integration might break cause of this call! 2023-11-29 13:05:42.629 WARNING (MainThread) [custom_components.senec.pysenec_ha] CookieJar.clear() have been called without ANY predicates! This would PURGE all cookies (IMHO nobody should do that inside a home assistant integration) - We will keep at least the mein.senec.de cookies! but any other integration might break cause of this call! 2023-11-29 13:05:42.642 WARNING (MainThread) [custom_components.senec.pysenec_ha] CookieJar.clear() have been called without ANY predicates! This would PURGE all cookies (IMHO nobody should do that inside a home assistant integration) - We will keep at least the mein.senec.de cookies! but any other integration might break cause of this call! 2023-11-29 13:05:42.665 WARNING (MainThread) [custom_components.senec.pysenec_ha] CookieJar.clear() have been called without ANY predicates! This would PURGE all cookies (IMHO nobody should do that inside a home assistant integration) - We will keep at least the mein.senec.de cookies! but any other integration might break cause of this call! 2023-11-29 13:05:42.673 WARNING (MainThread) [custom_components.senec.pysenec_ha] CookieJar.clear() have been called without ANY predicates! This would PURGE all cookies (IMHO nobody should do that inside a home assistant integration) - We will keep at least the mein.senec.de cookies! but any other integration might break cause of this call! 2023-11-29 13:05:42.679 WARNING (MainThread) [custom_components.senec.pysenec_ha] CookieJar.clear() have been called without ANY predicates! This would PURGE all cookies (IMHO nobody should do that inside a home assistant integration) - We will keep at least the mein.senec.de cookies! but any other integration might break cause of this call! 2023-11-29 13:05:42.684 WARNING (MainThread) [custom_components.senec.pysenec_ha] CookieJar.clear() have been called without ANY predicates! This would PURGE all cookies (IMHO nobody should do that inside a home assistant integration) - We will keep at least the mein.senec.de cookies! but any other integration might break cause of this call! 2023-11-29 13:05:42.686 WARNING (MainThread) [custom_components.senec.pysenec_ha] CookieJar.clear() have been called without ANY predicates! This would PURGE all cookies (IMHO nobody should do that inside a home assistant integration) - We will keep at least the mein.senec.de cookies! but any other integration might break cause of this call! 2023-11-29 13:05:42.691 WARNING (MainThread) [custom_components.senec.pysenec_ha] CookieJar.clear() have been called without ANY predicates! This would PURGE all cookies (IMHO nobody should do that inside a home assistant integration) - We will keep at least the mein.senec.de cookies! but any other integration might break cause of this call! 2023-11-29 13:05:42.706 WARNING (MainThread) [custom_components.senec.pysenec_ha] CookieJar.clear() have been called without ANY predicates! This would PURGE all cookies (IMHO nobody should do that inside a home assistant integration) - We will keep at least the mein.senec.de cookies! but any other integration might break cause of this call! 2023-11-29 13:05:42.715 WARNING (MainThread) [custom_components.senec.pysenec_ha] CookieJar.clear() have been called without ANY predicates! This would PURGE all cookies (IMHO nobody should do that inside a home assistant integration) - We will keep at least the mein.senec.de cookies! but any other integration might break cause of this call! 2023-11-29 13:05:42.718 WARNING (MainThread) [custom_components.senec.pysenec_ha] CookieJar.clear() have been called without ANY predicates! This would PURGE all cookies (IMHO nobody should do that inside a home assistant integration) - We will keep at least the mein.senec.de cookies! but any other integration might break cause of this call! 2023-11-29 13:05:42.726 WARNING (MainThread) [custom_components.senec.pysenec_ha] CookieJar.clear() have been called without ANY predicates! This would PURGE all cookies (IMHO nobody should do that inside a home assistant integration) - We will keep at least the mein.senec.de cookies! but any other integration might break cause of this call! 2023-11-29 13:05:42.738 WARNING (MainThread) [custom_components.senec.pysenec_ha] CookieJar.clear() have been called without ANY predicates! This would PURGE all cookies (IMHO nobody should do that inside a home assistant integration) - We will keep at least the mein.senec.de cookies! but any other integration might break cause of this call! 2023-11-29 13:05:42.776 WARNING (MainThread) [custom_components.senec.pysenec_ha] CookieJar.clear() have been called without ANY predicates! This would PURGE all cookies (IMHO nobody should do that inside a home assistant integration) - We will keep at least the mein.senec.de cookies! but any other integration might break cause of this call! 2023-11-29 13:05:42.819 WARNING (MainThread) [custom_components.senec.pysenec_ha] CookieJar.clear() have been called without ANY predicates! This would PURGE all cookies (IMHO nobody should do that inside a home assistant integration) - We will keep at least the mein.senec.de cookies! but any other integration might break cause of this call! 2023-11-29 13:05:42.850 WARNING (MainThread) [custom_components.senec.pysenec_ha] CookieJar.clear() have been called without ANY predicates! This would PURGE all cookies (IMHO nobody should do that inside a home assistant integration) - We will keep at least the mein.senec.de cookies! but any other integration might break cause of this call!

But it works. I´ll comment the line in your code....

[marq24]

at least you do not have to wait for a fix from petretiandrea... and you can control the "spam-frequency" via your automation... ;-)

[marq24]

with the next update you will get the warning only every 24h...