error getting device state SHC1

metaiiica commented 1 year ago

System Health details

Device is a empty Testsystem (Linux Mint with HA 8.4 in VM installed), only Hacs, SSH, Samba and Fileeditor installed. Integration only Livisi unofficial.

Checklist

[X] I have enabled debug logging for my installation.
[X] I have filled out the issue template to the best of my ability.
[X] This issue only contains 1 issue (if you have multiple issues, open one issue for each issue).
[X] This issue is not a duplicate issue of currently previous issues..

Describe the issue

see Screenshot. Many Value are not loaded

https://ibb.co/xj16zXN

Reproduction steps

install Livisi unofficial 1.31
reboot HA

Debug logs

Dieser Fehler wurde von einer benutzerdefinierten Integration verursacht

Logger: custom_components.livisi
Source: custom_components/livisi/aiolivisi.py:122
Integration: Livisi Unofficial (documentation, issues)
First occurred: 01:04:16 (130 occurrences)
Last logged: 01:12:34

Error getting device state
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 980, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore[return-value]  # noqa
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1085, in create_connection
    raise exceptions[0]
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1069, in create_connection
    sock = await self._connect_sock(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 973, in _connect_sock
    await self.sock_connect(sock, address)
  File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 628, in sock_connect
    return await fut
           ^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 668, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('192.168.178.124', 8080)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/config/custom_components/livisi/aiolivisi.py", line 222, in async_get_device_state
    return await self.async_send_authorized_request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/livisi/aiolivisi.py", line 84, in async_send_authorized_request
    return await self.async_send_request(method, url, payload, self._auth_headers)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/livisi/aiolivisi.py", line 122, in async_send_request
    response = await self._async_send_request(method, url, payload, headers)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/client.py", line 536, in _request
    conn = await self._connector.connect(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 540, in connect
    proto = await self._create_connection(req, traces, timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 901, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 1209, in _create_direct_connection
    raise last_exc
  File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 1178, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 988, in _wrap_create_connection
    raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host 192.168.178.124:8080 ssl:False [Connect call failed ('192.168.178.124', 8080)]

Diagnostics dump

No response

planbnet commented 1 year ago

@DanielHabenicht Your fix seems to also fail on the second try here.., Does it work for you?

The only reason I can think of is that the SHC v1 can only process one request in parallel, so this commit might be responsible https://github.com/planbnet/livisi_unofficial/commit/d012b3a3ceeddf4cde4589880d495e348991b554

DanielHabenicht commented 1 year ago

~I have no errors in my log.~ Found the same error, it occurred around 20 times. I will check if the SHC is capable of answering parallel requests. But many devices show unknown values for the first few minutes after the restart of the SHC1.

For the Humidity and Temperature readings I also often get unknown values. I haven't looked into why yet, but it seemingly has to do with the double entities for the room and the thermostat itself.

planbnet commented 1 year ago

This really sounds like a problem with parallel requests. On my v2 SHC everything works correctly. When initializing the devices, many requests at once will be sent (because the devices are initialized asynchonously by hass). Probably in a v1 SHC with many devices some of those requests fail and the values only appear as soon as an update is sent via the websocket. As my integration creates so much more entities, the problem is much more likely to appear here but theoretically also possible on the official integration...

planbnet commented 1 year ago

Please check if v1.3.2 which limits the amount of concurrent connections fixes this for you

metaiiica commented 1 year ago

server disconnects still exists...

Logger: homeassistant.components.websocket_api.http.connection Source: custom_components/livisi/aiolivisi.py:122 Integration: Home Assistant WebSocket API (documentation, issues) First occurred: 19:25:54 (1 occurrences) Last logged: 19:25:54

[140065050316992] Server disconnected Traceback (most recent call last): File "/config/custom_components/livisi/aiolivisi.py", line 120, in async_send_request response = await self._async_send_request(method, url, payload, headers) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/livisi/aiolivisi.py", line 135, in _async_send_request async with self._web_session.request( File "/usr/local/lib/python3.11/site-packages/aiohttp/client.py", line 1141, in aenter self._resp = await self._coro ^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/aiohttp/client.py", line 560, in _request await resp.start(conn) File "/usr/local/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 899, in start message, payload = await protocol.read() # type: ignore[union-attr] ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/aiohttp/streams.py", line 616, in read await self._waiter aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 226, in handle_call_service await hass.services.async_call( File "/usr/src/homeassistant/homeassistant/core.py", line 1974, in async_call response_data = await coro ^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/core.py", line 2011, in _execute_service return await target(service_call) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 235, in handle_service return await service.entity_service_call( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 870, in entity_service_call response_data = await _handle_entity_call( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 942, in _handle_entity_call result = await task ^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/climate/init.py", line 617, in async_service_temperature_set await entity.async_set_temperature(**kwargs) File "/config/custom_components/livisi/climate.py", line 92, in async_set_temperature response = await self.aio_livisi.async_set_state( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/livisi/aiolivisi.py", line 240, in async_set_state return await self.async_send_authorized_request( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/livisi/aiolivisi.py", line 84, in async_send_authorized_request return await self.async_send_request(method, url, payload, self._auth_headers) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/livisi/aiolivisi.py", line 122, in async_send_request response = await self._async_send_request(method, url, payload, headers) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/aiohttp/client.py", line 560, in _request await resp.start(conn) File "/usr/local/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 899, in start message, payload = await protocol.read() # type: ignore[union-attr] ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/aiohttp/streams.py", line 616, in read await self._waiter aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected

planbnet commented 1 year ago

But now it happens when setting the thermostat temperature???? And this a disconnection, not a "connection refused". Is the rest working now? The v1 SHC seems to be really picky...

DanielHabenicht commented 1 year ago

@planbnet I tested parallel requests on the SHC 1 and this does not seem to be an issue. I was able to request 14 URLs in parallel with no issue:

More are possible but it gets flaky then:

Same picture for querying all capabilities at the same time:

So I think there might be something else at play.

Code used:

import requests
import asyncio
import time
import aiohttp

url = 'http://192.168.178.141:8080/auth/token'
myobj = {
    "username": "admin",
    "password": "nichtmeinpasswort",
    "grant_type": "password"
}

x = requests.post(url, json = myobj, headers={'Content-Type': 'application/json', 'Authorization': 'Basic irgendeinbasictoken'})

# print(x.text)

urls = []
# urls = ['http://192.168.178.141:8080/status','http://192.168.178.141:8080/device', 'http://192.168.178.141:8080/device/states', 'http://192.168.178.141:8080/capability/states','http://192.168.178.141:8080/capability','http://192.168.178.141:8080/message', 'http://192.168.178.141:8080/location']
# urls = urls + urls + urls
headers={'Authorization': f'Bearer {x.json()["access_token"]}'}

x = requests.get('http://192.168.178.141:8080/capability', headers={'Authorization': f'Bearer {x.json()["access_token"]}'})

caps = x.json()
for cap in caps:
    urls.append(f'http://192.168.178.141:8080/capability/{cap["id"]}/state')

responses = []

async def get(url, session):
    try:
        async with session.request('GET', url=url, headers=headers) as response:
            resp = await response.read()
            print("Successfully {} got url {} with resp of length {}.".format(response.status ,url, len(resp)))
            responses.append(resp)
    except Exception as e:
        print("Unable to get url {} due to {}.".format(url, e.__class__))

async def main(urls):
    async with aiohttp.ClientSession() as session:
        ret = await asyncio.gather(*[get(url, session) for url in urls])
    print("Finalized all. Return is a list of len {} outputs.".format(len(ret)))

start = time.time()
asyncio.run(main(urls))
end = time.time()

print("Took {} seconds to pull {}/{} websites.".format(end - start, len(responses), len(urls)))

DanielHabenicht commented 1 year ago

In Home Assistant I can't reproduce the specific error mentioned above while setting my thermostat (but still Server disconnected errors). In addition, I get this one:

Traceback (most recent call last):
  File "/config/custom_components/livisi/aiolivisi.py", line 222, in async_get_device_state
    return await self.async_send_authorized_request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/livisi/aiolivisi.py", line 84, in async_send_authorized_request
    return await self.async_send_request(method, url, payload, self._auth_headers)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/livisi/aiolivisi.py", line 124, in async_send_request
    if "errorcode" in response:
       ^^^^^^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'NoneType' is not iterable

metaiiica commented 1 year ago

here the Screenshot

Screenshot

DanielHabenicht commented 1 year ago

Are the thermostates of different types?

planbnet commented 1 year ago

@DanielHabenicht I don't see how you come to the conclusion that parallel requests are not the problem when you can even reliably reproduce them above 14. When you have a few devices, initializing them in parallel can easily create more requests. Probably just my rate limiting code does not work.

Please try the following in your test code and play with the values of limit and limit_per_host (which i currently do not set):

async def main(urls):
    async with aiohttp.ClientSession(connector=TCPConnector(
            limit=1, limit_per_host=1
        )) as session:

metaiiica commented 1 year ago

Are the thermostates of different types?

no, all the same WST2

metaiiica commented 1 year ago

in the Livisiforum i have asked if any give you a SHC1 if needet to reproduce the issue what i have... so it have better chance to fix all issue...

https://community.livisi.de/forum/thread/4900-warum-verliert-ha-den-connect-zum-shc-nach-einiger-zeit/?postID=42188#post42190

metaiiica commented 1 year ago

Please let us write in this forum, here are more user and we can write in german so it is better for all

planbnet commented 1 year ago

Sorry, I have no interest to use that forum. Github is fine for me and the project description states: "This project is in "works for me" state and I do not advise anyone to use it nor can or will I provide support on how to install it." I don't have the time and motivation to act as an unpaid Livisi supporter for frustrated users that will soon lose access to their devices.

I already own an old SHC v1, and if I find the time and feel bored I might setup it to debug this issue, but at the moment this is not a very high priority for me.

Besides that I'll stick to english here, because the home assistant team should be able to understand the issues and PRs in this repo in case the code will someday be merged back to the official integration.

planbnet commented 1 year ago

Closing this because after the 1.4.0 rewrite and the 1.4.1 fix for SHC v1 this still might be a problem, but would materialize differently. If you still experience problems with the thermostats, please open a new issue.

planbnet / livisi_unofficial