Sensor Community Sensors changes from available to unavailable every few minutes

LuckyTriple7 commented 2 years ago

The problem

Every minutes, state of an Sensor Community Sensor switch tu unavailable, after a few minutes it comes back., then unavailable and so on ...

What version of Home Assistant Core has the issue?

core-2022.2.0b1

What was the last working version of Home Assistant Core?

core-2021.12.10

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Sensor Community

Link to integration documentation on our website

https://rc.home-assistant.io/integrations/luftdaten/

Example YAML snippet

No response

Anything in the logs that might be useful for us?

Logger: homeassistant.components.luftdaten
Source: components/luftdaten/__init__.py:43
Integration: Sensor.Community (documentation, issues)
First occurred: 09:55:39 (4 occurrences)
Last logged: 09:57:02

Unexpected error fetching luftdaten_16510 data:
Unexpected error fetching luftdaten_8001 data:
Unexpected error fetching luftdaten_61277 data:
Unexpected error fetching luftdaten_2544 data:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/anyio/streams/tls.py", line 108, in _call_sslobject_method
    result = func(*args)
  File "/usr/local/lib/python3.9/ssl.py", line 888, in read
    v = self._sslobj.read(len)
ssl.SSLWantReadError: The operation did not complete (read) (_ssl.c:2633)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 31, in read
    return await self._stream.receive(max_bytes=max_bytes)
  File "/usr/local/lib/python3.9/site-packages/anyio/streams/tls.py", line 171, in receive
    data = await self._call_sslobject_method(self._ssl_object.read, max_bytes)
  File "/usr/local/lib/python3.9/site-packages/anyio/streams/tls.py", line 115, in _call_sslobject_method
    data = await self.transport_stream.receive()
  File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1105, in receive
    await self._protocol.read_event.wait()
  File "/usr/local/lib/python3.9/asyncio/locks.py", line 226, in wait
    await fut
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions
    yield
  File "/usr/local/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 33, in read
    return b""
  File "/usr/local/lib/python3.9/site-packages/anyio/_core/_tasks.py", line 103, in __exit__
    raise TimeoutError
TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 308, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 244, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 228, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/usr/local/lib/python3.9/site-packages/httpcore/_async/http11.py", line 102, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.9/site-packages/httpcore/_async/http11.py", line 81, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
  File "/usr/local/lib/python3.9/site-packages/httpcore/_async/http11.py", line 143, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
  File "/usr/local/lib/python3.9/site-packages/httpcore/_async/http11.py", line 172, in _receive_event
    data = await self._network_stream.read(
  File "/usr/local/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 33, in read
    return b""
  File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
    raise to_exc(exc)
httpcore.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 187, in _async_refresh
    self.data = await self._async_update_data()
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 147, in _async_update_data
    return await self.update_method()
  File "/usr/src/homeassistant/homeassistant/components/luftdaten/__init__.py", line 43, in async_update
    await sensor_community.get_data()
  File "/usr/local/lib/python3.9/site-packages/luftdaten/__init__.py", line 29, in get_data
    response = await client.get(str(url))
  File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1736, in get
    return await self.request(
  File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1513, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1600, in send
    response = await self._send_handling_auth(
  File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1628, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1665, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1702, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 308, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadTimeout

Additional information

No response

probot-home-assistant[bot] commented 2 years ago

luftdaten documentation luftdaten source _{^{(message by IssueLinks)}}

probot-home-assistant[bot] commented 2 years ago

Hey there @fabaff, @frenck, mind taking a look at this issue as it has been labeled with an integration (luftdaten) you are listed as a code owner for? Thanks! _{^{(message by CodeOwnersMention)}}

LuckyTriple7 commented 2 years ago

New Diagnostic Data attached .. diag.zip .

hermanops commented 2 years ago

Logger: homeassistant.components.luftdaten Source: components/luftdaten/init.py:43 Integration: Sensor.Community (documentation, issues) First occurred: 18:31:57 (5 occurrences) Last logged: 19:12:17

Unexpected error fetching luftdaten_53519 data: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/anyio/streams/tls.py", line 108, in _call_sslobject_method result = func(*args) File "/usr/local/lib/python3.9/ssl.py", line 888, in read v = self._sslobj.read(len) ssl.SSLWantReadError: The operation did not complete (read) (_ssl.c:2633)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 31, in read return await self._stream.receive(max_bytes=max_bytes) File "/usr/local/lib/python3.9/site-packages/anyio/streams/tls.py", line 171, in receive data = await self._call_sslobject_method(self._ssl_object.read, max_bytes) File "/usr/local/lib/python3.9/site-packages/anyio/streams/tls.py", line 115, in _call_sslobject_method data = await self.transport_stream.receive() File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1105, in receive await self._protocol.read_event.wait() File "/usr/local/lib/python3.9/asyncio/locks.py", line 226, in wait await fut asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions yield File "/usr/local/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 33, in read return b"" File "/usr/local/lib/python3.9/site-packages/anyio/_core/_tasks.py", line 103, in exit raise TimeoutError TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions yield File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 308, in handle_async_request resp = await self._pool.handle_async_request(req) File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 244, in handle_async_request raise exc File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 228, in handle_async_request response = await connection.handle_async_request(request) File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request return await self._connection.handle_async_request(request) File "/usr/local/lib/python3.9/site-packages/httpcore/_async/http11.py", line 102, in handle_async_request raise exc File "/usr/local/lib/python3.9/site-packages/httpcore/_async/http11.py", line 81, in handle_async_request ) = await self._receive_response_headers(**kwargs) File "/usr/local/lib/python3.9/site-packages/httpcore/_async/http11.py", line 143, in _receive_response_headers event = await self._receive_event(timeout=timeout) File "/usr/local/lib/python3.9/site-packages/httpcore/_async/http11.py", line 172, in _receive_event data = await self._network_stream.read( File "/usr/local/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 33, in read return b"" File "/usr/local/lib/python3.9/contextlib.py", line 137, in exit self.gen.throw(typ, value, traceback) File "/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions raise to_exc(exc) httpcore.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 187, in _async_refresh self.data = await self._async_update_data() File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 147, in _async_update_data return await self.update_method() File "/usr/src/homeassistant/homeassistant/components/luftdaten/init.py", line 43, in async_update await sensor_community.get_data() File "/usr/local/lib/python3.9/site-packages/luftdaten/init.py", line 29, in get_data response = await client.get(str(url)) File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1736, in get return await self.request( File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1513, in request return await self.send(request, auth=auth, follow_redirects=follow_redirects) File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1600, in send response = await self._send_handling_auth( File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1628, in _send_handling_auth response = await self._send_handling_redirects( File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1665, in _send_handling_redirects response = await self._send_single_request(request) File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1702, in _send_single_request response = await transport.handle_async_request(request) File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 308, in handle_async_request resp = await self._pool.handle_async_request(req) File "/usr/local/lib/python3.9/contextlib.py", line 137, in exit self.gen.throw(typ, value, traceback) File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions raise mapped_exc(message) from exc httpx.ReadTimeout

kimosak commented 2 years ago

Same issue, diagnostic data attached Config_entry_liftdaten.zip

kimosak commented 2 years ago

IMG_20220308_151558 The issue is still not resolved, sensors are mostly not available

LuckyTriple7 commented 2 years ago

If the API is unreachable or overloaded, the last value should be preserved for a period of time.

hajo62 commented 2 years ago

I see the same issue with sensor 61096 on v2022.3.3.

jaimevisser commented 2 years ago

Probably similar?


File "/usr/local/lib/python3.9/site-packages/luftdaten/__init__.py", line 29, in get_data
File "/usr/src/homeassistant/homeassistant/components/luftdaten/config_flow.py", line 50, in async_step_user
File "/usr/local/lib/python3.9/site-packages/luftdaten/__init__.py", line 29, in get_data
2022-03-13 20:34:43 ERROR (MainThread) [homeassistant.components.luftdaten] Unexpected error fetching luftdaten_28730 data:
File "/usr/src/homeassistant/homeassistant/components/luftdaten/__init__.py", line 43, in async_update
File "/usr/local/lib/python3.9/site-packages/luftdaten/__init__.py", line 29, in get_data
2022-03-13 20:34:43 WARNING (MainThread) [homeassistant.config_entries] Config entry '28730' for luftdaten integration not ready yet; Retrying in background
2022-03-13 20:34:53 ERROR (MainThread) [homeassistant.components.luftdaten] Unexpected error fetching luftdaten_28730 data:
File "/usr/src/homeassistant/homeassistant/components/luftdaten/__init__.py", line 43, in async_update
File "/usr/local/lib/python3.9/site-packages/luftdaten/__init__.py", line 29, in get_data
2022-03-13 20:35:08 ERROR (MainThread) [homeassistant.components.luftdaten] Unexpected error fetching luftdaten_28730 data:
File "/usr/src/homeassistant/homeassistant/components/luftdaten/__init__.py", line 43, in async_update
File "/usr/local/lib/python3.9/site-packages/luftdaten/__init__.py", line 29, in get_data
2022-03-13 20:35:33 ERROR (MainThread) [homeassistant.components.luftdaten] Unexpected error fetching luftdaten_28730 data:
File "/usr/src/homeassistant/homeassistant/components/luftdaten/__init__.py", line 43, in async_update
File "/usr/local/lib/python3.9/site-packages/luftdaten/__init__.py", line 29, in get_data
2022-03-13 20:35:48 ERROR (MainThread) [homeassistant.components.luftdaten] Unexpected error fetching luftdaten_66317 data:
File "/usr/src/homeassistant/homeassistant/components/luftdaten/__init__.py", line 43, in async_update
File "/usr/local/lib/python3.9/site-packages/luftdaten/__init__.py", line 29, in get_data
2022-03-13 20:35:48 WARNING (MainThread) [homeassistant.config_entries] Config entry '66317' for luftdaten integration not ready yet; Retrying in background
2022-03-13 20:35:58 ERROR (MainThread) [homeassistant.components.luftdaten] Unexpected error fetching luftdaten_66317 data:
File "/usr/src/homeassistant/homeassistant/components/luftdaten/__init__.py", line 43, in async_update
File "/usr/local/lib/python3.9/site-packages/luftdaten/__init__.py", line 29, in get_data```

sethiele commented 2 years ago

Same issue with core-2022.3.4

hermanops commented 2 years ago

2022.3.5 also: ssl.SSLWantReadError: The operation did not complete (read) (_ssl.c:1129), asyncio.exceptions.CancelledError, httpcore.ConnectTimeout and httpx.ConnectTimeout

kimosak commented 2 years ago

Same with core-2022.4.2

kimosak commented 2 years ago

Same issue with core-2022.4.6

Akszyn commented 2 years ago

mee to on 2022.4.6b0

Logger: homeassistant.components.luftdaten Source: components/luftdaten/init.py:43 Integration: Sensor.Community (documentation, issues) First occurred: 22 kwietnia 2022, 21:45:26 (52 occurrences) Last logged: 21:33:09

Unexpected error fetching luftdaten_58990 data: Traceback (most recent call last): File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/site-packages/anyio/streams/tls.py", line 108, in _call_sslobject_method result = func(*args) File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/ssl.py", line 888, in read v = self._sslobj.read(len) ssl.SSLWantReadError: The operation did not complete (read) (_ssl.c:2633)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 31, in read return await self._stream.receive(max_bytes=max_bytes) File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/site-packages/anyio/streams/tls.py", line 171, in receive data = await self._call_sslobject_method(self._ssl_object.read, max_bytes) File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/site-packages/anyio/streams/tls.py", line 115, in _call_sslobject_method data = await self.transport_stream.receive() File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1105, in receive await self._protocol.read_event.wait() File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/asyncio/locks.py", line 226, in wait await fut asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions yield File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 33, in read return b"" File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/site-packages/anyio/_core/_tasks.py", line 103, in exit raise TimeoutError TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions yield File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request resp = await self._pool.handle_async_request(req) File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request raise exc File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request response = await connection.handle_async_request(request) File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request return await self._connection.handle_async_request(request) File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/site-packages/httpcore/_async/http11.py", line 102, in handle_async_request raise exc File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/site-packages/httpcore/_async/http11.py", line 81, in handle_async_request ) = await self._receive_response_headers(**kwargs) File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/site-packages/httpcore/_async/http11.py", line 143, in _receive_response_headers event = await self._receive_event(timeout=timeout) File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/site-packages/httpcore/_async/http11.py", line 172, in _receive_event data = await self._network_stream.read( File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 33, in read return b"" File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/contextlib.py", line 137, in exit self.gen.throw(typ, value, traceback) File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions raise to_exc(exc) httpcore.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/site-packages/homeassistant/helpers/update_coordinator.py", line 190, in _async_refresh self.data = await self._async_update_data() File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/site-packages/homeassistant/helpers/update_coordinator.py", line 150, in _async_update_data return await self.update_method() File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/site-packages/homeassistant/components/luftdaten/init.py", line 43, in async_update await sensor_community.get_data() File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/site-packages/luftdaten/init.py", line 29, in get_data response = await client.get(str(url)) File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/site-packages/httpx/_client.py", line 1729, in get return await self.request( File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/site-packages/httpx/_client.py", line 1506, in request return await self.send(request, auth=auth, follow_redirects=follow_redirects) File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/site-packages/httpx/_client.py", line 1593, in send response = await self._send_handling_auth( File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/site-packages/httpx/_client.py", line 1621, in _send_handling_auth response = await self._send_handling_redirects( File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/site-packages/httpx/_client.py", line 1658, in _send_handling_redirects response = await self._send_single_request(request) File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/site-packages/httpx/_client.py", line 1695, in _send_single_request response = await transport.handle_async_request(request) File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request resp = await self._pool.handle_async_request(req) File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/contextlib.py", line 137, in exit self.gen.throw(typ, value, traceback) File "/data/data/pl.sviete.dom/files/usr/lib/python3.9/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions raise mapped_exc(message) from exc httpx.ReadTimeout

github-actions[bot] commented 2 years ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

RuudvanMunster commented 2 years ago

This issue is still very topical. The problem occurs regularly for me as well. Please do not close this issue yet.

RuudvanMunster commented 2 years ago

Is there any news with respect to this issue? I can imagine this is a complex problem. As far as I understand it until now, there are issues at the following levels:

The air quality sensors sometimes loose their connections with the local wifi.
The sensor community website that stores the data has regular access (timing?) problems.
According to the HA logbook file: a lot of noise in the logbook is generated from this integration. I think that a failing connection generates more errors than might be necessary. I also think some timings with respect to connection with the sensor community website might be set to a value too critical?

Because of the complexity of the issue, I created a very simple watchdog to supervise my two sensor systems. It checks every 5 minutes whether the sensor has valid values or presents an error status. If a sensor is not available for a longer period, it switches the sensor off and on to force a very hard reset. However I see now many errors that are not from the sensor or its data itself, but from problems to create a communication session.

So I think we have a serious problem in the code of the integration.

s-hutter commented 2 years ago

a lot of noise in the logbook is generated from this integration

At least this part of the issue can be prevented by configuring the logger component appropriately. Add this to your configuration.yaml:

logger:
  default: warning
  logs:
    homeassistant.components.luftdaten: fatal

RuudvanMunster commented 2 years ago

a lot of noise in the logbook is generated from this integration

At least this part of the issue can be prevented by configuring the logger component appropriately. Add this to your configuration.yaml:
logger:
  default: warning
  logs:
    homeassistant.components.luftdaten: fatal

Thanks, this indeed helps a lot!

tuxick commented 2 years ago

Is there any news with respect to this issue? I can imagine this is a complex problem. As far as I understand it until now, there are issues at the following levels:
* The air quality sensors sometimes loose their connections with the local wifi.

* The sensor community website that stores the data has regular access (timing?) problems.

* According to the HA logbook file: a lot of noise in the logbook is generated from this integration. I think that a failing connection generates more errors than might be necessary. I also think some timings with respect to connection with the sensor community website might be set to a value too critical?
Because of the complexity of the issue, I created a very simple watchdog to supervise my two sensor systems. It checks every 5 minutes whether the sensor has valid values or presents an error status. If a sensor is not available for a longer period, it switches the sensor off and on to force a very hard reset. However I see now many errors that are not from the sensor or its data itself, but from problems to create a communication session.

So I think we have a serious problem in the code of the integration.

tuxick commented 2 years ago

Looks like the problem is timeouts on the Luftdaten API, i see 3 possible solutions:

ignore and return last value
increase timeout
add a retry

RuudvanMunster commented 2 years ago

With respect to this issue: * The air quality sensors sometimes loose their connections with the local wifi.

I can report some news: The Air Quality sensors are very sensitive in their cooperation with the wifi accespoints. I have a ASUS GT-AX11000, together with two RT-AX92U units, as a mesh network.

If the sensor is too close to the GT-AX11000, with a wifi quality of 100%, the connection is unstable.
Sometimes the sensor connects to one of the RT-AX92U units, and with a wifi quality of 60% or less, it has a more or less stable connection.
After a disconnect, the sensor first tries to connect to the GT-AX11000, even if it is assigned to connect to the RT-AX92U.

After these observations I activated my FritzBox wifi network (currently non-mesh), a few weeks ago. Since then, the sensor has a very stable connection with the local network and a good connection with the sensor community network.

However, no matter how stable the connection is locally, the connection of HA with the sensor community server is a disaster. It even looks worse than several weeks ago.

RuudvanMunster commented 2 years ago

Looks like the problem is timeouts on the Luftdaten API, i see 3 possible solutions:

ignore and return last value

increase timeout

add a retry

May be we should consider another approach. I understand there is an option to import the sensor data directly into a local program. That should also be more in line with the HA philosophy: processing locally as much as possible.

If this is a bridge too far for the moment, I thinkincreasing timeout or adding one or more retries might help. Somehow I think we should not opt for ignore and return last value

In addition a code review might be a good option. I do not understand the API good enough, but from the logging I get the impression that there is more happening than just a time-out.

RuudvanMunster commented 2 years ago

Looking at the data of my sensors, both at the Sensor Community graph and at the Sensor Community Sensor Map, I see a delay of about 1 minute in the data presented. I was wondering, is the missing data a result of a timeout or the result of a delay. Maybe a somewhat different approach could be using data buffering. (The data should be time stamped in that case).

RuudvanMunster commented 2 years ago

There is really a problem, see both views of the same sensor. (HA resp. Sensor Community) With this number of errors ignore and use last value will not work.

SNAG-0112 SNAG-0113

tuxick commented 2 years ago

Looks like the problem is timeouts on the Luftdaten API, i see 3 possible solutions:

ignore and return last value

increase timeout

add a retry

May be we should consider another approach. I understand there is an option to import the sensor data directly into a local program. That should also be more in line with the HA philosophy: processing locally as much as possible.

If this is a bridge too far for the moment, I thinkincreasing timeout or adding one or more retries might help. Somehow I think we should not opt for ignore and return last value

In addition a code review might be a good option. I do not understand the API good enough, but from the logging I get the impression that there is more happening than just a time-out.

In init.py I set response = await client.get(str(url),timeout=12) which results in for example DEBUG (MainThread) [homeassistant.components.luftdaten] Finished fetching luftdaten_56872 data in 10.818 seconds (success: True)

I'm afraid the problem is on luftdaten/sensor.community side.

Skorfulose commented 2 years ago

May be we should consider another approach. I understand there is an option to import the sensor data directly into a local program. That should also be more in line with the HA philosophy: processing locally as much as possible.

I understand your reasoning and the generic preference of local API. But in this integration‘s case, the cloud polling (also) makes sense. Think about the people who don’t run their own sensors, but want to ingest data from someone in their neighborhood. Like me 😉 After all, that‘s why sensor owners share the data with the Luftdaten network, right?

tuxick commented 2 years ago

Like me wink After all, that‘s why sensor owners share the data with the Luftdaten network, right? I've been wondering about a patch doing both, but there's not so much room on an ESP.

RuudvanMunster commented 2 years ago

I distinguish between "research" and "operational use". I mainly share the data via the sensor network to share research opportunities with others. The Citizens Science Network idea. You want information that is as complete as possible and can be shared with others. The real-time aspect is not particularly critical. For me, the use of the data within Home Assistant has mainly operational significance. I want to be warned if the air quality outside deteriorates. So I want to have the data real-time. By the shortest way. The question is therefore what is the purpose of use within HA. I personally like an approach where both are possible. I do have some doubts about how well the Sensor Community server can withstand a larger number of HA users who want to retrieve (almost) real-time data.

RuudvanMunster commented 2 years ago

Like me wink After all, that‘s why sensor owners share the data with the Luftdaten network, right? I've been wondering about a patch doing both, but there's not so much room on an ESP.

I am rather optimistic about that: In the sensor's config menu there is room for using your own api's: I wish I was already that skilled to create an api myself. :)

RuudvanMunster commented 2 years ago

I found this discussion about real-time local use of the data: https://forum.sensor.community/t/using-a-custom-server-api-for-real-time-data/1532. Looks interesting as a starting point.

maartenhendrix commented 1 year ago

https://community.home-assistant.io/t/luftdaten-info-local-api-sensors/46647

Could also be interesting

kulmegil commented 1 year ago

Using Restful integration instead with boosted timeout (120s) and reasonable scan interval (≥5 min, I used 10min) gives me much more stable results.
Consider using "bbox" api endpoint to grab data from multiple nearby sensors (and sensor types) at once in a single request https://data.sensor.community/airrohr/v1/filter/box=min_latitude,min_longitude,max_latitude,max_longitude

Results, not perfect but huge improvement is visible: (while pulling data from more sensors at the same time) luftdaten_vs_restful

RuudvanMunster commented 1 year ago

https://community.home-assistant.io/t/luftdaten-info-local-api-sensors/46647

Could also be interesting

It really is! A very easy solution. Datamodel of the JSON file depends on sensor type. A good way to find out how the JSON file is organised is to apply the following command in CMD: curl http://192.168.178.20/data.json

- platform: command_line
  name: "Luchtkwaliteit Opzij PM10 (SDS)"
  command: 'curl http://192.168.178.20/data.json'
  value_template: "{{ value_json.sensordatavalues[0].value | round(1) }}"
  unit_of_measurement: "µg/m³"
- platform: command_line
  name: "Luchtkwaliteit Opzij PM2.5 (SDS)"
  command: 'curl http://192.168.178.20/data.json'
  value_template: "{{ value_json.sensordatavalues[1].value | round(1) }}"
  unit_of_measurement: "µg/m³"

Cloud PM10 Local PM10

RuudvanMunster commented 1 year ago

However a new problem came up. Spikes in de measurment. Only the PM2.5. signal of the sensor and the temperature, pressure and humidity sensor show this problem. PM2.5 a spike of a fixed value. The others vary. Spikes in local temperature values

Spikes in local data

vantop77 commented 1 year ago

The native integration is still not performing well. My observations are based on version 2023.2.2.

RuudvanMunster commented 1 year ago

The native integration is still not performing well. My observations are based on version 2023.2.2.

To my opinion the best way is to create local sensors, based upon the local data, in addition to sending the data by the sensors themself to sensor community. I found a solution to the problem I stated before. I was inspired by the restfull solution suggested by kulmegil (https://github.com/kulmegil) by replacing the code:

- platform: command_line
  name: "Luchtkwaliteit Achter PM1 (SPS30_P0)"
  command: 'curl http://192.168.178.23/data.json'
  value_template: "{{ value_json.sensordatavalues[0].value | round(1) }}"
  unit_of_measurement: "µg/m³"

by:

- platform: rest
  name: JSON airquality
  unique_id: airquality_opzij
  json_attributes:
    - software_version
    - age
    - sensordatavalues
  resource: http://192.168.178.20/data.json
  value_template: "{{ value_json.software_version }}"

- platform: template
  sensors:
    airquality_opzij_software_version:
      unique_id: airquality_opzij_software_version
      friendly_name: "Airquality Opzij Software version"
      value_template: "{{ state_attr('sensor.airquality', 'software_version') }}"
    airquality_opzij_age:
      unique_id: airquality_opzij_data_age
      friendly_name: "Airquality Opzij Data Age"
      value_template: "{{ state_attr('sensor.airquality', 'age') }}"

- platform: template
  sensors:
    air_quality_opzij_pm10_new:
      unique_id: airquality_opzij_PM10_REST
      friendly_name: "Air quality opzij PM10 (REST)"
      unit_of_measurement: "µg/m³"
      value_template: >-
        {% if state_attr('sensor.airquality', 'sensordatavalues')[0].value_type == 'SDS_P1' %}
          {{ state_attr('sensor.airquality', 'sensordatavalues')[0].value  | round(1) }}
        {% else %} 0 
        {% endif %}

- platform: template
  sensors:
    air_quality_opzij_pm25_new:
      unique_id: airquality_opzij_PM25_REST
      friendly_name: "Air quality opzij PM2.5 (REST)"
      unit_of_measurement: "µg/m³"
      value_template: >-
        {% if state_attr('sensor.airquality', 'sensordatavalues')[1].value_type == 'SDS_P2' %}
          {{ state_attr('sensor.airquality', 'sensordatavalues')[1].value  | round(1) }}
        {% endif %}

This solution checks whether the sensors are at the right position in the JSON file. It compares the value_type of the sensor data against the expected value_type. The problem I described above was because some of the data are simply missing in the datafile if there is a sensor error (e.g. if the measurment failed) (I am still searching for the cause of that)

The new template ignores the data if it is at the wrong position in the JSON file. You see two variations on this theme:

The first sensor template adds zero's for the missing data. Which is convenient for testing purposes.
The second sensor simply ignores the wrong data. Which is beter for operational use, I think.

There must be a better way to code this, using a list construction which links the data_value to the right sensor based upon the value_type. I am still a novice in this kind of programming, in a steep learning curve. So suggestions here are very welcome.

RuudvanMunster commented 1 year ago

However a new problem came up. Spikes in de measurment. Only the PM2.5. signal of the sensor and the temperature, pressure and humidity sensor show this problem. PM2.5 a spike of a fixed value. The others vary.

I think I found out what happened. Sometimes datapoints in the JSON file are missing. Mostly the airquality values themselves. The solution I used assigns the data values to the sensors based upon the position in the data file. Therefor the datavalues are sometimes assigned to the wrong sensor. Today I published an alternative to solve this problem.

RuudvanMunster commented 1 year ago

https://community.home-assistant.io/t/luftdaten-info-local-api-sensors/46647 Could also be interesting

It really is! A very easy solution. Datamodel of the JSON file depends on sensor type. A good way to find out how the JSON file is organised is to apply the following command in CMD: curl http://192.168.178.20/data.json
- platform: command_line
  name: "Luchtkwaliteit Opzij PM10 (SDS)"
  command: 'curl http://192.168.178.20/data.json'
  value_template: "{{ value_json.sensordatavalues[0].value | round(1) }}"
  unit_of_measurement: "µg/m³"
- platform: command_line
  name: "Luchtkwaliteit Opzij PM2.5 (SDS)"
  command: 'curl http://192.168.178.20/data.json'
  value_template: "{{ value_json.sensordatavalues[1].value | round(1) }}"
  unit_of_measurement: "µg/m³"

I think I found out what happened. Sometimes datapoints in the JSON file are missing. Mostly the airquality values themselves. The solution I used assigns the data values to the sensors based upon the position in the data file. Therefor the datavalues are sometimes assigned to the wrong sensor. Today I published an alternative to solve this problem.

vantop77 commented 1 year ago

I think it is better to use the native solution, this is the idea behind any integration. We will wait for Home Assistant developers to look at the issue and fix it.

issue-triage-workflows[bot] commented 1 year ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

home-assistant / core