search

home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
72.81k stars 30.51k forks source link

2022.7 haveibeenpwned integration stopped working #74787

Closed locorotondo-gitter closed 1 year ago

locorotondo-gitter commented 2 years ago

The problem

Since upgrading to 2022.7 (0,1,2) the HaveIBeenPwned integration stopped working. It no longer fetches breached/ non-breached statuses.

The log shows errors as indicated further down in the log-section.

Happy to provide more info/ test things out for you.

What version of Home Assistant Core has the issue?

2022.07.2

What was the last working version of Home Assistant Core?

2202.06.x

What type of installation are you running?

Home Assistant OS

Integration causing the issue

HaveIBeenPwned

Link to integration documentation on our website

https://www.home-assistant.io/integrations/haveibeenpwned/

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/anyio/streams/tls.py", line 130, in _call_sslobject_method
    result = func(*args)
  File "/usr/local/lib/python3.10/ssl.py", line 917, in read
    v = self._sslobj.read(len)
ssl.SSLWantReadError: The operation did not complete (read) (_ssl.c:2548)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/httpcore/backends/asyncio.py", line 33, in read
    return await self._stream.receive(max_bytes=max_bytes)
  File "/usr/local/lib/python3.10/site-packages/anyio/streams/tls.py", line 195, in receive
    data = await self._call_sslobject_method(self._ssl_object.read, max_bytes)
  File "/usr/local/lib/python3.10/site-packages/anyio/streams/tls.py", line 137, in _call_sslobject_method
    data = await self.transport_stream.receive()
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 1265, in receive
    await self._protocol.read_event.wait()
  File "/usr/local/lib/python3.10/asyncio/locks.py", line 214, in wait
    await fut
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions
    yield
  File "/usr/local/lib/python3.10/site-packages/httpcore/backends/asyncio.py", line 31, in read
    with anyio.fail_after(timeout):
  File "/usr/local/lib/python3.10/site-packages/anyio/_core/_tasks.py", line 118, in __exit__
    raise TimeoutError
TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 105, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 84, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 148, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 177, in _receive_event
    data = await self._network_stream.read(
  File "/usr/local/lib/python3.10/site-packages/httpcore/backends/asyncio.py", line 30, in read
    with map_exceptions(exc_map):
  File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
    raise to_exc(exc)
httpcore.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 205, in _async_refresh
    self.data = await self._async_update_data()
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 164, in _async_update_data
    return await self.update_method()
  File "/usr/src/homeassistant/homeassistant/components/luftdaten/__init__.py", line 43, in async_update
    await sensor_community.get_data()
  File "/usr/local/lib/python3.10/site-packages/luftdaten/__init__.py", line 29, in get_data
    response = await client.get(str(url))
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1751, in get
    return await self.request(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1614, in send
    response = await self._send_handling_auth(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1642, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1679, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1716, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 352, in handle_async_request
    with map_httpcore_exceptions():
  File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadTimeout
2022-07-09 11:12:23 ERROR (SyncWorker_0) [homeassistant.components.haveibeenpwned.sensor] Failed fetching data for [email adress being checked]

Additional information

No response

probot-home-assistant[bot] commented 2 years ago

haveibeenpwned documentation haveibeenpwned source (message by IssueLinks)

locorotondo-gitter commented 2 years ago

I noticed that the following integrations also stopped working:

As far as I can tell, all 3 integrations/ sensors make use of some form of http(s) to get their data? Might that be a pointer towards a root cause?

adwoodworth commented 2 years ago

FYI: I have found the issue persists in 2022.7.3.

locorotondo-gitter commented 2 years ago

FYI: I have found the issue persists in 2022.7.3.

Thanks. For now I have just rolled back to 2022.06.7 (I think 🤔) as too many things stopped working throwing these http-related (at least, what I can make of it) error messages. I am more than happy to test things out for the dev.

adwoodworth commented 2 years ago

FYI: I have found the issue persists in 2022.7.4 and the issue persists in 2022.7.5.

locorotondo-gitter commented 2 years ago

Is there a maintainer for this integration? Reason I am asking is that with other reported bugs around (core) integrations, the bot flags the listed maintainer,which does not happen here....

nickrout commented 2 years ago

The manifest.json has an empty codeowner, so no maintainer.

locorotondo-gitter commented 2 years ago

Maybe I'll raise tickets for the other ones which stopped working with similar errors. Or ask @frenck if he can shed some light on the ownership. 🤔

adwoodworth commented 2 years ago

FYI: I have found the issue persists in all versions between 2022.7.3 and 2022.7.7.

locorotondo-gitter commented 2 years ago

I popped a message to Franck and Paulus to see if they know who is maintaining the code so we can work on fixing it. Fingers crossed.

frenck commented 2 years ago

Everybody is maintaining the code, that is how open source works. Feel free to jump in and fix it 👍

locorotondo-gitter commented 2 years ago

I don't think that my knowledge of algol-60 and assembler for a z80 will be of much help. I am more than willing (and have done so frequently) to help with debugging and testing things out, but if the "solution" is for me to learn programming in python and then try to fix something, which as far as I can tell is triggered by the new python version (2 more integrations throw the same http errors), my choice is simple; not upgrade anymore and just accept that 2022.06.7 is the end of the home-assistant line for me. I'll close the ticket.

frenck commented 2 years ago

I'll close the ticket.

Why? Multiple people are experienced and participants in this issue. I'm not saying you should fix it. Someone can or may fix it. Closing the issue won't help with that...

adwoodworth commented 2 years ago

FYI: I've cloned the repo and am working on seeing if I can understand the issue any better.

locorotondo-gitter commented 2 years ago

@frenck "Feel free to jump in and fix it 👍". How should I read that any different than "fix it yourself", especially as I made it clear that I am more than willing to help squash it? @adwoodworth let me know if you would like me to test things out.

frenck commented 2 years ago

Re-opening the issue as it has not been resolved.

frenck commented 2 years ago

How should I read that any different than "fix it yourself", especially as I made it clear that I am more than willing to help squash it?

I'm not following? Did I say something wrong? This is an open-source project, that relies on open source contributions. Anyone can contribute.

adwoodworth commented 2 years ago

I have a working version; I'll clean up the code a bit and see if I can figure out the github stuff to get it into production.

adwoodworth commented 2 years ago

If you add the attached folder to /config/custom_components and restart, this custom version of the integration will run. You can test it from there.

I've created a pull request for the changes.

haveibeenpwned.zip /

locorotondo-gitter commented 2 years ago

Will do. What was the issue?

adwoodworth commented 2 years ago

Will do. What was the issue?

I am not entirely sure, other than requests object wasn't handling the request properly. The parameters were being appended to the URL as strings (?parameter=xxx) but the docs for the requests object wants parameters to be passed as a dictionary object. I did that and it appears to be working.

The only problem is the changes are rejected for inclusion in the main repository because the current integration is older than the requirement that HA integrations do not talk directly to the APIs, they should import a module from PyPi to talk with the API. Any updates to the integration need to conform to the new requirements, which means authoring a fix also required rewriting the integration to use the PyPi modules. I decided to take a stab at that,

PyPi already has a haveibeenpwned module, found here https://pypi.org/project/haveibeenpwned/, so I mocked up some code to use the PyPi module instead. It should have made the integration very simple, but the module in PyPi appears to be broken too, throwing an error when processing the JSON response from the API. See https://github.com/marinko-peso/haveibeenpwned/issues/4, which was created by me as I raised an issue for the errors I am seeing.

I could start messing with the PyPi module for HaveIbeenPwned, but I just don't have the time or the skills to start working my way down a dependency tree to sort out issues like this. If I get a notice that the code owner for the issue I raised has corrected it, I might return and try again, but for now I have a custom version that fixes the issue for me, so I might just have to decide that is close enough for now.

locorotondo-gitter commented 2 years ago

If you add the attached folder to /config/custom_components and restart, this custom version of the integration will run. You can test it from there.

I've created a pull request for the changes.

haveibeenpwned.zip /

Done and it is chucking through the list of email addresses again instead of hanging at the first one in the list. Works like a charm.

locorotondo-gitter commented 2 years ago

Any updates to the integration need to conform to the new requirements, which means authoring a fix also required rewriting the integration to use the PyPi modules. I decided to take a stab at that,

PyPi already has a haveibeenpwned module, found here https://pypi.org/project/haveibeenpwned/, so I mocked up some code to use the PyPi module instead. It should have made the integration very simple, but the module in PyPi appears to be broken too, throwing an error when processing the JSON response from the API. See marinko-peso/haveibeenpwned#4, which was created by me as I raised an issue for the errors I am seeing.

I could start messing with the PyPi module for HaveIbeenPwned, but I just don't have the time or the skills to start working my way down a dependency tree to sort out issues like this. If I get a notice that the code owner for the issue I raised has corrected it, I might return and try again, but for now I have a custom version that fixes the issue for me, so I might just have to decide that is close enough for now.

Many thanks from my side for the work done already. I used to do a lot of programming, but the most "modern" language I have programmed in was Turbo Pascal when that still was a thing ;-). I most definitely don't have the skills anymore at the moment to fix things. I can -sort of- follow the code 'as is' and figure out what is going on, but chasing things down in a dependency tree and fixing things in python is way above my pay grade.

adwoodworth commented 2 years ago

So I looked into it deeper and found the PyPi library for HaveIBeenPwned was outdated. It uses version 2 of the HaveIBeenPwned API, which has been deprecated with breaking changes. Version 3 of the API is a requirement. There is no way to ensure backwards compatibility with the library that uses V2 of the API because V3 requires the use of an API key. I just went through the effort of rewriting a new library that uses V3 of the API. It is tested and working.

When I get some time, I'll see if I can get this included in PyPi, either as a new module specific for V3 of the API or as an upgrade to the existing module. Since it has breaking changes for the existing module, it might be better to add it as a new module until the previous version can be deprecated (even though, in theory, it is already broken since the v2 API is no longer functional). I think it would make sense to have one library in PyPi for each version of the HaveIBeenPwned API that is active.

I expect there will be a bunch of documentation and whatnot that I'll have to prepared. Once that is complete, I can update the HA integration to use the new PyPi library that supports V3 of the HaveIBeenPwned API.

At least there is light at the end of the tunnel.

github-actions[bot] commented 2 years ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

cezmunsta commented 1 year ago

@adwoodworth this was the exception that I saw with the latest stable: