Bug Report: DNS Lookup for URL ghcr.io failed

sharbich commented 5 months ago

OS Version

Debian GNU/Linux 11 (bullseye)

System Information

Linux ha-su-deb 5.10.0-9-amd64 SMP Debian 5.10.70-1

What happened?

Can't install ghcr.io/home-assistant/amd64-hassio-audio:2023.12.0

Machine Type

generic-x86-64

Installer output

Hello, when installing the homeassistant-supervisior package, the following URL 'Get \"https://ghcr.io/v2/\": dial tcp: lookup ghcr.io: no such host"' cannot be accessed. I get the following message when I try to open this URL via a browser: "{"errors":[{"code":"UNAUTHORIZED","message":"authentication required"}]}"
What can I do?
Greetings from Stefan Harbich

Relevant log output

Jan 26 13:19:06 dsme01 hassio_supervisor[57904]: 24-01-26 13:19:06 INFO (MainThread) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/amd64-hassio-audio with tag 2023.12.0.
Jan 26 13:19:06 dsme01 dockerd[57904]: time="2024-01-26T13:19:06.220113483+01:00" level=warning msg="Error getting v2 registry: Get \"https://ghcr.io/v2/\": dial tcp: lookup ghcr.io: no such host"
Jan 26 13:19:06 dsme01 dockerd[57904]: time="2024-01-26T13:19:06.220173341+01:00" level=info msg="Attempting next endpoint for pull after error: Get \"https://ghcr.io/v2/\": dial tcp: lookup ghcr.io: no such host"
Jan 26 13:19:06 dsme01 dockerd[57904]: time="2024-01-26T13:19:06.252387306+01:00" level=error msg="Handler for POST /v1.44/images/create returned error: Get \"https://ghcr.io/v2/\": dial tcp: lookup ghcr.io: no such host"
Jan 26 13:19:06 dsme01 hassio_supervisor[57904]: 24-01-26 13:19:06 ERROR (MainThread) [supervisor.docker.interface] Can't install ghcr.io/home-assistant/amd64-hassio-audio:2023.12.0: 500 Server Error for http+docker://localhost/v1.44/images/create?tag=2023.12.0&fromImage=ghcr.io%2Fhome-assistant%2Famd64-hassio-audio&platform=linux%2Famd64: Internal Server Error ("Get "https://ghcr.io/v2/": dial tcp: lookup ghcr.io: no such host")
Jan 26 13:19:06 dsme01 hassio_supervisor[57904]: 24-01-26 13:19:06 WARNING (MainThread) [supervisor.plugins.audio] Error on installing Audio plugin, retrying in 30sec

ADR

[X] I have read through the ADR and have confirmed that my system is compliant with the requirements
[X] I understand that if my system is found to not be compliant, my issue will be closed immediately without further investigation

Code of Conduct

[X] I have read and agree to the Home Assistant Code of Conduct

donaldguy commented 5 months ago

This is reporting a DNS lookup failure for the github container registry

As such it is/was either:

a transient "real" DNS failure on Github's part (possible, given the Microsoft net kerfuffle of the other day; potentially also with your LAN or your ISP)
a DNS misconfiguration on your host level:
- try nslookup ghcr.io and/or dig ghcr.io and
- if those fail, also try e.g. dig @8.8.8.8 ghcr.io (looking the record up directly on Google public DNS rather than your configured default DNS servers) - if that one then works, you need to change your /etc/resolv.conf, or mess with NetworkManager setup, and/or fix what DNS servers your router is serving you)
A failure somehow isolated to the level of your dockerd (as a whole or somehow, I don't think this is particularly possible, in the context of being called from the supervisor container in particular.)
- I'd say if the first commands from (2) work fine but this persists,
- confirm with docker exec hassio_supervisor nslookup ghcr.io (this actually seems particularly apt to fail whereas my result suggests its using the hassio_dns container as resolver - but I'd be surprised if that applied to a pull for the audio image)
- and e.g. docker exec hassio_supervisor python -c 'import docker; print(docker.APIClient().pull("ghcr.io/home-assistant/aarch64-hassio-audio:2023.12.0"))'
- if your hassio_supervisor container isn't living long enough to exec into for these commands, you can do e.g. sh <(sed 's/docker container create/docker run --rm -it --entrypoint \/bin\/bash/' /usr/sbin/hassio-supervisor) to start a copy of the container set up the same-as-usual but running a shell instead of the normal supervisor init process (and in here you can skip the docker exec hassio_supervisor parts)

Like I said, I'd be rather surprised if its somehow option (3), but if it is try doing a full teardown (e.g. systemctl stop hassio-supervisor && docker rm -f $(docker ps -aq) && docker network prune and probs reboot your system) and see if that does anythning (such as break a circular dep on the hassio_dns container)

If there really is a docker-specific DNS issues you'll need to tweak some stuff / look at /etc/docker/daemon.json, and/or the ExecStart line of your docker.service (systemd cat docker.service), as there are --dns and --dns-opt options to dockerd that do exist (and may be at fault or offer workarounds)

past there (unlikely) we'd need to get into weeds of brctl, ip, maybe some iptables stuff

donaldguy commented 5 months ago

But also to be clear Debian 11 is outside the supported spec per the ADR (only Debian 12), so you might be about to get this issue axed

(The lack of title is also a bad look)

marcofranssen commented 4 months ago

Facing the same issue.

Logger: homeassistant.components.websocket_api.http.connection
Source: components/websocket_api/commands.py:240
Integration: Home Assistant WebSocket API ([documentation](https://www.home-assistant.io/integrations/websocket_api), [issues](https://github.com/home-assistant/core/issues?q=is%3Aissue+is%3Aopen+label%3A%22integration%3A+websocket_api%22))
First occurred: 13:18:56 (1 occurrences)
Last logged: 13:18:56

[547170694848] Error updating Home Assistant Supervisor: Update of Supervisor failed: Can't install ghcr.io/home-assistant/aarch64-hassio-supervisor:2024.02.0: 500 Server Error for http+docker://localhost/v1.44/images/create?tag=2024.02.0&fromImage=ghcr.io%2Fhome-assistant%2Faarch64-hassio-supervisor&platform=linux%2Farm64: Internal Server Error ("Get "https://ghcr.io/v2/": dial tcp: lookup ghcr.io: Temporary failure in name resolution")
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/hassio/update.py", line 258, in async_install
    await async_update_supervisor(self.hass)
  File "/usr/src/homeassistant/homeassistant/components/hassio/handler.py", line 55, in _wrapper
    raise HassioAPIError(data["message"])
homeassistant.components.hassio.handler.HassioAPIError: Update of Supervisor failed: Can't install ghcr.io/home-assistant/aarch64-hassio-supervisor:2024.02.0: 500 Server Error for http+docker://localhost/v1.44/images/create?tag=2024.02.0&fromImage=ghcr.io%2Fhome-assistant%2Faarch64-hassio-supervisor&platform=linux%2Farm64: Internal Server Error ("Get "https://ghcr.io/v2/": dial tcp: lookup ghcr.io: Temporary failure in name resolution")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 240, in handle_call_service
    response = await hass.services.async_call(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2279, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2316, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 892, in entity_service_call
    single_response = await _handle_entity_call(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 962, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/update/__init__.py", line 161, in async_install
    await entity.async_install_with_progress(version, backup)
  File "/usr/src/homeassistant/homeassistant/components/update/__init__.py", line 465, in async_install_with_progress
    await self.async_install(version, backup)
  File "/usr/src/homeassistant/homeassistant/components/hassio/update.py", line 260, in async_install
    raise HomeAssistantError(
homeassistant.exceptions.HomeAssistantError: Error updating Home Assistant Supervisor: Update of Supervisor failed: Can't install ghcr.io/home-assistant/aarch64-hassio-supervisor:2024.02.0: 500 Server Error for http+docker://localhost/v1.44/images/create?tag=2024.02.0&fromImage=ghcr.io%2Fhome-assistant%2Faarch64-hassio-supervisor&platform=linux%2Farm64: Internal Server Error ("Get "https://ghcr.io/v2/": dial tcp: lookup ghcr.io: Temporary failure in name resolution")

I tried dig at ghcr.io in both the host system as well a execed into the suporvisor container. both return the IP of ghcr.io so it doesn't seem to be a DNS issue as both are able to resolve.

Could it be python related?

github-actions[bot] commented 2 months ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

github-actions[bot] commented 5 days ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

home-assistant / supervised-installer