Cannot connect to host 172.30.32.1:8123 (Fails to boot sometimes)

[Fusseldieb]

Hi, I've been experiencing this bug since I first knew Home Assistant (and it's still there with 0.80.3), so I'm not so sure if this can be resolved in HassOS, but following does happen sometimes when I restart Home Assistant:

Sometimes when I restart Home Assistant, it doesn't come up again. The page says Connection Refused and that's it. The first time this happened to me I almost freaked out, but now I know that a simple router restart fixes it. It's strange, Today it happened to me again and I wanted some concrete proof of what is happening so that I can submit this here.

So, the SSH Server still runs normally during this time, so there it is where it errored out: (Log was collected while HA was attempting to restart)

core-ssh:~# hassio su logs
Error decoding json invalid character '\x1b' looking for beginning of value: 18-10-21 06:26:39 INFO (MainThread) [hassio.homeassistant] Updated Home Assistant API token
18-10-21 06:56:39 INFO (MainThread) [hassio.homeassistant] Updated Home Assistant API token
18-10-21 07:26:39 INFO (MainThread) [hassio.homeassistant] Updated Home Assistant API token
18-10-21 07:56:39 INFO (MainThread) [hassio.homeassistant] Updated Home Assistant API token
18-10-21 08:26:39 INFO (MainThread) [hassio.homeassistant] Updated Home Assistant API token
18-10-21 08:43:56 INFO (SyncWorker_9) [hassio.docker.interface] Stop homeassistant/armhf-addon-mosquitto Docker application
18-10-21 08:43:57 INFO (SyncWorker_9) [hassio.docker.interface] Clean homeassistant/armhf-addon-mosquitto Docker application
18-10-21 08:44:00 INFO (SyncWorker_19) [hassio.docker.addon] Start Docker add-on homeassistant/armhf-addon-mosquitto with version 2.0
18-10-21 08:44:34 INFO (MainThread) [hassio.api.proxy] /homeassistant/api/services access from core_configurator
18-10-21 08:44:34 INFO (MainThread) [hassio.api.proxy] /homeassistant/api/events access from core_configurator
18-10-21 08:44:34 INFO (MainThread) [hassio.api.proxy] /homeassistant/api/states access from core_configurator
18-10-21 08:56:39 INFO (MainThread) [hassio.homeassistant] Updated Home Assistant API token
18-10-21 09:29:43 INFO (MainThread) [hassio.api.proxy] /homeassistant/api/services/homeassistant/restart access from core_configurator
18-10-21 09:29:43 INFO (MainThread) [hassio.homeassistant] Updated Home Assistant API token
18-10-21 09:29:43 INFO (SyncWorker_15) [hassio.docker] Run command 'python3 -m homeassistant -c /config --script check_config' on homeassistant/raspberrypi3-homeassistant
18-10-21 09:30:30 INFO (MainThread) [hassio.homeassistant] Home Assistant config is valid
18-10-21 09:30:30 INFO (SyncWorker_6) [hassio.docker.interface] Stop homeassistant/raspberrypi3-homeassistant Docker application
18-10-21 09:31:02 INFO (SyncWorker_6) [hassio.docker.interface] Clean homeassistant/raspberrypi3-homeassistant Docker application
18-10-21 09:31:03 INFO (SyncWorker_8) [hassio.docker.homeassistant] Start homeassistant homeassistant/raspberrypi3-homeassistant with version 0.80.3
18-10-21 09:32:50 INFO (MainThread) [hassio.api.proxy] /homeassistant/api/services access from core_configurator
18-10-21 09:32:50 ERROR (MainThread) [hassio.homeassistant] Error on call http://172.30.32.1:8123/api/services: Cannot connect to host 172.30.32.1:8123 ssl:None [Connection refused]
18-10-21 09:32:50 ERROR (MainThread) [hassio.api.proxy] Error on API for request services
18-10-21 09:35:01 INFO (MainThread) [hassio.api.security] /homeassistant/logs access from core_ssh
18-10-21 09:35:13 INFO (MainThread) [hassio.api.security] /supervisor/logs access from core_ssh

Note: The HA log contains no error at all, just normal initialization things as usual. Note 2: A supervisor restart doesn't resolve it, I need to restart the whole router and then restart Home Assistant (again). Idea: Maybe this can be fixed by disabling and re-enabling the ethenet port on the RPi, if that specific error happens, causing the RPi to refresh it's IP, DHCP and whatnot. Would be great if someone could implement this on a newer build. I would for sure test it out.

[mattclar]

Updated to HassOS 3.11 didnt help :(

actually scratch that, seems to have fixed it

[gyanev]

I just had a similar situation. Everything worked initially, after installing hassio on linux using this script https://raw.githubusercontent.com/home-assistant/hassio-installer/master/hassio_install.sh. After the first restart I was not able to access some of the addons from the sidebar. In the log file I see a bunch of: ERROR (MainThread) [hassio.api.proxy] Client error on WebSocket API Cannot connect to host 172.30.32.1:8123 ssl:False [Host is unreachable] For me the issue was that the host firewall was in fact blocking access to 172.30.32.1. To test it out I've run iptables -I INPUT -s 172.30.32.0/23 -d 172.30.32.0/23 -j ACCEPT on the host and it worked for me.

Any chance some of you are affected by the same problem?

[bschatzow]

I am on a Raspberry Pi 3 B

[fernmac2]

Same error here, any update?

[dereitz]

One clue here for me is that my frontend is trying to connect to 172.30.32.2, but if I look at the output of ifconfig on my host, my current hassio interface is actually 172.30.32.1. Anyone know if the IP is dynamic?

[rlpowell]

OK, so, I have no idea if this is a general problem, but let me tell you how I fixed this for my system, which may just be about my system but who knows?, maybe it'll help you.

I have hass.io running under docker on my x86_64 Linux box, so if anything I'm saying seems wildly inapplicable, that's why. All the commands I give are run on the Docker host itself, unless otherwise specified.

The short version: This error means that your supervisor instance can't reach port 8123 on your homeassistant instance. docker uses iptables to set up the networking between instances, so inspect the iptables rules very carefully, and see if something wacky is going on.

The specific commands that solved my problem, but are unlikely to solve yours, are:

sudo iptables -I INPUT -s 172.16.0.0/12 -j ACCEPT sudo iptables -I INPUT -d 172.16.0.0/12 -j ACCEPT

(If those commands actually do solve this problem for a bunch of people, then something super wacky is going on with the supervisor's docker setup and someone should open a new bug saying that these commands solve this problem, but that seems super unlikely.)

More details:

On my system, at least, I can test the issue directly by using "docker ps -a" to find the supervisor instance, then "docker exec -it [id] bash" and running "curl http://172.30.32.1:8123/", where 172.30.32.1 seems to be the normal IP for the homeassistant instance. You can also do "docker exec -it [homassistant-instance-id] bash" and run "ip addr" to confirm which IP is which.

If that curl hangs or is rejected, but the same curl works from the homeassistant instance, or works from the docker instance itself, then the issue is probably that the docker networking between the two instances is broken.

Here's the details of the specific issue that was blocking docker networking in my case, but "docker networking is broken" is a HUGE topic; there are literally tens of thousands of threads on such issues all over the web.

For myself, I have iptables locked down pretty tight; I have default deny turned on for the INPUT chain in iptables. That is, the last rule in the INPUT chain is:

-A INPUT -j REJECT --reject-with icmp-host-prohibited

, and above that I have a bunch of white-listed stuff like port 22 or whatever.

Maintaining such a whitelist for all of my Docker stuff is a huge pain in the ass and is also kind of pointless, since I don't really care about communications between two docker instances as long as nothing's talking to the internet, so I have a special Docker whitelist like so:

-A INPUT -s 172.17.0.0/16 -j ACCEPT -A INPUT -d 172.17.0.0/16 -j ACCEPT

This is the normal range of Docker instance IPs I've seen in the past.

It turns out, however, that for whatever reason, the instances the hass.io supervisor generates aren't in that range; they're in 172.30.0.0/16

The fix was to just grab the entire 172.16 private range, which I should have done in the first place.

In other words, these commands solved the problem for me; I would be surprised if they solved the problem for you, but who knows?:

sudo iptables -I INPUT -s 172.16.0.0/12 -j ACCEPT sudo iptables -I INPUT -d 172.16.0.0/12 -j ACCEPT

[Devqon]

Same issue here, almost always after restarting HA it becomes unavailable. Sometimes a ha host reboot works, but more often I have to unplug the Raspberry Pi and replug it in, then it will start up everything. Very annoying to have to replug everytime I have to restart HA.

Running Hassio on Raspberry Pi 3b

[blamaz]

I start seeing this error right after the installation of Mosquito MQTT broker addon.

[Mecallie]

I start seeing this error right after the installation of Mosquito MQTT broker addon.

I did not made the link (installed some more addons) but indeed this seems to be causing an issue when upgrading!

My issue: after clicking update to 107.5 (from 107.1) the HassIO on my RPI3B fails to boot. I have to unplug it and reboot. The old version (.1) would then still be running, with the above error in the logs.

After disabling the mqtt broker addon and stopping it from running on boot the update went through fine! And enabling the addon after that does not seem to cause any issues anymore...

I don't know how docker get's it ip's, but is it possible that two containers are trying to use the same IP maybe? (Different port, thus generating the error...)

[blamaz]

I start seeing this error right after the installation of Mosquito MQTT broker addon.

I did not made the link (installed some more addons) but indeed this seems to be causing an issue when upgrading!

My issue: after clicking update to 107.5 (from 107.1) the HassIO on my RPI3B fails to boot. I have to unplug it and reboot. The old version (.1) would then still be running, with the above error in the logs.

After disabling the mqtt broker addon and stopping it from running on boot the update went through fine! And enabling the addon after that does not seem to cause any issues anymore...

I don't know how docker get's it ip's, but is it possible that two containers are trying to use the same IP maybe? (Different port, thus generating the error...)

Add your comment to issue #590, this one is closed.

mattclar mentioned this issue on 18 Feb Reopen issue #224 #590

[z99powell]

What worked for me was removing the static ip reservation on my router and the restarting home assistant

[paboman]

I have exactly the same problem, running HA on OVA virtual machine on a synology nas. getting command not found for this command via SSH: iptables -I INPUT -s 172.30.32.0/23 -d 172.30.32.0/23 -j ACCEPT

[rlpowell]

No idea what Synology uses for its firewall; that command is for standard Linux, but also Linux is now using nftables. The nftables equivalent appears to be nft insert rule ip filter INPUT ip saddr 172.30.32.0/23 ip daddr 172.30.32.0/23 counter accept.

[Bagunda]

I found the problem and was able to fix it. Problem in the Firewall! Dropping all the rules with the command iptables -t nat -F I saw that the File Editor addon opens as it should. Then I restored the firewall to the state "as it was before" with the reboot command. Or with netfilter-persistent reload commannd if apt-get install iptables-persistent is installed. (In order to save the current state of the firewall, I did netfilter-persistent save a few months ago.

After restoring the firewall, I removed one rule after another with the commands:

iptables --table nat -D POSTROUTING 5
iptables --table nat -D POSTROUTING 4

and after each removal, I looked to see if the addon worked.

In the end, I realized that the rules interfered: MASQUERADE all -- anywhere anywhere

I had several. To delete multiple rules, you need to enter the command: iptables -t nat -D POSTROUTING -j MASQUERADE several times until it gives an error iptables: Bad rule (does a matching rule exist in that chain?).

I had the following rules:

Chain PREROUTING (policy ACCEPT)
num target prot opt source destination
1 DNAT tcp -- anywhere haumea.vds.sh tcp dpt:3333 to:11.11.11.3:3000
2 DNAT tcp -- anywhere haumea.vds.sh tcp dpt:8888 to:11.11.11.4:80
3 DNAT tcp -- anywhere haumea.vds.sh tcp dpt:1122 to:11.11.11.22:22
4 DNAT tcp -- anywhere haumea.vds.sh tcp dpt:1122 to:10.11.12.10:80
5 DNAT tcp -- anywhere haumea.vds.sh tcp dpt:1123 to:10.11.12.10:443
6 DNAT tcp -- anywhere haumea.vds.sh tcp dpt:1122 to:11.11.11.18:80
7 DNAT tcp -- anywhere haumea.vds.sh tcp dpt:1123 to:11.11.11.18:443
8 DNAT tcp -- anywhere haumea.vds.sh tcp dpt:1180 to:11.11.11.18:80
9 DNAT tcp -- anywhere haumea.vds.sh tcp dpt:1181 to:11.11.11.18:443
10 DNAT tcp -- anywhere haumea.vds.sh tcp dpt:1182 to:11.11.11.18:8123
11 DOCKER all -- anywhere anywhere ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
num target prot opt source destination

Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
1 DNAT tcp -- anywhere haumea.vds.sh tcp dpt:http to:11.11.11.4
2 DOCKER all -- anywhere !127.0.0.0/8 ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
num target prot opt source destination
1 MASQUERADE all -- 172.17.0.0/16 anywhere
2 MASQUERADE all -- 172.30.32.0/23 anywhere
3 MASQUERADE all -- 11.11.11.0/24 anywhere
4 MASQUERADE all -- 172.16.0.0/24 anywhere
5 SNAT all -- anywhere anywhere to:172.16.0.1
6 SNAT tcp -- anywhere 11.11.11.4 tcp dpt:http to:11.11.11.1
7 MASQUERADE all -- anywhere anywhere
8 MASQUERADE all -- anywhere anywhere
9 MASQUERADE all -- anywhere anywhere
10 MASQUERADE all -- anywhere anywhere
11 MASQUERADE all -- anywhere anywhere
12 MASQUERADE tcp -- 172.17.0.2 172.17.0.2 tcp dpt:9000
13 MASQUERADE tcp -- 172.30.32.6 172.30.32.6 tcp dpt:http
14 MASQUERADE all -- 172.16.0.0/24 anywhere
15 SNAT all -- anywhere anywhere to:172.16.0.1
16 MASQUERADE all -- anywhere anywhere

Chain Docker (2 references)
num target prot opt source destination
1 RETURN all -- anywhere anywhere
2 RETURN all -- anywhere anywhere
3 DNAT tcp -- anywhere anywhere tcp dpt:4357 to:172.30.32.6:80

And in order for addons to work, now I have such a firewall:

Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination
1    DNAT       tcp  --  anywhere             haumea.vds.sh        tcp dpt:3333 to:11.11.11.3:3000
2    DNAT       tcp  --  anywhere             haumea.vds.sh        tcp dpt:8888 to:11.11.11.4:80
3    DNAT       tcp  --  anywhere             haumea.vds.sh        tcp dpt:1122 to:11.11.11.22:22
4    DNAT       tcp  --  anywhere             haumea.vds.sh        tcp dpt:1122 to:10.11.12.10:80
5    DNAT       tcp  --  anywhere             haumea.vds.sh        tcp dpt:1123 to:10.11.12.10:443
6    DNAT       tcp  --  anywhere             haumea.vds.sh        tcp dpt:1122 to:11.11.11.18:80
7    DNAT       tcp  --  anywhere             haumea.vds.sh        tcp dpt:1123 to:11.11.11.18:443
8    DNAT       tcp  --  anywhere             haumea.vds.sh        tcp dpt:1180 to:11.11.11.18:80
9    DNAT       tcp  --  anywhere             haumea.vds.sh        tcp dpt:1181 to:11.11.11.18:443
10   DNAT       tcp  --  anywhere             haumea.vds.sh        tcp dpt:1182 to:11.11.11.18:8123
11   DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination
1    DNAT       tcp  --  anywhere             haumea.vds.sh        tcp dpt:http to:11.11.11.4
2    DOCKER     all  --  anywhere            !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
num  target     prot opt source               destination
1    MASQUERADE  all  --  172.17.0.0/16        anywhere
2    MASQUERADE  all  --  172.30.32.0/23       anywhere
3    MASQUERADE  all  --  11.11.11.0/24        anywhere
4    MASQUERADE  all  --  172.16.0.0/24        anywhere
5    SNAT       all  --  anywhere             anywhere             to:172.16.0.1
6    SNAT       tcp  --  anywhere             11.11.11.4           tcp dpt:http to:11.11.11.1
7    MASQUERADE  tcp  --  172.17.0.2           172.17.0.2           tcp dpt:9000
8    MASQUERADE  tcp  --  172.30.32.6          172.30.32.6          tcp dpt:http
9    MASQUERADE  all  --  172.16.0.0/24        anywhere
10   SNAT       all  --  anywhere             anywhere             to:172.16.0.1
11   MASQUERADE  all  --  anywhere             anywhere

Chain DOCKER (2 references)
num  target     prot opt source               destination
1    RETURN     all  --  anywhere             anywhere
2    RETURN     all  --  anywhere             anywhere
3    DNAT       tcp  --  anywhere             anywhere             tcp dpt:4357 to:172.30.32.6:80

Command to list all rules in NAT tables with the line numbers: iptables --table nat --list --line-numbers