containrrr / watchtower

A process for automating Docker container base image updates.
https://containrrr.dev/watchtower/
Apache License 2.0
19.76k stars 874 forks source link

Watchtower failed to create containers on host network #1906

Open c-def opened 10 months ago

c-def commented 10 months ago

Describe the bug

Watchguard has been keeping my containers up to date for years. Today I installed an update to Docker, and when Watchtower attempted to update two of my containers on the host network, it failed with the following error:

invalid config for network host: invalid endpoint settings: network-scoped alias is supported only for containers in user defined networks

These were both created via docker-compose with network_mode set to host

Steps to reproduce

Make Watchtower attempt to update a container on the host network.

Expected behavior

New container is created successfully.

Screenshots

No response

Environment

Your logs

ERRO[12591] Error response from daemon: invalid config for network host: invalid endpoint settings:
network-scoped alias is supported only for containers in user defined networks 
INFO[12591] Creating /sonarr                             
INFO[12595] Creating /home-assistant                     
ERRO[12595] Error response from daemon: invalid config for network host: invalid endpoint settings:
network-scoped alias is supported only for containers in user defined networks

Here is the log from one week ago when it worked just fine:
2024-01-13T10:02:31.051969197Z INFO[3032453] Creating /home-assistant                     
2024-01-13T10:02:34.702164592Z INFO[3032457] Creating /sonarr

Additional context

No response

github-actions[bot] commented 10 months ago

Hi there! πŸ‘‹πŸΌ As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! πŸ™

rany2 commented 10 months ago

I'm also facing the same issue:

time="2024-01-20T18:16:36Z" level=info msg="Creating /container1"
time="2024-01-20T18:16:36Z" level=error msg="Error response from daemon: invalid config for network host: invalid endpoint settings:\nnetwork-scoped alias is supported only for containers in user defined networks"
time="2024-01-20T18:16:36Z" level=info msg="Creating /container2"
time="2024-01-20T18:16:36Z" level=error msg="Error response from daemon: invalid config for network host: invalid endpoint settings:\nnetwork-scoped alias is supported only for containers in user defined networks"
jgeusebroek commented 10 months ago

It also happens with non-host networking.

manderss99 commented 10 months ago

I also have this with containers that use both host and bridge network, started recently, possibly by docker-ce update. docker-ce/jammy,now 5:25.0.0-1~ubuntu.22.04~jammy

jmnovak50 commented 10 months ago

Seeing also with Raspberry Pi Docker version 25.0.0, build e758fe5

BrodyBuster commented 10 months ago

I also have this with containers that use both host and bridge network, started recently, possibly by docker-ce update. docker-ce/jammy,now 5:25.0.0-1~ubuntu.22.04~jammy

Debian Bookworm - Docker version 25.0.0, build e758fe5

Being there hasn't been an update to watchtower in some time, I also suspect this is a docker issue. Following along here, in the event a solution is found. For the time being I had to disable watchtower.

MrEAlderson commented 10 months ago

Same problem. What's worse is that it suddenly deleted the whole container without recreating it...

alexandervnuchkov commented 10 months ago

In my instance, this affects containers created using docker run. Containers started with docker compose update without any problem. The issues started after docker updated to version 25.0.0

rany2 commented 10 months ago

Containers started with docker compose update without any problem.

Actually, my containers are all started with docker compose and had this problem.

c-def commented 10 months ago

Just wanted to mention I have the same issue with Portainer if I try to edit/duplicate these same containers.

farces commented 10 months ago

Chiming in as well, created using docker-compose and on the bridge network. There was a docker-ce update a few days ago so it's entirely possible that's the culprit, but the first symptom I had was the error during watchtower recreating the container.

Starting manually in Portainer worked fine afterwards.

time="2024-01-22T00:07:24Z" level=info msg="Found new jwilder/nginx-proxy:latest image (c55e0435bcab)"
time="2024-01-22T00:07:25Z" level=info msg="Stopping /nginx-proxy (b23e7978ed10) with SIGTERM"
time="2024-01-22T00:07:35Z" level=info msg="Creating /nginx-proxy"
time="2024-01-22T00:07:35Z" level=error msg="Error response from daemon: invalid config for network bridge: invalid endpoint settings:\nnetwork-scoped alias is supported only for containers in user defined networks"
time="2024-01-22T00:07:35Z" level=info msg="Session done" Failed=1 Scanned=7 Updated=0 notify=no

Docker version details:

face@localhost:~/nginx-proxy$ docker version
Client: Docker Engine - Community
 Version:           25.0.0
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        e758fe5
 Built:             Thu Jan 18 17:09:49 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          25.0.0
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       615dfdf
  Built:            Thu Jan 18 17:09:49 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.27
  GitCommit:        a1496014c916f9e62104b33d1bb5bd03b0858e59
 runc:
  Version:          1.1.11
  GitCommit:        v1.1.11-0-g4bccb38
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
jmnovak50 commented 10 months ago

Not sure this has something to do with it. This also failed for a container being configured on the system defined bridge network...Created a non-system created bridge network and then recreated the container under that user-defined bridge network. An update came in for that container and it recreated just fine. Wondering if this has to do with the system defined networks (bridge, host, none) specifically? Worth noting I also moved the Watchtower container to the user defined bridge network.

CJO100293 commented 10 months ago

Just chiming in to say ive also got this same problem. Started around the same time as everyone else.

rursache commented 10 months ago

same issue here, watchtower kills and delete the containers but never re-creates them. as someone with over 100 containers, playing "what disappeared last night" everyday is very annoying. this bug should be highest priority not only "medium"

time="2024-01-21T05:59:35+02:00" level=error msg="Error response from daemon: invalid config for network host: invalid endpoint settings:\nnetwork-scoped alias is supported only for containers in user defined networks"
time="2024-01-21T05:59:35+02:00" level=info msg="Creating /immich"
time="2024-01-21T05:59:35+02:00" level=error msg="Error response from daemon: invalid config for network bridge: invalid endpoint settings:\nnetwork-scoped alias is supported only for containers in user defined networks"
time="2024-01-21T05:59:35+02:00" level=info msg="Creating /privatebin"
time="2024-01-21T05:59:35+02:00" level=error msg="Error response from daemon: invalid config for network bridge: invalid endpoint settings:\nnetwork-scoped alias is supported only for containers in user defined networks"
time="2024-01-21T05:59:35+02:00" level=info msg="Creating /fluidd"
time="2024-01-21T05:59:35+02:00" level=error msg="Error response from daemon: invalid config for network bridge: invalid endpoint settings:\nnetwork-scoped alias is supported only for containers in user defined networks"
time="2024-01-21T05:59:35+02:00" level=info msg="Creating /calibre-web"
time="2024-01-21T05:59:35+02:00" level=error msg="Error response from daemon: invalid config for network bridge: invalid endpoint settings:\nnetwork-scoped alias is supported only for containers in user defined networks"
time="2024-01-21T05:59:35+02:00" level=info msg="Creating /yourls"
time="2024-01-21T05:59:35+02:00" level=error msg="Error response from daemon: invalid config for network bridge: invalid endpoint settings:\nnetwork-scoped alias is supported only for containers in user defined networks"
time="2024-01-21T05:59:35+02:00" level=info msg="Creating /vscodeserver"
time="2024-01-21T05:59:35+02:00" level=error msg="Error response from daemon: invalid config for network bridge: invalid endpoint settings:\nnetwork-scoped alias is supported only for containers in user defined networks"
time="2024-01-21T05:59:35+02:00" level=info msg="Creating /sonarr"
time="2024-01-21T05:59:35+02:00" level=error msg="Error response from daemon: invalid config for network bridge: invalid endpoint settings:\nnetwork-scoped alias is supported only for containers in user defined networks"
time="2024-01-21T05:59:35+02:00" level=info msg="Creating /pairdrop"
time="2024-01-21T05:59:35+02:00" level=error msg="Error response from daemon: invalid config for network bridge: invalid endpoint settings:\nnetwork-scoped alias is supported only for containers in user defined networks"
time="2024-01-21T05:59:35+02:00" level=info msg="Creating /homepage"
time="2024-01-21T05:59:35+02:00" level=error msg="Error response from daemon: invalid config for network bridge: invalid endpoint settings:\nnetwork-scoped alias is supported only for containers in user defined networks"

seems to be happening since updating docker to 25.0.0

spupuz commented 10 months ago

same error and problem here

BrodyBuster commented 10 months ago

FWIW, I moved all my containers that were on the default system HOST network (network_mode: host) to a user bridge network, and was able to eliminate the error. All containers now update per usual.

xinud190 commented 10 months ago

FWIW, I moved all my containers that were on the default system HOST network (network_mode: host) to a user bridge network, and was able to eliminate the error. All containers now update per usual.

Can you recover containers Watchdog had stopped and creating (but received this error) ?

Can you make the change you mentioned and this will allow them to recover / watchdog complete the creation?

spupuz commented 10 months ago

this is happening not only on host container but also on container already configured with bridge, and this is random since container in bridge are not recrarete but at the next update after a compose restart they update work properly.

BrodyBuster commented 10 months ago

FWIW, I moved all my containers that were on the default system HOST network (network_mode: host) to a user bridge network, and was able to eliminate the error. All containers now update per usual.

Can you recover containers Watchdog had stopped and creating (but received this error) ?

Can you make the change you mentioned and this will allow them to recover / watchdog complete the creation?

When my containers failed, prior to moving them to a user bridge, the updated image was downloaded, the container was stopped and deleted, and watchtower failed to recreate the container and start it. I had recreate the container using docker run / docker compose.

This morning I moved containers that had failed previously on the default host network to a user bridge, with the exception of one container. All of these containers had available updates. The bridge containers updated fine, the host network container failed.

I will continue to monitor this to see if the bridge containers end up failing gain.

rmtsrc commented 10 months ago

FWIW, I moved all my containers that were on the default system HOST network (network_mode: host) to a user bridge network, and was able to eliminate the error. All containers now update per usual.

I'm facing the same problem, however some containers need access to my host network, so switching them to a bridged network wouldn't work for me.

CJO100293 commented 10 months ago

FWIW, I moved all my containers that were on the default system HOST network (network_mode: host) to a user bridge network, and was able to eliminate the error. All containers now update per usual.

Can you recover containers Watchdog had stopped and creating (but received this error) ? Can you make the change you mentioned and this will allow them to recover / watchdog complete the creation?

When my containers failed, prior to moving them to a user bridge, the updated image was downloaded, the container was stopped and deleted, and watchtower failed to recreate the container and start it. I had recreate the container using docker run / docker compose.

This morning I moved containers that had failed previously on the default host network to a user bridge, with the exception of one container. All of these containers had available updates. The bridge containers updated fine, the host network container failed.

I will continue to monitor this to see if the bridge containers end up failing gain.

Unfortunately ive got several containers that need access to the host network, so like rmtsrc, using bridge network unfortunately isnt a fix.

etho201 commented 10 months ago

Many containers supporting the home-assistant ecosystem also require the containers to share its network namespace with the host machine. These have been failing when updated using Watchtower; however, Docker compose is able to bring them up without issue. Hoping the issue with Watchtower is resolved soon!

xinud190 commented 10 months ago

I ended up restoring my Proxmox VM Backup of the Linux VM I'm using running docker. Restored all my containers and I disabled Watchtower so it doesn't upgrade containers to latest image.

Hoping this gets resolved soon.

3shirts commented 10 months ago

I have this too but it doesn't seem to be exclusive to Watchtower. If I manually try to redeploy a container from Portainer I get this same error. Redeploying the stack works though so docker compose is the workaround. Looks like a docker-related change that's caused this.

alfchao commented 10 months ago

msg="Error response from daemon: invalid config for network host: invalid endpoint settings:\nnetwork-scoped alias is supported only for containers in user defined networks"

xinud190 commented 10 months ago

Looks like they are fixing this in docker ...

https://github.com/docker/for-linux/issues/1481#issuecomment-1905621489

ptr727 commented 10 months ago

Same here:
time="2024-01-22T02:03:26-08:00" level=error msg="Error response from daemon: invalid config for network host: invalid endpoint settings:\nnetwork-scoped alias is supported only for containers in user defined networks"

xinud190 commented 10 months ago

FYI - Problem for docker was resolved and fixes released

https://github.com/docker/for-linux/issues/1481#issuecomment-1907136921

https://github.com/docker/for-linux/issues/1481

Personally, I will be waiting a bit before updating docker and re-enabling Watchtower.

If anyone else tests updated docker with Watchtower to confirm its fixed love to hear it.

CJO100293 commented 10 months ago

FYI - Problem for docker was resolved and fixes released

docker/for-linux#1481 (comment)

docker/for-linux#1481

Personally, I will be waiting a bit before updating docker and re-enabling Watchtower.

If anyone else tests updated docker with Watchtower to confirm its fixed love to hear it.

Ive tested and confirmed that the newest update to docker-ce (25.0.1) has resolved the above issues for me

farces commented 10 months ago

Can also confirm 25.0.1 of docker-ce resolves the issue. I think this can be closed as it’s not a watchtower specific issue regardless.