search

appveyor / ci

AppVeyor community support repository
https://www.appveyor.com
344 stars 65 forks source link

BYOC Docker: Error creating network #3819

Closed f4exb closed 2 years ago

f4exb commented 2 years ago

Trying to run a build on BYOC / Docker. It fails right away with:

Creating network...
Error creating network 'av-43576354'

https://ci.appveyor.com/project/f4exb/sdrangel/builds/43576354/job/kdgxk1sqcr2cn6jo

FeodorFitsner commented 2 years ago

Could you give some more information about the environment: names, editions, versions, etc?

f4exb commented 2 years ago

The Yaml file that drives the build is here: https://github.com/f4exb/sdrangel/blob/master/.appveyor.yml

Docker info ``` Client: Context: default Debug Mode: false Plugins: scan: Docker Scan (Docker Inc., v0.17.0) Server: Containers: 1 Running: 0 Paused: 0 Stopped: 1 Images: 14 Server Version: 20.10.7 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 1 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc Default Runtime: runc Init Binary: docker-init containerd version: runc version: init version: Security Options: apparmor seccomp Profile: default Kernel Version: 4.15.0-176-generic Operating System: Ubuntu 18.04.6 LTS OSType: linux Architecture: x86_64 CPUs: 8 Total Memory: 31.37GiB Name: nasrock ID: GZMP:ZBJV:WAAN:ZM2B:PCNC:VC3I:XTVR:FTKW:QJ7I:VFJI:GK4T:UT3B Docker Root Dir: /data/docker Debug Mode: false Username: f4exb06 Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false WARNING: No swap limit support ```
Appveyor host agent log ``` May 18 06:19:20 nasrock appveyor-host-agent[5428]: Assign job 's0rqdbsildo56qrr' to cloud 1450 May 18 06:19:20 nasrock appveyor-host-agent[5428]: info: Appveyor.HostAgent.AppHost[0] May 18 06:19:20 nasrock appveyor-host-agent[5428]: Trying to assign job s0rqdbsildo56qrr to cloud 1450. May 18 06:19:20 nasrock appveyor-host-agent[5428]: info: Appveyor.HostAgent.AppHost[0] May 18 06:19:20 nasrock appveyor-host-agent[5428]: Updating cloud availability (buildCloudId=1450): 19 worker(s) available, 1 worker(s) busy May 18 06:19:21 nasrock appveyor-host-agent[5428]: info: Appveyor.HostAgent.AppHost[0] May 18 06:19:21 nasrock appveyor-host-agent[5428]: Job s0rqdbsildo56qrr assigned to worker 65fb3964c2094ad4ba3dc199f198cb5a. May 18 06:19:21 nasrock appveyor-host-agent[5428]: info: Appveyor.HostAgent.Docker.WorkerCloud[0] May 18 06:19:21 nasrock appveyor-host-agent[5428]: [worker-1450-001] Received job message: JobId=s0rqdbsildo56qrr, JobName=f4exb/sdrangel/1439-master, JobTimeout=60, ImageName=Linux May 18 06:19:22 nasrock appveyor-host-agent[5428]: info: Appveyor.HostAgent.Docker.WorkerCloud[0] May 18 06:19:22 nasrock appveyor-host-agent[5428]: Create build 'bin' directory: /tmp/av-build-43579462 May 18 06:19:23 nasrock appveyor-host-agent[5428]: info: Appveyor.HostAgent.AppHost[0] May 18 06:19:23 nasrock appveyor-host-agent[5428]: Worker job started event: c151729bec0442948dae715a3f0a7ea5 May 18 06:19:23 nasrock appveyor-host-agent[5428]: appveyor-host-agent Information: 0 : May 18 06:19:23 nasrock appveyor-host-agent[5428]: docker version --format "{{.Client.Version}} {{.Client.Os}} {{.Client.Experimental}} {{.Server.Version}} {{.Server.Os}} {{.Server.Experimental}}" May 18 06:19:23 nasrock appveyor-host-agent[5428]: appveyor-host-agent Information: 0 : docker version --format "{{.Client.Version}} {{.Client.Os}} {{.Client.Experimental}} {{.Server.Version}} {{.Server.Os}} {{.Server.Experimental}}" May 18 06:19:23 nasrock appveyor-host-agent[5428]: appveyor-host-agent Information: 0 : May 18 06:19:23 nasrock appveyor-host-agent[5428]: appveyor-host-agent Information: 0 : Docker version: May 18 06:19:23 nasrock appveyor-host-agent[5428]: { May 18 06:19:23 nasrock appveyor-host-agent[5428]: "client": { May 18 06:19:23 nasrock appveyor-host-agent[5428]: "version": "20.10.7", May 18 06:19:23 nasrock appveyor-host-agent[5428]: "os": "linux", May 18 06:19:23 nasrock appveyor-host-agent[5428]: "experimental": true May 18 06:19:23 nasrock appveyor-host-agent[5428]: }, May 18 06:19:23 nasrock appveyor-host-agent[5428]: "server": { May 18 06:19:23 nasrock appveyor-host-agent[5428]: "version": "20.10.7", May 18 06:19:23 nasrock appveyor-host-agent[5428]: "os": "linux", May 18 06:19:23 nasrock appveyor-host-agent[5428]: "experimental": false May 18 06:19:23 nasrock appveyor-host-agent[5428]: } May 18 06:19:23 nasrock appveyor-host-agent[5428]: } May 18 06:19:23 nasrock appveyor-host-agent[5428]: appveyor-host-agent Information: 0 : docker image inspect appveyor-byoc-linux May 18 06:19:23 nasrock appveyor-host-agent[5428]: Docker version: { "client": { "version": "20.10.7", "os": "linux", "experimental": true }, "server": { "version": "20.10.7", "os": "linux", "experimental": false } } May 18 06:19:23 nasrock appveyor-host-agent[5428]: appveyor-host-agent Information: 0 : May 18 06:19:23 nasrock appveyor-host-agent[5428]: docker image inspect appveyor-byoc-linux May 18 06:19:23 nasrock appveyor-host-agent[5428]: appveyor-host-agent Information: 0 : May 18 06:19:23 nasrock appveyor-host-agent[5428]: appveyor-host-agent Information: 0 : docker network ls --format="{{.ID}}" --filter="label=av-build-43579462" May 18 06:19:23 nasrock appveyor-host-agent[5428]: docker network ls --format="{{.ID}}" --filter="label=av-build-43579462" May 18 06:19:23 nasrock appveyor-host-agent[5428]: appveyor-host-agent Information: 0 : May 18 06:19:23 nasrock appveyor-host-agent[5428]: docker network create --label="av-cloud-1450" --label="av-build-43579462" av-43579462 May 18 06:19:23 nasrock appveyor-host-agent[5428]: appveyor-host-agent Information: 0 : docker network create --label="av-cloud-1450" --label="av-build-43579462" av-43579462 May 18 06:19:26 nasrock appveyor-host-agent[5428]: info: Appveyor.HostAgent.Docker.WorkerCloud[0] May 18 06:19:26 nasrock appveyor-host-agent[5428]: [worker-1450-001] Started new container May 18 06:19:26 nasrock appveyor-host-agent[5428]: info: Appveyor.HostAgent.Docker.WorkerCloud[0] May 18 06:19:26 nasrock appveyor-host-agent[5428]: [worker-1450-001] Ensure job s0rqdbsildo56qrr has finished in 60 minutes... May 18 06:19:26 nasrock appveyor-host-agent[5428]: info: Appveyor.HostAgent.AppHost[0] May 18 06:19:26 nasrock appveyor-host-agent[5428]: Decommission worker event: c151729bec0442948dae715a3f0a7ea5 May 18 06:19:26 nasrock appveyor-host-agent[5428]: info: Appveyor.HostAgent.Docker.WorkerCloud[0] May 18 06:19:26 nasrock appveyor-host-agent[5428]: [worker-1450-001] Worker c151729bec0442948dae715a3f0a7ea5 has been decommissioned. Reverting worker... May 18 06:19:26 nasrock appveyor-host-agent[5428]: info: Appveyor.HostAgent.Docker.WorkerCloud[0] May 18 06:19:26 nasrock appveyor-host-agent[5428]: [worker-1450-001] Delete Docker container May 18 06:19:26 nasrock appveyor-host-agent[5428]: appveyor-host-agent Information: 0 : LocalDockerBuildCloudService.DeleteContainer c151729bec0442948dae715a3f0a7ea5 May 18 06:19:26 nasrock appveyor-host-agent[5428]: appveyor-host-agent Information: 0 : May 18 06:19:26 nasrock appveyor-host-agent[5428]: appveyor-host-agent Information: 0 : LocalDockerBuildCloudService.DeleteContainer - Deleting container av-c151729bec0442948dae715a3f0a7ea5 May 18 06:19:26 nasrock appveyor-host-agent[5428]: appveyor-host-agent Information: 0 : docker container ls -a --format="{{.ID}}" --filter="label=av-worker-c151729bec0442948dae715a3f0a7ea5" May 18 06:19:26 nasrock appveyor-host-agent[5428]: LocalDockerBuildCloudService.DeleteContainer c151729bec0442948dae715a3f0a7ea5 May 18 06:19:26 nasrock appveyor-host-agent[5428]: appveyor-host-agent Information: 0 : May 18 06:19:26 nasrock appveyor-host-agent[5428]: LocalDockerBuildCloudService.DeleteContainer - Deleting container av-c151729bec0442948dae715a3f0a7ea5 May 18 06:19:26 nasrock appveyor-host-agent[5428]: appveyor-host-agent Information: 0 : May 18 06:19:26 nasrock appveyor-host-agent[5428]: docker container ls -a --format="{{.ID}}" --filter="label=av-worker-c151729bec0442948dae715a3f0a7ea5" May 18 06:19:26 nasrock appveyor-host-agent[5428]: appveyor-host-agent Information: 0 : May 18 06:19:26 nasrock appveyor-host-agent[5428]: docker volume ls --format="{{.Name}}" --filter="label=av-worker-c151729bec0442948dae715a3f0a7ea5" May 18 06:19:26 nasrock appveyor-host-agent[5428]: appveyor-host-agent Information: 0 : docker volume ls --format="{{.Name}}" --filter="label=av-worker-c151729bec0442948dae715a3f0a7ea5" May 18 06:19:26 nasrock appveyor-host-agent[5428]: info: Appveyor.HostAgent.Docker.WorkerCloud[0] May 18 06:19:26 nasrock appveyor-host-agent[5428]: [worker-1450-001] Docker container with ID (null) has been deleted May 18 06:19:27 nasrock appveyor-host-agent[5428]: info: Appveyor.HostAgent.Docker.WorkerCloud[0] May 18 06:19:27 nasrock appveyor-host-agent[5428]: [worker-1450-001] Create worker May 18 06:19:27 nasrock appveyor-host-agent[5428]: info: Appveyor.HostAgent.AppHost[0] May 18 06:19:27 nasrock appveyor-host-agent[5428]: Updating cloud availability (buildCloudId=1450): 20 worker(s) available, 0 worker(s) busy May 18 06:19:27 nasrock appveyor-host-agent[5428]: info: Appveyor.HostAgent.Docker.WorkerCloud[0] May 18 06:19:27 nasrock appveyor-host-agent[5428]: [worker-1450-001] Ready May 18 06:19:27 nasrock appveyor-host-agent[5428]: info: Appveyor.HostAgent.AppHost[0] May 18 06:19:27 nasrock appveyor-host-agent[5428]: Updating cloud availability (buildCloudId=1450): 20 worker(s) available, 0 worker(s) busy May 18 06:19:32 nasrock appveyor-host-agent[5428]: appveyor-host-agent Information: 0 : May 18 06:19:32 nasrock appveyor-host-agent[5428]: appveyor-host-agent Information: 0 : LocalDockerBuildCloudService.CleanupBuildResources [43579462] May 18 06:19:32 nasrock appveyor-host-agent[5428]: appveyor-host-agent Information: 0 : docker volume ls --format="{{.Name}}" --filter="label=av-build-43579462" May 18 06:19:32 nasrock appveyor-host-agent[5428]: LocalDockerBuildCloudService.CleanupBuildResources [43579462] May 18 06:19:32 nasrock appveyor-host-agent[5428]: appveyor-host-agent Information: 0 : May 18 06:19:32 nasrock appveyor-host-agent[5428]: docker volume ls --format="{{.Name}}" --filter="label=av-build-43579462" May 18 06:19:32 nasrock appveyor-host-agent[5428]: appveyor-host-agent Information: 0 : May 18 06:19:32 nasrock appveyor-host-agent[5428]: appveyor-host-agent Information: 0 : docker network ls --format="{{.ID}}" --filter="label=av-build-43579462" May 18 06:19:32 nasrock appveyor-host-agent[5428]: docker network ls --format="{{.ID}}" --filter="label=av-build-43579462" May 18 06:19:32 nasrock appveyor-host-agent[5428]: appveyor-host-agent Information: 0 : May 18 06:19:32 nasrock appveyor-host-agent[5428]: sudo rm -r "/tmp/av-build-43579462" May 18 06:19:32 nasrock appveyor-host-agent[5428]: info: Appveyor.HostAgent.Docker.WorkerCloud[0] May 18 06:19:32 nasrock appveyor-host-agent[5428]: Deleting build 'bin' directory: /tmp/av-build-43579462 May 18 06:19:32 nasrock appveyor-host-agent[5428]: appveyor-host-agent Information: 0 : sudo rm -r "/tmp/av-build-43579462" May 18 06:19:32 nasrock sudo[8960]: appveyor : TTY=unknown ; PWD=/opt/appveyor/host-agent ; USER=root ; COMMAND=/bin/rm -r /tmp/av-build-43579462 May 18 06:19:32 nasrock sudo[8960]: pam_unix(sudo:session): session opened for user root by (uid=0) May 18 06:19:32 nasrock sudo[8960]: pam_unix(sudo:session): session closed for user root May 18 06:19:47 nasrock appveyor-host-agent[5428]: info: Appveyor.HostAgent.AppHost[0] May 18 06:19:47 nasrock appveyor-host-agent[5428]: Checking Host Agent connection: 5b0f0ad3-eee0-43ee-ab4b-932585026ef6 May 18 06:19:48 nasrock appveyor-host-agent[5428]: info: Appveyor.HostAgent.AppHost[0] May 18 06:19:48 nasrock appveyor-host-agent[5428]: Host Agent connection is alive ```
f4exb commented 2 years ago

OK issue was on the host I've solved the issue thanks to this post: https://stackoverflow.com/questions/43720339/docker-error-could-not-find-an-available-non-overlapping-ipv4-address-pool-am

Log of actions ``` 06:28:48 f4exb@nasrock:/mnt/z1/development/sdrangel$ docker network create --label="av-cloud-1450" --label="av-build-43579462" av-43579462 Error response from daemon: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network 06:28:55 f4exb@nasrock:/mnt/z1/development/sdrangel$ docker network ls NETWORK ID NAME DRIVER SCOPE 6ea7e1977841 av-41051142 bridge local e00a04aad9d4 av-41057119 bridge local 62fa79533f2c av-41065114 bridge local d7eca89db239 av-41065509 bridge local d6bbaec259eb av-41115911 bridge local 280fdb4a4d01 av-41117322 bridge local afb98b367346 av-41120458 bridge local 2ac4d4928774 av-41122630 bridge local 0b22b2b98b2d av-41128607 bridge local 4527195e5d0f av-41131153 bridge local cbd3e1f5ebae av-41137008 bridge local b37aafb594a5 av-41149702 bridge local 12aba7ce5116 av-41388922 bridge local 021f13b068ff av-41607910 bridge local ae3d894a4f13 av-41764699 bridge local 1b96c00e9fb1 av-41986294 bridge local cde7d73ffa83 av-42283032 bridge local 276bcbd18c78 av-42338954 bridge local 427a0a4d4e50 av-42749972 bridge local ee3f52f91422 av-42798231 bridge local a961714e8fe5 av-42945387 bridge local 177ba4f34565 av-42961794 bridge local 743aae77ab5b av-43368926 bridge local 5c0e65de5bde av-43540537 bridge local 237db859eb3e av-43544204 bridge local 669f475dfa80 av-43544205 bridge local 32399445fa5a av-43548079 bridge local dea99b87eaab av-43550960 bridge local e3c01ef36ae9 av-43550961 bridge local b84e39ad5670 bridge bridge local 5511cbfc9486 host host local 0a30c9ee59e2 none null local 06:30:15 f4exb@nasrock:/mnt/z1/development/sdrangel$ docker network prune WARNING! This will remove all custom networks not used by at least one container. Are you sure you want to continue? [y/N] y Deleted Networks: av-41115911 av-43550960 av-42945387 av-41149702 av-41131153 av-43544205 av-41764699 av-41065509 av-42798231 av-41607910 av-42338954 av-41065114 av-42283032 av-41986294 av-41117322 av-41128607 av-42961794 av-43550961 av-42749972 av-41137008 av-41122630 av-43540537 av-43368926 av-41120458 av-41388922 av-43544204 av-41057119 av-43548079 av-41051142 ```

Now I don't know if this is normal that the previous builds have left so many dangling networks behind them...

f4exb commented 2 years ago

Now that I know how to fix this problem you may close this issue if there is no follow up on your side.

FeodorFitsner commented 2 years ago

Thanks for the update with a solution!