search

Azure / batch-shipyard

Simplify HPC and Batch workloads on Azure
MIT License
277 stars 121 forks source link

ERROR - Error response from daemon: Head "https://mcr.microsoft.com/v2/azure-batch/shipyard/manifests/3.9.1-cargo": dial tcp 204.79.197.219:443: i/o timeout #376

Closed AndyPengYong closed 2 years ago

AndyPengYong commented 2 years ago

Problem Description

2022-06-08T05:12:34,397626166+0000 - DEBUG - Pulling Docker Image: mcr.microsoft.com/azure-batch/shipyard:3.9.1-cargo (fallback: 0) 2022-06-08T05:13:04,792495588+0000 - ERROR - Error response from daemon: Head "https://mcr.microsoft.com/v2/azure-batch/shipyard/manifests/3.9.1-cargo": dial tcp 204.79.197.219:443: i/o timeout 2022-06-08T05:13:04,794030532+0000 - ERROR - No fallback registry specified, terminating

Batch Shipyard Version

3.9.1

Batch Pool Configuration

pool_specification: id: scrum-pool vm_configuration: platform_image: offer: centos-container publisher: microsoft-azure-batch sku: 7-8 vm_count: dedicated: 1 low_priority: 0 vm_size: Standard_A2_v2 reboot_on_start_task_failed: true

Additional Logs

Linux 15f0febd29bf4e55b151a692b61a1eac000000 3.10.0-1127.19.1.el7.x86_64 #1 SMP Tue Aug 25 17:23:54 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux 2022-06-08T05:12:25,187903573+0000 - WARNING - Unknown DISTRIB_CODENAME. 2022-06-08T05:12:25,202975095+0000 - INFO - Prep start Configuration: Custom image: 0 Native mode: 1 OS Distribution: centos 7 Batch Shipyard version: 3.9.1 Blobxfer version: 1.9.4 Singularity version: User mountpoint: Mount path: /mnt/resource/batch/tasks/mounts Batch Insights: 0 Prometheus: NE=, CA=, Network optimization: 0 Encryption cert thumbprint: Install Kata Containers: 0 Default container runtime: runc Install BeeGFS BeeOND: 0 Storage cluster mounts (1): Custom mount: Install LIS: GPU: GPU ignore warnings: Azure Blob: 0 Azure File: 0 GlusterFS on compute: 0 HPN-SSH: 0 Enable Azure Batch group for Docker access: Fallback registry: Docker image preload delay: 0 Cascade via container: 1 Concurrent source downloads: 10 Block on images: # Singularity decryption certs:

2022-06-08T05:12:25,265201338+0000 - INFO - Ephemeral disk discovered as /dev/sdb 2022-06-08T05:12:25,281373591+0000 - DEBUG - lsblk: /dev/sdb1 8:17 0 20G 0 part /mnt/resource 2022-06-08T05:12:25,298301166+0000 - INFO - ephemeral: /dev/sdb1 (encrypted=0 user=/mnt/resource) 2022-06-08T05:12:25,370942501+0000 - INFO - VmSize=standard_a2_v2 RDMA=0 2022-06-08T05:12:25,374938513+0000 - INFO - LIS installation not required 2022-06-08T05:12:25,376557558+0000 - INFO - No singularity decryption certificates defined ● docker.service - Docker Application Container Engine Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2022-06-08 05:10:04 UTC; 2min 20s ago Docs: https://docs.docker.com Main PID: 1164 (dockerd) Tasks: 12 Memory: 843.6M CGroup: /system.slice/docker.service └─1164 /usr/bin/dockerd -H fd:// --containerd /var/run/containerd/containerd.sock

Jun 08 05:09:59 15f0febd29bf4e55b151a692b61a1eac000000 dockerd[1164]: time="2022-06-08T05:09:59.844055023Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc Jun 08 05:09:59 15f0febd29bf4e55b151a692b61a1eac000000 dockerd[1164]: time="2022-06-08T05:09:59.844076823Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/containerd/containerd.sock 0 }] }" module=grpc Jun 08 05:09:59 15f0febd29bf4e55b151a692b61a1eac000000 dockerd[1164]: time="2022-06-08T05:09:59.844087822Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc Jun 08 05:10:00 15f0febd29bf4e55b151a692b61a1eac000000 dockerd[1164]: time="2022-06-08T05:10:00.549123795Z" level=info msg="Loading containers: start." Jun 08 05:10:02 15f0febd29bf4e55b151a692b61a1eac000000 dockerd[1164]: time="2022-06-08T05:10:02.622634623Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address" Jun 08 05:10:03 15f0febd29bf4e55b151a692b61a1eac000000 dockerd[1164]: time="2022-06-08T05:10:03.637093010Z" level=info msg="Loading containers: done." Jun 08 05:10:04 15f0febd29bf4e55b151a692b61a1eac000000 dockerd[1164]: time="2022-06-08T05:10:04.350500980Z" level=info msg="Docker daemon" commit=847da184ad5048b27f5bdf9d53d070f731b43180 graphdriver(s)=overlay2 version=20.10.11+azure-3 Jun 08 05:10:04 15f0febd29bf4e55b151a692b61a1eac000000 dockerd[1164]: time="2022-06-08T05:10:04.351261171Z" level=info msg="Daemon has completed initialization" Jun 08 05:10:04 15f0febd29bf4e55b151a692b61a1eac000000 systemd[1]: Started Docker Application Container Engine. Jun 08 05:10:04 15f0febd29bf4e55b151a692b61a1eac000000 dockerd[1164]: time="2022-06-08T05:10:04.707034818Z" level=info msg="API listen on /var/run/docker.sock" Client: Version: 20.10.11+azure-3 API version: 1.41 Go version: go1.16.12 Git commit: dea9396e184290f638ea873c76db7c80efd5a1d2 Built: Wed Nov 17 23:49:46 2021 OS/Arch: linux/amd64 Context: default Experimental: true

Server: Engine: Version: 20.10.11+azure-3 API version: 1.41 (minimum version 1.12) Go version: go1.16.12 Git commit: 847da184ad5048b27f5bdf9d53d070f731b43180 Built: Thu Nov 18 00:21:59 2021 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.4.12+azure-1 GitCommit: 7b11cfaabd73bb80907dd23182b9347b4245eb5d runc: Version: 1.0.3 GitCommit: f46b6ba2c9314cfc8caae24a32ec5fe9ef1059fe docker-init: Version: 0.19.0 GitCommit:
Client: Context: default Debug Mode: false

Server: Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 2 Server Version: 20.10.11+azure-3 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 1 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux Default Runtime: runc Init Binary: docker-init containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d runc version: f46b6ba2c9314cfc8caae24a32ec5fe9ef1059fe init version: Kernel Version: 3.10.0-1127.19.1.el7.x86_64 Operating System: CentOS Linux 7 (Core) OSType: linux Architecture: x86_64 CPUs: 2 Total Memory: 3.701GiB Name: 15f0febd29bf4e55b151a692b61a1eac000000 ID: 5NBN:SBAR:2RJ4:4N7S:MW57:D6OV:YS2L:LIOZ:BXZ7:QMWC:J5AU:KNJJ Docker Root Dir: /mnt/resource/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false

2022-06-08T05:12:25,770292688+0000 - INFO - Docker root dir within ephemeral temp disk: /mnt/resource/docker 2022-06-08T05:12:25,771988035+0000 - INFO - Checking for Nvidia Hardware 00:00.0 Host bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX Host bridge (AGP disabled) (rev 03) 00:07.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 01) 00:07.1 IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev 01) 00:07.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 02) 00:08.0 VGA compatible controller: Microsoft Corporation Hyper-V virtual VGA 2022-06-08T05:12:26,119040957+0000 - INFO - No Nvidia card(s) detected! 2022-06-08T05:12:26+0000 - DEBUG - Logging into 1 Docker registry servers... 2022-06-08T05:12:26+0000 - DEBUG - Logging into Docker registry: scrumsalesdockerprd.azurecr.io with user: scrumsalesdockerprd WARNING! Using --password via the CLI is insecure. Use --password-stdin. WARNING! Your password will be stored unencrypted in /mnt/resource/batch/tasks/startup/wd/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded 2022-06-08T05:12:26+0000 - INFO - Docker registry logins completed. 2022-06-08T05:12:26+0000 - WARNING - No Singularity registry servers found. 2022-06-08T05:12:26,348386782+0000 - DEBUG - VM size standard_a2_v2 does not have IB RDMA 2022-06-08T05:12:26,349782721+0000 - DEBUG - Not an RDMA capable VM size, skipping IB detection/setup 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:22:48:e8:38:ff brd ff:ff:ff:ff:ff:ff inet 10.6.3.23/25 brd 10.6.3.127 scope global noprefixroute eth0 valid_lft forever preferred_lft forever inet6 fe80::222:48ff:fee8:38ff/64 scope link valid_lft forever preferred_lft forever 3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:bf:43:8c:ee brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever 2022-06-08T05:12:26,354430051+0000 - INFO - Batch Insights disabled. 2022-06-08T05:12:26,355991295+0000 - INFO - Prometheus node exporter disabled. 2022-06-08T05:12:26,360116311+0000 - INFO - Prometheus cAdvisor disabled. 2022-06-08T05:12:26,361768457+0000 - DEBUG - Pulling Docker Image: mcr.microsoft.com/blobxfer:1.9.4 (fallback: 0) 1.9.4: Pulling from blobxfer 89d9c30c1d48: Pulling fs layer 6de18253c5d3: Pulling fs layer 89d9c30c1d48: Verifying Checksum 89d9c30c1d48: Download complete 6de18253c5d3: Verifying Checksum 6de18253c5d3: Download complete 89d9c30c1d48: Pull complete 6de18253c5d3: Pull complete Digest: sha256:94192812382de05b77d8766720cf4c22cc84fd15c86a838a043366fa2047af83 Status: Downloaded newer image for mcr.microsoft.com/blobxfer:1.9.4 mcr.microsoft.com/blobxfer:1.9.4 2022-06-08T05:12:34,397626166+0000 - DEBUG - Pulling Docker Image: mcr.microsoft.com/azure-batch/shipyard:3.9.1-cargo (fallback: 0) 2022-06-08T05:13:04,792495588+0000 - ERROR - Error response from daemon: Head "https://mcr.microsoft.com/v2/azure-batch/shipyard/manifests/3.9.1-cargo": dial tcp 204.79.197.219:443: i/o timeout 2022-06-08T05:13:04,794030532+0000 - ERROR - No fallback registry specified, terminating

Additonal Comments

This issue happens around 1 time per month. we added the following parameter in pool.yaml file. But this parameter did not work as expected. reboot_on_start_task_failed: true

AndyPengYong commented 2 years ago

image

alfpark commented 2 years ago

reboot_on_start_task_failed is not applicable to auto_pool.