unimplemented call: SetDeadline, build on context fails with ssh connection errors

basz commented 1 year ago

Description

Hi, I have the following workflow on a swarm manager.

build images with docker compose from a docker-compose.yml (and build context)
deploy swarm stack with same docker-compose.yml

For this I do the following;

docker use context <somecontext>

I have created an overlay network once.

docker network create --attachable --driver=overlay proxy-tier

Then I run (locally) after my images have changed to build a new image with the image name from docker-compose-stack.yml.

docker compose -f docker-compose-stack.yml build

And finally I deploy my stack with

docker stack deploy <stackname> --compose-file docker-compose-stack.yml

This used to work with an docker-compose-stack.yml file as follows

version: '3.3'

networks:
  proxy-tier:
    external: true

services:
  nginx:
    image: ghcr.io/somename/somename:1.0.3

    build:
      context: nginx
      dockerfile: ./my.Dockerfile
      container_name: web-test
    networks: 
      - proxy-tier

It seems between the latest docker-ce (20.10.22, build 3a2c30b) and - not sure, i believe - version 18, this stopped working. I am receiving ssh errors.

docker compose --verbose -f docker-compose-stack.yml build

DEBU[0000] using default config store "/Users/bas/.docker/buildx" 
DEBU[0000] commandconn: starting ssh with [-l root -- xxxx.xxxx.nl docker system dial-stdio] 
DEBU[0001] commandconn: starting ssh with [-l root -- xxxx.xxxx.nl docker system dial-stdio] 
[+] Building 8.0s (2/2) FINISHED                                                                                                                                                                                                                                              
 => CANCELED [internal] load .dockerignore                                                                                                                                                                                                                               0.0s
 => CANCELED [internal] load build definition from Dockerfile                                                                                                                                                                                                            0.0s
DEBU[0001] commandconn: starting ssh with [-l root -- xxxx.xxxx.nl docker system dial-stdio] 
DEBU[0002] unimplemented call: SetDeadline(2023-02-08 14:17:28.708204 +0100 CET m=+21.784783292) 
DEBU[0002] unimplemented call: SetDeadline(0001-01-01 00:00:00 +0000 UTC) 
DEBU[0002] unimplemented call: SetReadDeadline(0001-01-01 00:00:00 +0000 UTC) 
DEBU[0002] commandconn: starting ssh with [-l root -- xxxx.xxxx.nl docker system dial-stdio] 
DEBU[0002] stopping session                              span="load buildkit capabilities"
 EBU[0002] commandconn (ssh):ssh: connect to host xxxx.xxxx.nl port 22: Connection refused
DEBU[0002] commandconn: starting ssh with [-l root -- xxxx.xxxx.nl docker system dial-stdio] 
 EBU[0002] commandconn (ssh):ssh: connect to host xxxx.xxxx.nl port 22: Connection refused
DEBU[0002] stopping session                             
failed to dial gRPC: command [ssh -l root -- xxxx.xxxx.nl docker system dial-stdio] has exited with exit status 255, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=ssh: connect to host xxxx.xxxx.nl port 22: Connection refused

Steps To Reproduce

No response

Compose Version

Docker Compose version v2.15.1

Docker Environment

➜  website git:(main) ✗ docker info
Client:
 Context:    bpm
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.10.0)
  compose: Docker Compose (Docker Inc., v2.15.1)
  dev: Docker Dev Environments (Docker Inc., v0.0.5)
  extension: Manages Docker extensions (Docker Inc., v0.2.17)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.23.0)

Server:
 Containers: 2
  Running: 2
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 23.0.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: 2qsi2tl71hi59vw5vyldoz06m
  Is Manager: true
  ClusterID: evzt0zmbjofhd3xupxanz9in5
  Managers: 1
  Nodes: 1
  Default Address Pool: 10.0.0.0/8  
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 10.18.0.5
  Manager Addresses:
   10.18.0.5:2377
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 31aa4358a36870b21a992d3ad2bef29e1d693bec
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-58-generic
 Operating System: Ubuntu 22.04.1 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 1.929GiB
 Name: bpm
 ID: 349570d0-8e23-4bab-a771-052e85f3cb0a
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

No response

milas commented 1 year ago

The unimplemented call debug lines should be fine - those options aren't supported for SSH connections but just mean the deadlines won't be respected.

Your error shows that the connection is being refused by the host:

connect to host xxxx.xxxx.nl port 22: Connection refused

Typically, network errors like this are not caused by Compose, but are the result of a misconfiguration elsewhere. For example, are all firewall ports open? Can you build images using docker buildx build using the same context?

basz commented 1 year ago

Yes well that's what I thought. However an older version (v18 I believe) did it no problem. I upgraded to the latest and then I run into this.

I have disabled the firewall and also tried on a new digital ocean box.

I've noticed sshd starts blocking me for 30 seconds. I believe due to many auth attempts. But I just don understand why the attempt would be incorrect...

Good to know the unimplemented thing is harmless.

Anyway to get more debug info?

basz commented 1 year ago

turns out it was the firewall. It had rate limiting enabled for ssh which means the 6th connection within 30 seconds is denied. sorry for the confusion...

docker / compose