search

docker / desktop-linux

Bug reports for Docker Desktop for Linux
https://docs.docker.com/desktop/linux/
73 stars 5 forks source link

Can not list all ports the system LISTEN to when docker daemon crashed #187

Open KES777 opened 5 months ago

KES777 commented 5 months ago

Description

I had not visited my host about 1 year. There is SSHD daemon and docker with different containers which expose their ports. This is Ubuntu system. Docker service seems is down.

Reproduce

# uname -a
Linux ubuntu-2gb-nbg1-2 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

# netstat -na
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:52697         0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN     
tcp6       0      0 ::1:52697               :::*                    LISTEN     
tcp6       0      0 :::22                   :::*                    LISTEN     
udp        0      0 127.0.0.53:53           0.0.0.0:*                          
udp        0      0 myip:68       0.0.0.0:*                          
udp        0      0 0.0.0.0:4789            0.0.0.0:*                          
raw6       0      0 :::58                   :::*                    7          
Active UNIX domain sockets (servers and established)

# docker ps
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

# systemctl status docker
× docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2023-12-21 05:41:24 UTC; 1 week 5 days >
TriggeredBy: × docker.socket
       Docs: https://docs.docker.com
   Main PID: 3068220 (code=exited, status=1/FAILURE)
        CPU: 96ms

Notice: journal has been rotated since unit was started, output may be incomplete.

# docker --version
Docker version 20.10.21, build baeda1f

$ curl -k https://myip:9443
<!doctype html>...

But I know that some containers are still running, because I can open a VPN connection and access to https://myip:9443.

Why I do not see this :9443 port from netstat -na output?

UPD
I cleanup the space: https://superuser.com/a/1824057/431840 After starting docker service back I can see invisible services:

# netstat -na
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:9443            0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:52697         0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN     
tcp6       0      0 ::1:52697               :::*                    LISTEN     
tcp6       0      0 :::9443                 :::*                    LISTEN     
tcp6       0      0 :::22                   :::*                    LISTEN     
tcp6       0      0 :::80                   :::*                    LISTEN     
tcp6       0      0 :::443                  :::*                    LISTEN     
udp        0      0 127.0.0.53:53           0.0.0.0:*                          
udp        0      0 myip:68       0.0.0.0:*                          
udp        0      0 0.0.0.0:4789            0.0.0.0:*                          
raw6       0      0 :::58                   :::*                    7

Expected behavior

If I can access 9443 port (exposed from portainer container), then system listens on 9443 port and this should be displayed regrading docker service is run or not.

docker version

# docker version
Client: Docker Engine - Community
 Version:           20.10.21
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        baeda1f
 Built:             Tue Oct 25 18:01:58 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.21
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       3056208
  Built:            Tue Oct 25 17:59:49 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.9
  GitCommit:        1c90a442489720eec95342e1789ee8a5e1b9536f
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

# docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.9.1-docker)
  compose: Docker Compose (Docker Inc., v2.12.2)
  scan: Docker Scan (Docker Inc., v0.21.0)

Server:
 Containers: 30
  Running: 8
  Paused: 0
  Stopped: 22
 Images: 39
 Server Version: 20.10.21
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: local
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: error
  NodeID: 
  Error: error while loading TLS certificate in /mnt/docker-overlay/swarm/certificates/swarm-node.crt: certificate (1 - 38oh62w7k7zugmteg0xyzqn9g) not valid after Wed, 27 Dec 2023 01:39:00 UTC, and it is currently Tue, 02 Jan 2024 22:16:41 UTC: x509: certificate has expired or is not yet valid: 
  Is Manager: false
  Node Address: 167.235.58.111
 Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 1c90a442489720eec95342e1789ee8a5e1b9536f
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.15.0-56-generic
 Operating System: Ubuntu 22.04.1 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 7.578GiB
 Name: ubuntu-2gb-nbg1-2
 ID: X6IR:CQPM:NB3G:ZGBT:WWGU:UIC3:UDD4:TPNV:GHFF:4GRR:YVKC:OUMF
 Docker Root Dir: /mnt/docker-overlay
 Debug Mode: true
  File Descriptors: 94
  Goroutines: 97
  System Time: 2024-01-02T22:32:20.341046307Z
  EventsListeners: 2
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Diagnostics ID

no id

Additional Info

I suppose docker was crushed because of storage issue. Do not know why it takes so much space. I configured log rotation:

# ls -la
total 6818696
drwx--x---  4 root root       4096 Jan  2 22:29 .
drwx--x--- 33 root root       4096 Jan 26  2023 ..
-rw-r-----  1 root root 6978717361 Jan  2 22:29 a90baaa15da31f2ab9a4683af417b232166e7bf84591a1cd4f96aed858f06e8a-json.log
drwx------  2 root root       4096 Jan 16  2023 checkpoints
-rw-------  1 root root       3661 Jan  2 22:29 config.v2.json
-rw-r--r--  1 root root       1485 Jan  2 22:29 hostconfig.json
-rw-r--r--  1 root root         13 Jan  2 22:29 hostname
-rw-r--r--  1 root root        150 Jan  2 22:29 hosts
drwx--x---  2 root root       4096 Jan 16  2023 mounts
-rw-r--r--  1 root root         53 Jan  2 22:29 resolv.conf
-rw-r--r--  1 root root         71 Jan  2 22:29 resolv.conf.hash
root@ubuntu-2gb-nbg1-2:/mnt/docker-overlay/containers/a90baaa15da31f2ab9a4683af417b232166e7bf84591a1cd4f96aed858f06e8a# cat /etc/docker/daemon.json 
{
  "debug": true,
  "data-root": "/mnt/docker-overlay",
  "features":
    { "buildkit": true },
  "log-driver": "local",
  "log-opts": {
    "max-size": "500m",
    "max-file": "3"
  }
}

Latest logs from portainer service (port 9443):

{"time":1703948774,"message":"http: TLS handshake error from 87.236.176.41:41127: EOF"}
{"time":1703948807,"message":"http: TLS handshake error from 87.236.176.43:37637: EOF"}
{"time":1703950290,"message":"http: TLS handshake error from 185.233.19.154:38080: EOF"}
{"time":1703950290,"message":"http: TLS handshake error from 185.233.19.154:38158: EOF"}
{"time":1703950290,"message":"http: TLS handshake error from 185.233.19.154:38328: tls: no cipher suite supported by both client and server"}
{"time":1703950291,"message":"http: TLS handshake error from 185.233.19.154:38440: tls: client requested unsupported application protocols ([http/0.9 http/1.0 spdy/1 spdy/2 spdy/3 h2c hq])"}
{"time":1703950291,"message":"http: TLS handshake error from 185.233.19.154:38862: tls: client requested unsupported application protocols ([hq h2c spdy/3 spdy/2 spdy/1 http/1.0 http/0.9])"}
{"time":1703950291,"message":"http: TLS handshake error from 185.233.19.154:39022: tls: client offered only unsupported versions: [302 301]"}
{"time":1703950292,"message":"http: TLS handshake error from 185.233.19.154:39136: EOF"}
{"time":1703950292,"message":"http: TLS handshake error from 185.233.19.154:39376: EOF"}
{"time":1703950293,"message":"http: TLS handshake error from 185.233.19.154:39628: EOF"}
{"time":1703950293,"message":"http: TLS handshake error from 185.233.19.154:40084: EOF"}
{"time":1703950294,"message":"http: TLS handshake error from 185.233.19.154:40328: EOF"}
{"time":1703951039,"message":"http: TLS handshake error from 80.66.88.204:65062: tls: first record does not look like a TLS handshake"}
{"time":1703951155,"message":"http: TLS handshake error from 165.154.244.17:47702: EOF"}
{"time":1703951155,"message":"http: TLS handshake error from 165.154.244.17:50186: tls: first record does not look like a TLS handshake"}
{"time":1703951175,"message":"http: TLS handshake error from 165.154.244.17:54624: EOF"}
{"time":1703951176,"message":"http: TLS handshake error from 165.154.244.17:54852: EOF"}
{"time":1703951177,"message":"http: TLS handshake error from 165.154.244.17:55080: tls: no cipher suite supported by both client and server"}
{"time":1703951178,"message":"http: TLS handshake error from 165.154.244.17:55360: tls: client requested unsupported application protocols ([http/0.9 http/1.0 spdy/1 spdy/2 spdy/3 h2c hq])"}
{"time":1703951178,"message":"http: TLS handshake error from 165.154.244.17:55462: tls: client requested unsupported application protocols ([hq h2c spdy/3 spdy/2 spdy/1 http/1.0 http/0.9])"}
{"time":1703951180,"message":"http: TLS handshake error from 165.154.244.17:55558: tls: client offered only unsupported versions: [302 301]"}
{"time":1703951181,"message":"http: TLS handshake error from 165.154.244.17:55860: EOF"}
{"time":1703951182,"message":"http: TLS handshake error from 165.154.244.17:56074: EOF"}
{"time":1703951183,"message":"http: TLS handshake error from 165.154.244.17:56270: EOF"}
{"time":1703951184,"message":"http: TLS handshake error from 165.154.244.17:56438: EOF"}
{"time":1703977018,"message":"http: TLS handshake error from 205.210.31.17:50013: tls: client offered only unsupported versions: [302 301]"}
{"time":1703998157,"message":"http: TLS handshake error from 162.243.152.18:39820: tls: first record does not look like a TLS handshake"}
{"time":1704040224,"message":"http: TLS handshake error from 185.233.19.145:38628: EOF"}
{"time":1704040225,"message":"http: TLS handshake error from 185.233.19.145:38722: EOF"}
{"time":1704040225,"message":"http: TLS handshake error from 185.233.19.145:39130: tls: no cipher suite supported by both client and server"}
{"time":1704040225,"message":"http: TLS handshake error from 185.233.19.145:39190: tls: client requested unsupported application protocols ([http/0.9 http/1.0 spdy/1 spdy/2 spdy/3 h2c hq])"}
{"time":1704040226,"message":"http: TLS handshake error from 185.233.19.145:39256: tls: client requested unsupported application protocols ([hq h2c spdy/3 spdy/2 spdy/1 http/1.0 http/0.9])"}
{"time":1704040226,"message":"http: TLS handshake error from 185.233.19.145:39280: tls: client offered only unsupported versions: [302 301]"}
{"time":1704040227,"message":"http: TLS handshake error from 185.233.19.145:39306: EOF"}
{"time":1704040227,"message":"http: TLS handshake error from 185.233.19.145:39380: EOF"}
{"time":1704040227,"message":"http: TLS handshake error from 185.233.19.145:39406: EOF"}
{"time":1704040228,"message":"http: TLS handshake error from 185.233.19.145:39428: EOF"}
{"time":1704040228,"message":"http: TLS handshake error from 185.233.19.145:39524: EOF"}
{"time":1704043958,"message":"http: TLS handshake error from 183.136.225.42:55640: EOF"}
{"time":1704043958,"message":"http: TLS handshake error from 183.136.225.42:42830: EOF"}
{"time":1704043959,"message":"http: TLS handshake error from 183.136.225.42:5875: tls: no cipher suite supported by both client and server"}
{"time":1704043959,"message":"http: TLS handshake error from 183.136.225.42:49147: tls: client requested unsupported application protocols ([http/0.9 http/1.0 spdy/1 spdy/2 spdy/3 h2c hq])"}
{"time":1704043960,"message":"http: TLS handshake error from 183.136.225.42:63690: tls: client requested unsupported application protocols ([hq h2c spdy/3 spdy/2 spdy/1 http/1.0 http/0.9])"}
{"time":1704043960,"message":"http: TLS handshake error from 183.136.225.42:37466: tls: client offered only unsupported versions: [302 301]"}
{"time":1704043961,"message":"http: TLS handshake error from 183.136.225.42:3462: EOF"}
{"time":1704043961,"message":"http: TLS handshake error from 183.136.225.42:12900: EOF"}
{"time":1704043962,"message":"http: TLS handshake error from 183.136.225.42:49272: EOF"}
{"time":1704043963,"message":"http: TLS handshake error from 183.136.225.42:59873: EOF"}
{"time":1704044947,"message":"http: TLS handshake error from 80.66.88.215:65340: tls: first record does not look like a TLS handshake"}
{"time":1704046636,"message":"http: TLS handshake error from 117.13.169.92:20792: EOF"}
{"time":1704069953,"message":"http: TLS handshake error from 94.102.61.25:56162: EOF"}
{"time":1704085903,"message":"http: TLS handshake error from 198.235.24.127:55872: tls: client offered only unsupported versions: [302 301]"}
{"time":1704103441,"message":"http: TLS handshake error from 198.199.116.114:43932: tls: first record does not look like a TLS handshake"}
{"time":1704136910,"message":"http: TLS handshake error from 45.227.254.8:65392: tls: first record does not look like a TLS handshake"}
{"time":1704142048,"message":"http: TLS handshake error from 185.170.144.3:64932: tls: first record does not look like a TLS handshake"}
{"time":1704165499,"message":"http: TLS handshake error from 205.210.31.231:50846: tls: client offered only unsupported versions: [302 301]"}
{"time":1704205688,"message":"http: TLS handshake error from 87.236.176.168:44027: EOF"}
{"time":1704205721,"message":"http: TLS handshake error from 87.236.176.170:59003: EOF"}
{"time":1704205754,"message":"http: TLS handshake error from 87.236.176.149:37635: tls: no cipher suite supported by both client and server"}
{"time":1704205787,"message":"http: TLS handshake error from 87.236.176.162:58875: tls: client requested unsupported application protocols ([http/0.9 http/1.0 spdy/1 spdy/2 spdy/3 h2c hq])"}
{"time":1704205820,"message":"http: TLS handshake error from 87.236.176.159:43167: tls: client requested unsupported application protocols ([hq h2c spdy/3 spdy/2 spdy/1 http/1.0 http/0.9])"}
{"time":1704205853,"message":"http: TLS handshake error from 87.236.176.169:33443: tls: client offered only unsupported versions: [302 301]"}
{"time":1704205886,"message":"http: TLS handshake error from 87.236.176.171:32789: EOF"}
{"time":1704205919,"message":"http: TLS handshake error from 87.236.176.176:45919: EOF"}
{"time":1704205952,"message":"http: TLS handshake error from 87.236.176.177:48709: EOF"}
{"time":1704205985,"message":"http: TLS handshake error from 87.236.176.176:48585: EOF"}
{"time":1704226521,"message":"http: TLS handshake error from 104.158.164.191:52286: local error: tls: bad record MAC"}
{"time":1704228486,"message":"http: TLS handshake error from 159.203.192.10:39954: tls: first record does not look like a TLS handshake"}
{"time":1704232286,"message":"http: TLS handshake error from 45.227.254.48:65434: tls: first record does not look like a TLS handshake"}
KES777 commented 5 months ago

I found why disk space was consumed: I need to rebuild my containers. https://superuser.com/a/1824057/431840

It would be nice to have some tool which will allow to find containers which need to be reconfigured to use the current docker configuration.