search

docker / for-win

Bug reports for Docker Desktop for Windows
https://www.docker.com/products/docker#/windows
1.86k stars 288 forks source link

kubernetes stop working after one year #13598

Open csckcac opened 1 year ago

csckcac commented 1 year ago

Description

Certificate is expired, no actual solutions except reset the Kubernetes which is a workaround.

Certificate under this location is expired.

%LOCALAPPDATA%\Docker\pki\apiserver.crt %LOCALAPPDATA%\Docker\pki\apiserver-etcd-client.crt %LOCALAPPDATA%\Docker\pki\front-proxy-client.crt

Deleting the pki folder will also reset the kubernetes, thus losing data

There is no such tools like kubeadm to renew certificate or any "docker" guide to solve this problem.

Reproduce

  1. Install Docker Desktop for windows
  2. Enable Kubernetes
  3. Create a namspace using kubectl
  4. Stop docker
  5. Deleting the folder %LOCALAPPDATA%\Docker\pki
  6. Restart docker
  7. Namespace is lost

Expected behavior

No response

docker version

Client:
 Cloud integration: v1.0.31
 Version:           23.0.5
 API version:       1.42
 Go version:        go1.19.8
 Git commit:        bc4487a
 Built:             Wed Apr 26 16:20:14 2023
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Desktop 4.19.0 (106363)
 Engine:
  Version:          23.0.5
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.19.8
  Git commit:       94d3ad6
  Built:            Wed Apr 26 16:17:45 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.20
  GitCommit:        2806fc1057397dbaeefbea0e4e17bddfbd388f38
 runc:
  Version:          1.1.5
  GitCommit:        v1.1.5-0-gf19387a
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.17.3
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.19
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-scan.exe
  scout: Command line tool for Docker Scout (Docker Inc.)
    Version:  v0.10.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 32
  Running: 26
  Paused: 0
  Stopped: 6
 Images: 38
 Server Version: 23.0.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2806fc1057397dbaeefbea0e4e17bddfbd388f38
 runc version: v1.1.5-0-gf19387a
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 5.15.90.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 18
 Total Memory: 17.57GiB
 Name: docker-desktop
 ID: d2bd7448-a19a-4dba-86f5-1f33fa4c526f
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

Diagnostics ID

DFE7A969-EBA4-4489-AB49-A48BADC30F7A/20230713093613

Additional Info

No response

mj3c commented 1 year ago

Just encountered this issue on MacOS as well. Tried deleting the pki directory as that is what is suggested in a few threads online. The certificates get generated and Docker Desktop + K8s starts up fine, but all resources (including PVCs) are gone for whatever reason.

This is a huge problem for anyone running databases or whatever using StatefulSets. Why would all resources get wiped if the pki directory is gone?

ps-rock commented 10 months ago

hi, any update? I am facing this issue too. I found many solutions to rotate the certificates on Internet but seems not work on docker desktop for Windows.

csckcac commented 10 months ago

hi, any update? I am facing this issue too. I found many solutions to rotate the certificates on Internet but seems not work on docker desktop for Windows.

Please check if this works

Prepare nsenter1 (First time only)

git clone https://github.com/justincormack/nsenter1.git cd nscenter1 docker build -t yournsenter1 .

Create a temporary volume (First time only)

docker volume create tmp

Exec into the Docker VM:

docker run -it --rm --privileged --pid=host yournsenter1

Get ContainerD namespaces from VM:

ctr namespaces list

List ContainerD containers:

ctr --namespace services.linuxkit containers list

Exec on Docker services container:

ctr --namespace services.linuxkit tasks exec -t --exec-id 3000 docker /bin/sh

Renew all certs:

kubeadm certs renew all

This folder will get updated with kubeadm:

ls -lha /var/lib/kubeadm/pki/ # updates with kubeadm

While this folder doesn't get updated:

ls -lha /run/config/pki/ # doesn't update with kubeadm

Copy the new cert to volume tmp:

cp -R /var/lib/kubeadm/pki/* /var/lib/docker/volumes/tmp/_data/

Replace the new cert on windows:

copy \\wsl$\docker-desktop-data\data\docker\volumes\tmp\_data\* to %localappdata%\Docker\pki Restart Docker desktop

ps-rock commented 10 months ago

hi, any update? I am facing this issue too. I found many solutions to rotate the certificates on Internet but seems not work on docker desktop for Windows.

Please check if this works

Prepare nsenter1 (First time only)

git clone https://github.com/justincormack/nsenter1.git cd nscenter1 docker build -t yournsenter1 .

Create a temporary volume (First time only)

docker volume create tmp

Exec into the Docker VM:

docker run -it --rm --privileged --pid=host yournsenter1

Get ContainerD namespaces from VM:

ctr namespaces list

List ContainerD containers:

ctr --namespace services.linuxkit containers list

Exec on Docker services container:

ctr --namespace services.linuxkit tasks exec -t --exec-id 3000 docker /bin/sh

Renew all certs:

kubeadm certs renew all

This folder will get updated with kubeadm:

ls -lha /var/lib/kubeadm/pki/ # updates with kubeadm

While this folder doesn't get updated:

ls -lha /run/config/pki/ # doesn't update with kubeadm

Copy the new cert to volume tmp:

cp -R /var/lib/kubeadm/pki/* /var/lib/docker/volumes/tmp/_data/

Replace the new cert on windows:

copy \\wsl$\docker-desktop-data\data\docker\volumes\tmp\_data\* to %localappdata%\Docker\pki Restart Docker desktop

I tried. The certs are updated but then after restart docker desktop, Kubernetes failed to start. I found an error from the log

[com.docker.backend.exe][W] cannot create kubernetes PKI: error creating PKI assets: failed to write or validate certificate "apiserver-etcd-client": certificate apiserver-etcd-client is not signed by corresponding CA. Cluster will need to be reset.

I need to reset the k8s cluster. All pods are gone 😢

CreativeWarlock commented 1 day ago

EDIT: I checked C:\Users\\AppData\Local\Docker\pki and found that all my certificates are exactly 1 year old. I renamed folder 'pki' to '_pki' and restarted Docker. A new pki folder has been created by Docker. Now everything works again. No need to reinstall Kubernetes.

Important: ONLY Remove old '_pki' folder if you're sure everything works!

After recreating the pki folder you could run into the issue of mismatching certificate-authority-data, see more here: https://stackoverflow.com/questions/46234295/kubectl-unable-to-connect-to-server-x509-certificate-signed-by-unknown-authori

Credits go to wp4nuv, https://www.reddit.com/r/docker/comments/t2ssqr/issue_windows_trying_to_start_docker_desktop/