search

docker / for-win

Bug reports for Docker Desktop for Windows
https://www.docker.com/products/docker#/windows
1.84k stars 281 forks source link

All volumes, containers and images are missing after Docker for Windows upgrade (docker-desktop-distro moved to another drive/location) #14126

Closed dotysan closed 3 weeks ago

dotysan commented 3 weeks ago

Description

After upgrading from 4.30.0 (149282) to 4.31.1 (153621), there are no containers, images and volumes visible in the UI.

Reproduce

  1. install the latest update (version 4.31.1)
  2. launch Docker for Windows

Expected behavior

Where'd my stuff go?!?

docker version

Client:
 Version:           26.1.4
 API version:       1.45
 Go version:        go1.21.11
 Git commit:        5650f9b
 Built:             Wed Jun  5 11:29:54 2024
 OS/Arch:           windows/amd64
 Context:           desktop-linux

Server: Docker Desktop 4.31.1 (153621)
 Engine:
  Version:          26.1.4
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.11
  Git commit:       de5c9cf
  Built:            Wed Jun  5 11:29:22 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.33
  GitCommit:        d2d58213f83a351ca8f528a95fbd145f5654e957
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    26.1.4
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.14.1-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.27.1-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.32
    Path:     C:\Program Files\Docker\cli-plugins\docker-debug.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.24
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     C:\Program Files\Docker\cli-plugins\docker-feedback.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.2.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scout: Docker Scout (Docker Inc.)
    Version:  v1.9.3
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 26.1.4
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d2d58213f83a351ca8f528a95fbd145f5654e957
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 5.15.146.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.56GiB
 Name: docker-desktop
 ID: 7080c920-ee3e-4415-b52f-85cc710c2aaa
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=npipe://\\.\pipe\docker_cli
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: daemon is not using the default seccomp profile

Diagnostics ID

1F426F3F-E329-4D5D-A1B5-EE0B70E796AB/20240612013103

Additional Info

I can see my old docker-desktop-data WSL distro is full of stuff. Hopefully all my lost stuff!

andrea-reale commented 3 weeks ago

Hi @dotysan,

thank you for sharing the issue and I am sorry if it caused disruption on your work.

Be reassured that, having your docker-desktop-data distribution around, your data is fully recoverable. I am suggesting how to at the end of this message.

In order to help make sure we properly root cause your issue:

UPDATE:

The following build should address the issue in all the cases where the source of the problem is having moved the docker-desktop-data distribution manually (not through the UI) to a different location:

Docker Desktop Installer 4.31.1 - build 154084

Please, feel free to install it, use it and share any positive or negative feedback about it.

As usual, it can be good practice to take a backup of your existing docker-desktop-data distribution. You can do so by running the following command, which will save a backup in your current folder:

wsl.exe --export docker-desktop-data docker-desktop-data-backup.tar

I am leaving the workaround I previously suggested below struck out for archival reasons. The build above should be preferred to the workaround in all cases.

If different, please take note of the BasePath you see in the output of the previous command. Let's say that for example it is D:\my_custom\path\data. Please, note that it is important that the path ends by \data; if it doesn't please see this comment.

In order to make sure that DockerDesktop will properly recognize your distribution you'll need to update your settings.json file. To do so:

1. Quit DockerDestkop 2. Go to %APPDATA%\Docker (typically, C:\Users\<username>\AppData\Roaming\Docker) and opensettings.json` with your favorite editor. 3. Find an entry called customWslDistroDir or create a new one if that doesn't exist. 4. Enter the path that you have noted down before where your docker-desktop-data distribution resides, without the final data component and using double slashes as your path separator (see example below). 5. Remove the old docker-desktop distribution (note well: docker-desktop, and not docker-desktop-data) with. At the next startup, it will be recreated at the new path you've specified in the settings.json at the previous step.

   wsl.exe --unregister docker-desktop

6. Shut down wsl with wsl.exe --shutdown (note that this will also stop any other WSL distribution you might have) 7. Start Docker Desktop again

* For example, let's pretend your BasePath is D:\my_custom\path\data; your new entry in settings.json should look like (note again the double slashes (\\) and the absence of the data suffix):

      ...
      "customWslDistroDir": "D:\\my_custom\\path",
      ...

Please, let us know if this workaround works - I am happy to help further if it doesn't.

empiricompany commented 3 weeks ago

@andrea-reale i've changed customWslDistroDir i now see my containars

"customWslDistroDir": "G:\\wsl\\Docker",
"dataFolder": "C:\\ProgramData\\DockerDesktop\\vm-data",

but i cannot bind volumes

(HTTP code 400) unexpected - failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/9600bab832d53d5597f1a7f3ea6b3a07e17bd6ae3b71d3507217110581b28b0b" to rootfs at "/etc/mysql/conf.d/devilbox.cnf": mount /run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/9600bab832d53d5597f1a7f3ea6b3a07e17bd6ae3b71d3507217110581b28b0b:/etc/mysql/conf.d/devilbox.cnf (via /proc/self/fd/6), flags: 0x5000: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

this is my config

Hive: HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Lxss

Name                           Property
----                           --------
{09b470d7-5de5-4784-8df3-79c91 State            : 1
a01128f}                       DistributionName : docker-desktop
                               Version          : 2
                               BasePath         : \\?\C:\Users\tony\AppData\Local\Docker\wsl\main
                               Flags            : 15
                               DefaultUid       : 0
{0c2781b3-376f-487e-b7ac-cbdeb State            : 1
2ef197c}                       DistributionName : Ubuntu
                               Version          : 2
                               BasePath         : \\?\G:\wsl\Ubuntu
                               Flags            : 15
                               DefaultUid       : 1000
{c61c3faa-8dfd-4e8d-b380-c4f5b State            : 1
2c0f528}                       DistributionName : docker-desktop-data
                               Version          : 2
                               BasePath         : \\?\G:\wsl\Docker\data
                               Flags            : 15
                               DefaultUid       : 0

i have to change dataFolder with my docker-desktop new location too?

andrea-reale commented 3 weeks ago

thanks @empiricompany. Can you try to:

  1. quit docker desktop
  2. Do wsl --unregister docker-desktop (not docker-desktop-data)
  3. Shut down WSL with wsl --shutdown (note that this will also stop any other WSL distribution you might have)
  4. restart docker desktop

At startup Docker Desktop will re-create the docker-desktop distribution under G:\wsl\Docker\main. It is OK to delete the old one, as it does not hold any state. Let me know if that helps.

If the problem still persists, please share a new diagnostics ID and I'll take a further look.

empiricompany commented 3 weeks ago

@andrea-reale i've quit docker destop then unregister docker-desktop distro

 wsl --unregister docker-desktop
Annullamento della registrazione.
Operazione completata.
PS C:\Program Files\Docker\Docker\resources> Get-ChildItem "HKCU:\Software\Microsoft\Windows\CurrentVersion\Lxss" -Recurse

    Hive: HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Lxss

Name                           Property
----                           --------
{0c2781b3-376f-487e-b7ac-cbdeb State            : 1
2ef197c}                       DistributionName : Ubuntu
                               Version          : 2
                               BasePath         : \\?\G:\wsl\Ubuntu
                               Flags            : 15
                               DefaultUid       : 1000
{c61c3faa-8dfd-4e8d-b380-c4f5b State            : 1
2c0f528}                       DistributionName : docker-desktop-data
                               Version          : 2
                               BasePath         : \\?\G:\wsl\Docker\data
                               Flags            : 15
                               DefaultUid       : 0

when i start Docker Desktop new docker-desktop image is being recreated in main 

 Get-ChildItem "HKCU:\Software\Microsoft\Windows\CurrentVersion\Lxss" -Recurse

    Hive: HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Lxss

Name                           Property
----                           --------
{0c2781b3-376f-487e-b7ac-cbdeb State            : 1
2ef197c}                       DistributionName : Ubuntu
                               Version          : 2
                               BasePath         : \\?\G:\wsl\Ubuntu
                               Flags            : 15
                               DefaultUid       : 1000
{5d74a824-7471-4e85-905e-7137f State            : 1
0a1e3b4}                       DistributionName : docker-desktop
                               Version          : 2
                               BasePath         : \\?\G:\wsl\Docker\main
                               Flags            : 15
                               DefaultUid       : 0
{c61c3faa-8dfd-4e8d-b380-c4f5b State            : 1
2c0f528}                       DistributionName : docker-desktop-data
                               Version          : 2
                               BasePath         : \\?\G:\wsl\Docker\data
                               Flags            : 15
                               DefaultUid       : 0

but still have the same error, i cannot start mysql image (HTTP code 400) unexpected - failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/9600bab832d53d5597f1a7f3ea6b3a07e17bd6ae3b71d3507217110581b28b0b" to rootfs at "/etc/mysql/conf.d/devilbox.cnf": mount /run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/9600bab832d53d5597f1a7f3ea6b3a07e17bd6ae3b71d3507217110581b28b0b:/etc/mysql/conf.d/devilbox.cnf (via /proc/self/fd/6), flags: 0x5000: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

this is my Diagnostic ID: 7668749B-AB75-45D5-B980-5DD616348E49/20240612091352

thanks for support

empiricompany commented 3 weeks ago

i see my volumes docker_volumes

the last one "devilbox_devilbox-mariadb-10.5" show 0 size but is not empty, i can navigate into them this is used by my container mysql that return errrors

i see strange things, in my images tab could see images then i refreshed and see it correct images2

maybe the same thing could happen with volumes? seems is not updated

andrea-reale commented 3 weeks ago

@empiricompany I think this may be a different issue. I opened a new discussion here to dive deeper.

schmittjoh commented 3 weeks ago

@andrea-reale thanks for the work-around steps - is it important to have the data/ suffix, or what to do when the BasePath does not have it? i.e. something like this:

{f0f362f1-9d97-4206-a300-1f422 State            : 1
20fd2e5}                       DistributionName : docker-desktop-data
                               Version          : 2
                               BasePath         : \\?\D:\Docker
                               Flags            : 15
                               DefaultUid       : 0
andrea-reale commented 3 weeks ago

EDIT: deprecated, please, just install the development build in the comment below

Hi @schmittjoh, and thanks for highlighting this case.

Yes, the \data suffix in the path is important. Docker Desktop assumes to find one when looking for the distribution.

If you don't have it, you can always:

1. export the content of the distribution to your current directory. Make sure there are no errors.~

> wsl.exe --export docker-desktop-data docker-desktop-data-archive.tar

2. create the data directory at your favorite location. As an example we'll use D:\Docker\data.

> mkdir "D:\Docker\data"

3. unregister the docker-desktop-data distribution, so that you can import it again in the next step

> wsl.exe --unregister docker-desktop-data

4. import the docker-desktop-data distribution again at the new location

> wsl.exe --import docker-desktop-data "D:\Docker\data" docker-desktop-data-archive.tar

Let me know how that goes.

andrea-reale commented 3 weeks ago

The following development build should solve the issue and bring the data back for those who have moved their docker-desktop-data distribution to another location. It does not require any change in the existing settings or whatsoever:

Docker Desktop Installer - 154084 dev build

Please, try this out and feel free to report any issue.

dotysan commented 3 weeks ago

Thank you @andrea-reale. Yes indeed this system has a small C: and large D: so many things get moved to the larger volume.

I was able to successfully perform your suggested workaround of editing settings.json with a new customWslDistroDir. And all my stuff came back!

Might I suggest you un-strike this part of your comment above?

Before starting, I recommend to take a backup of your existing docker-desktop-data distribution. You can do so by running the following command, which will save a backup in your current folder:

wsl.exe --export docker-desktop-data docker-desktop-data-backup.tar
crtl commented 3 weeks ago

The following development build should solve the issue and bring the data back for those who have moved their docker-desktop-data distribution to another location. It does not require any change in the existing settings or whatsoever:

Docker Desktop Installer - 154084 dev build

Please, try this out and feel free to report any issue.

This solved it. Ive installed it, restarted computer and start docker. All containers are now visible again after disabliing "Only show running containers"

actinetwork commented 3 weeks ago

I had the same problem after upgrading to 4.31.1. All containers were missing. Indeed, "docker-desktop-data" is on another disk. The latest build (154084) works and everything is back in order...

Alexr03 commented 2 weeks ago

The following development build should solve the issue and bring the data back for those who have moved their docker-desktop-data distribution to another location. It does not require any change in the existing settings or whatsoever:

Docker Desktop Installer - 154084 dev build

Please, try this out and feel free to report any issue.

Thanks for this. My data is now back and Docker working as expected. My question now is how do we migrate over to the new data mode? I see your walkthrough here https://github.com/docker/for-win/issues/14118#issuecomment-2160071111 However, the ext4.vhdx is too big for my C: drive so cannot move it into AppData so I must have it in a different drive which I currently do.

andrea-reale commented 2 weeks ago

Thanks all for confirming the fix.

@Alexr03 there is no need to manually migrate right now, as Docker Desktop will keep working exactly the same with the old and new architecture. In future versions, we plan that Docker Desktop will automatically perform the migration for you taking into account all your current setting, including a drive location different than the default one.

ExchangesInc commented 1 week ago

The following development build should solve the issue and bring the data back for those who have moved their docker-desktop-data distribution to another location. It does not require any change in the existing settings or whatsoever:

Docker Desktop Installer - 154084 dev build

Please, try this out and feel free to report any issue.

@andrea-reale I also have docker-desktop-data on another logical Drive 'D:\Docker\wsl\data' . Can I safe delete %LocalAppData%\Docker\wsl\disk\docker_data.vhdx after installing Build 154084? P.S. All my volumes, containers and images come back.

andrea-reale commented 1 week ago

@ExchangesInc yes, you can. Thank you

xlavs commented 1 week ago

I am using 4.31.1.153621 and I am loosing data in the volumes that I have for my application that I am trying to manage with docker-compose.

My docker-compose is like this:

version: "3.7"

services:
  nginx:
    build: docker/nginx
    restart: always
    hostname: "nginx"
    volumes:
      - tng-app:/var/www/html
    ports:
      - "8080:80"
    depends_on:
      - php
    networks:
      - tng-network

  php:
    build: docker/php
    hostname: "php"
    volumes:
      - tng-app:/var/www/html
      - ./data-backups:/var/www/html/backups
      - ./data-gedcom:/var/www/html/gedcom

    depends_on:
      - db
    networks:
      - tng-network

  db:
    image: mariadb:lts
    container_name: db
    hostname: "db"
    environment:
      MARIADB_USER: ${DB_USER}
      MARIADB_PASSWORD: ${DB_PASSWORD}
      MARIADB_DATABASE: ${DB_NAME}
      MARIADB_ROOT_PASSWORD: ${DB_ROOT_PASSWORD}
    ports:
      - "3306:3306"
    restart: always
    volumes:
      - tng-db:/var/lib/mysql
    networks:
      - tng-network

networks:
  tng-network:
    driver: bridge
    ipam:
     config:
       - subnet: 172.28.0.0/16
         gateway: 172.28.0.1

volumes:
  tng-app:
  tng-db:

I can't tell that I loose data every time, but occasionally I boot up with docker compose up only to realize that tng-app volume is completely empty. I have lost the installation twice already.

In docker-desktop I can see multiple volumes:

tng-app - with seems as it is still having the data that I restored two days ago but Docker claims it is not in use actually tng-genealogy_tng-app - a new volume that is now empty for some reason tng-genealogy_tng-db - a volume that seems like it is still having my database data, but I am not completely sure.

How did I end up loosing data just now:

E:\DevelopmentE\docker\tng-genealogy>docker exec -ti tng-genealogy-php-1 bash
php:/var/www/html# ls
backups         custom-backups  gedcom
php:/var/www/html#

It seems to me that there is some change in behavior on how this version of Docker Desktop maintains the volumes and when it is using defaults like in this example:

volumes:
  tng-app:
  tng-db:

... he erases and recreates volumes every time You start up the application.

Is there any way to force persistence in this scenario?

Update:

I updated to the version: Docker Desktop Installer - 154084 dev build

I will try to recreate the application again, using volumes, and try again. It seems like the only safe way to use this, current version is using volumes as links to the local Windows folders.

andrea-reale commented 1 week ago

Hi @xlavs ,

sorry about this behaviour. I think that what you are describing is independent on the issue discussed in this ticket (likely a separate problem).

Would you mind opening a new ticket with the information you have provided here, please, so that we can take a look into it?

In addition to the information you have already given it would be great if:

Thank you.

xlavs commented 1 week ago
  • even more ideally, if you could describe a simplified setup that we could replicate to replicate and address the issue.

I am now running a 154084 dev build. I will test my application there now and if that happens I will try to make a simpler setup to replicate this behavior. Once I will have the situation to replicate I will open a new ticket,

Thank You!

rs-pontes commented 1 week ago

The following development build should solve the issue and bring the data back for those who have moved their docker-desktop-data distribution to another location. It does not require any change in the existing settings or whatsoever: Docker Desktop Installer - 154084 dev build Please, try this out and feel free to report any issue.

This solved it. Ive installed it, restarted computer and start docker. All containers are now visible again after disabliing "Only show running containers"

Of all the awnsers, in my case,this was the one that resolved the problem! Thanks a lot!