hcsshim::ImportLayer failed in Win32: The system cannot find the path specified. (0x3) on Docker Desktop for Windows Community edition 2.3.0.3(45519)

[cosmo0920]

I am/we are encountering "hcsshim::ImportLayer failed in Win32: The system cannot find the path specified. (0x3)" on Windows 10 Pro 1909 with Docker Community 19.03.8:

re-exec error: exit status 1: output: time="2020-06-05T14:05:25+09:00" level=error msg="hcsshim::ImportLayer - failed failed in Win32: The system cannot find the path specified. (0x3)" error="hcsshim::ImportLayer - failed failed in Win32: The system cannot find the path specified. (0x3)" importFolderPath="C:\\ProgramData\\Docker\\tmp\\hcs949648159" path="\\\\?\\C:\\ProgramData\\Docker\\windowsfilter\\3a7d2d6226be0f3cca2d17d5b29069918b0f14dd889484ecd8bb208329d382b8"
hcsshim::ImportLayer - failed failed in Win32: The system cannot find the path specified. (0x3)

> docker version
Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b
 Built:             Wed Mar 11 01:23:10 2020
 OS/Arch:           windows/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.24)
  Go version:       go1.12.17
  Git commit:       afacb8b
  Built:            Wed Mar 11 01:37:20 2020
  OS/Arch:          windows/amd64
  Experimental:     false

It happens during msys2 installation of the minimal reproducible Dockerfile:

# Reproducible (0x3) error docker file

FROM mcr.microsoft.com/windows/servercore:ltsc2019

# Do not split this into multiple RUN!
# Docker creates a layer for every RUN-Statement
RUN powershell -Command "Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))"

# Fluentd depends on cool.io whose fat gem is only available for Ruby < 2.5, so need to specify --platform ruby when install Ruby > 2.5 and install msys2 to get dev tools
RUN choco install -y ruby --version 2.6.5.1 --params "'/InstallDir:C:\ruby26'" \

This issue is also happen with using mcr.microsoft.com/windows/servercore:1909 and mcr.microsoft.com/windows/servercore:1903 base images.

We also hit this issue with this Dockerfile.

And I didn't solve this issue with using https://github.com/jhowardmsft/docker-ci-zap and delete all docker related data.

How should we avoid this issue?

[cosmo0920]

I'd read the similar issues:

• https://github.com/microsoft/hcsshim/issues/769
• https://github.com/microsoft/hcsshim/issues/368

But they didn't help us. Our situation is freshly installed and docker-ci-zap execution didn't help....

[schribl]

I am experiencing exactly the same issue as @cosmo0920 when trying to create a docker container that uses choco install msys2. The docker version also matches. It runs on Windows 10 Enterprise 1809 and the base images is mcr.microsoft.com/dotnet/framework/runtime:4.8

[cosmo0920]

This issue also happens on edge channel.

• Docker Desktop Community 2.3.1.0 Windows containers

(added) This issue also happens on Windows 10 Pro 2004 with Docker Desktop for Windows Community edition 2.3.0.3(45519).

[jeffreycoe]

I'm experiencing this exact issue on a Windows Server 2019 (10.0.17763) instance running Docker 19.03.5, too.

[farzonl]

I am also seeing this issue

SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
WORKDIR C:\
COPY Windows/choco.ps1 choco.ps1
RUN .\choco.ps1

and the contents of the choco.ps1 is

Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))

choco install cygwin wget -y

[kempniu]

I am experiencing the same problem with Windows Server 2016 as the host system, Docker 19.03.5 (which is the latest version currently available for that system through DockerMsftProvider), and microsoft/dotnet-framework:3.5-sdk-windowsservercore-ltsc2016 as the base image.

In my case, installing cygwin using choco install triggers a hcsshim::ImportLayer error during docker build. When run in a "live" container (created using docker run), the same command works fine, but subsequently running docker commit for such a container triggers the exact same error. The Dockerfile I am using was working fine back in January 2020 with the same Docker version.

I tried using docker-ci-zap and starting the Docker daemon with --storage-opt size=50G to no avail.

I would be happy to do any testing that could help resolve this issue as it is currently preventing me from provisioning any new CI server.

[QShen3]

The same as @kempniu

Re-install the Docker and re-pull the base image can not solve this problem.

[SergeyMN]

Not sure if this related in some way or not but I have the issue on windows server 2019 with level=error msg="hcsshim::ImportLayer - failed failed in Win32: The system cannot find the path specified. (0x3)" whenever I start sysmon (particularly sysmondrv) until it is unloaded or uninstalled.

[eliasfilipec]

I had this problem and solved it with the following steps: 1 ° I executed the following commands: to clear the images: docker images prune 2 ° I executed the one that cleans everything (Images, Containers, Cache and Networks) docker system prune

[woernsn]

Same problem for me. The problem seems to be cygwin.

Client: Docker Engine - Community Version: 19.03.12 API version: 1.40 Go version: go1.13.10 Git commit: 48a66213fe Built: Mon Jun 22 15:43:18 2020 OS/Arch: windows/amd64 Experimental: false

Server: Docker Engine - Community Engine: Version: 19.03.12 API version: 1.40 (minimum version 1.24) Go version: go1.13.10 Git commit: 48a66213fe Built: Mon Jun 22 15:57:30 2020 OS/Arch: windows/amd64 Experimental: false

[chvjak]

@kempniu @farzonl @schribl I run into same problem trying build an image with Cygwin. It seems that it is hardlinks which Cygwin uses a lot are not handled correctly by Docker. After I got rid of the hardlinks in Cygwin installation I was able to commit the image without problems. To get rid of the hardlinks I have zipped and unzipped Cygwin folder.

[jaunruh]

I am encountering the same issue. I am using the mcr.microsoft.com/dotnet/framework/aspnet:4.8 container in a Dockerfile. When building using this Dockerfile I get the error level=error msg="hcsshim::ImportLayer - failed failed in Win32: The system cannot find the path specified. (0x3). The error is thrown on some step that downloads some larger files. BUT when using powershell inside of the container to download the files, I do not encounter the issue.

[directhex]

For the Cygwin folks, my workaround looks like this:

RUN setx /M CYGWIN "winsymlinks:lnk"

RUN curl -SL --output %TEMP%\cygsetup.exe https://cygwin.com/setup-x86_64.exe `
    && powershell -Command `
        Start-Process %TEMP%\cygsetup.exe -Wait -ArgumentList '--quiet-mode --wait --root c:\cygwin --local-package-dir c:\cygwin --site http://mirrors.kernel.org/sourceware/cygwin/ --packages autoconf' `
    && C:\cygwin\bin\bash.exe -c 'for i in `/bin/find / -xdev -type l`; do j=`/bin/readlink $i`; /bin/rm -f $i; /bin/ln -sf $j $i; done' `
    && powershell -Command `
        Remove-Item -Recurse -Force c:\cygwin\http*

This forces Cygwin to use legacy (.lnk) based links, then recursively goes through Cygwin's directory tree & recreates every symlink (some of which are NTFS junction points), such that there are no junction points on disk when docker tries to commit the RUN step's changes. It works for me locally.

Sadly, I'm getting the same hcsshim error trying to install MSVC build tools on Server Core (but NOT on regular server) images. I checked, and there are no stray junction points created.

[chvjak]

with regards cygwin - exatcly my findings. My approch was before sending cygwin to docker - install it on host, and zip instalation folder (to avoid hardlinks), send zip to docker context and unzip in the container.

Sadly, I'm getting the same hcsshim error trying to install MSVC build tools on Server Core (but NOT on regular server) images. I checked, and there are no stray junction points created.

this dockerfile works for me to install build tools on server core

FROM mcr.microsoft.com/dotnet/framework/sdk:4.8-windowsservercore-ltsc2019

ADD https://aka.ms/vs/16/release/channel C:\TEMP\VisualStudio.chman
ADD https://aka.ms/vs/16/release/vs_buildtools.exe C:\TEMP\vs_buildtools.exe

SHELL ["cmd", "/S", "/C"]

RUN C:\TEMP\vs_buildtools.exe --quiet --wait --norestart --nocache `
    --installPath C:\BuildTools `
    --channelUri C:\Temp\VisualStudio.chman `
    --installChannelUri C:\Temp\VisualStudio.chman `
    --add Microsoft.VisualStudio.Workload.VCTools;includeRecommended `
    --add Microsoft.Component.MSBuild `
    --add Microsoft.VisualStudio.Component.VC.ATL `
    --add Microsoft.VisualStudio.Component.VC.ATLMFC `
|| IF "%ERRORLEVEL%"=="3010" EXIT 0

[lbergnehr]

I've built a cygwin image successfully with the above script in combination with, before installing cygwin, setting the CYGWIN=winsymlinks env var.

[soroush]

I am having exact same issue on Windows Server 20H2. None of the proposed workarounds work for me. This happens when trying to commit a fairly large layer (compile qt with vcpkg) at build time. I've also changed the default container size to 500GB.

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker Application (Docker Inc., v0.8.0)
  cluster: Manage Mirantis Container Cloud clusters (Mirantis Inc., v1.9.0)
  registry: Manage Docker registries (Docker Inc., 0.1.0)

Server:
 Server Version: 20.10.6
 Storage Driver: windowsfilter
  Windows:
 Logging Driver: none
 Plugins:
  Volume: local
  Network: ics internal l2bridge l2tunnel nat null overlay private transparent
  Log: awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog
 Swarm: inactive
 Default Isolation: process
 Kernel Version: 10.0 19042 (19041.1.amd64fre.vb_release.191206-1406)
 Operating System: Windows Server Datacenter Version 2009 (OS Build 19042.508)
 OSType: windows
 Architecture: x86_64
 CPUs: 48
 Total Memory: 62GiB

[Yachtie2020]

I've encountered this issue a number of times recently, like others installing VS Buildtools or any other large program seems to cause it. I was doing a lot of builds, that where failing as I was getting the Dockerfile right and also ctrl+c to kill. This resulted in a number of images (and sometimes containers) labelled NONE (hanging). As per Stack Exchange comments, pruning the containers and images so only my base Windows image remained allowed me to re try and build successfully. I did have the issue a couple times at the end of script creating and I would go back a couple commands in the Dockerfile, add a RUN echo hi to force new cache layers etc, stop at the issue (buildtools) go and remove the RUN echo hi and let it run again. This seem to work too (but I also had far less hanging images from pruning), so at least in my case, I would be suspicious that the cache and/or layers are corrupt or something since clean builds after pruning or forcing some layers to be rebuilt fixed the issue.

FYI I also changed storage-opts to 120GB and ran the build with -m 4GB, but the issue still occurred.

[meirgbinahai]

For me, I had two commands

COPY settings.default settings
RUN something
COPY . .

I changed it to

COPY . .
RUN something

It's stupid, doesn't make sense, and I have no idea why it happens. All I can say is, it doesn't happen when I copy all the build-context files at once.

[wa6smn]

I solved this problem by allocating a 64 GB disk for my VMware Workstation Pro and initializing it as drive F:. In the DockerEngine settings, I provided Docker with a place to do its work with the data-root entry. { "registry-mirrors": [], "insecure-registries": [], "debug": false, "experimental": false, "data-root": "F:\\docker-root" }

[darkvertex]

I got to a point where I had an image that kept dying at build time after a few steps even though I had just about enough free space as the last final image size from the previous build, but it seems a lot of extra space is necessary while building and while running.

Neither hcsshim nor Docker Desktop for Windows gave any comprehensive errors that space was an issue. 🤷‍♂️

As pointed above, I too solved this error by pointing "data-root" to a bigger drive and rebuilding my images.

[JayaRamaRajuGitHub]

I am still facing the issue. I have added the below Daemon.json file. How to troubleshoot it further?

{ "storage-opts":["size=200G"], "data-root": "D:\docker-root" }

[maranov]

I've encountered this with Engine 20.10.9 on Windows Server 2019 while building Boost libraries with VS Build Tools and Windows PowerShell. The data root is set to a directory on a secondary drive that's dedicated to Docker and has plenty of free space. Increasing the storage-opts, pruning or a complete reinstall (including removal of files under Program Files, ProgramData and the data root) didn't help. Both the dockerd process and data root are excluded from Defender scans.

What helped, was adding a wait at the end of the RUN script:

sleep -Seconds 300

Why does that work, I have no idea. There shouldn't be anything interfering with the filesystem.

Paradoxically the VS Build Tools and Chocolatey + CMake installation in one of the previous steps is not a problem.

[gabyx]

@maranov: Facing exeactly the same problem. Did you solve this differently???

[wa6smn]

Docker ran out of disk space in the working directory. I allocated a new disk and set the data-root to the new space. Resolved.

Sent from MailDroid

-----Original Message----- From: "Gabriel Nützi" @.> To: microsoft/hcsshim @.> Cc: Mark Edwards @.>, Comment @.> Sent: Wed, 12 Apr 2023 3:20 Subject: Re: [microsoft/hcsshim] hcsshim::ImportLayer failed in Win32: The system cannot find the path specified. (0x3) on Docker Desktop for Windows Community edition 2.3.0.3(45519) (#835)

@maranov: Facing exeactly the same problem. Did you solve this differently???

-- Reply to this email directly or view it on GitHub: https://github.com/microsoft/hcsshim/issues/835#issuecomment-1504945821 You are receiving this because you commented.

Message ID: @.***>

[mpashaasu]

Run in CMD/PS as admin

docker build -t test:v1 . -m 4GB

with below configs

{ "experimental": false, "storage-opts": [ "size=150GB" ], "data-root": "C:\\docker-root" }

base image is mcr.microsoft.com/dotnet/framework/sdk:4.8-windowsservercore-ltsc2022

during intermediate run I think the size crosses 20GB

Gets below error -

re-exec error: exit status 1: output: hcsshim::ImportLayer failed in Win32: The system cannot find the path specified. (0x3)

Very annoying is there a fix for this?

[wa6smn]

A bigger data-root will probably fix it. Sent from MailDroid Mark C. Edwards 779 Monika Ct Chubbuck, ID 83202

-----Original Message----- From: mpashaasu @.> To: microsoft/hcsshim @.> Cc: Mark Edwards @.>, Comment @.> Sent: Thu, 21 Dec 2023 17:34 Subject: Re: [microsoft/hcsshim] hcsshim::ImportLayer failed in Win32: The system cannot find the path specified. (0x3) on Docker Desktop for Windows Community edition 2.3.0.3(45519) (#835)

Run in CMD/PS as admin

docker build -t test:v1 . -m 4GB

with below configs

{ "experimental": false, "storage-opts": [ "size=150GB" ], "data-root": "C:\docker-root" }

base image is mcr.microsoft.com/dotnet/framework/sdk:4.8-windowsservercore-ltsc2022

during intermediate run I think the size crosses 20GB

Gets below error -

re-exec error: exit status 1: output: hcsshim::ImportLayer failed in Win32: The system cannot find the path specified. (0x3)

Very annoying is there a fix for this?

-- Reply to this email directly or view it on GitHub: https://github.com/microsoft/hcsshim/issues/835#issuecomment-1867087055 You are receiving this because you commented.

Message ID: @.***>

[mpashaasu]

data-root has around 450GB of space on the disk

[wa6smn]

"C:\docker-root" Add a backslash in the JSON value Sent from MailDroid Mark C. Edwards 779 Monika Ct Chubbuck, ID 83202

-----Original Message----- From: mpashaasu @.> To: microsoft/hcsshim @.> Cc: Mark Edwards @.>, Comment @.> Sent: Thu, 21 Dec 2023 17:34 Subject: Re: [microsoft/hcsshim] hcsshim::ImportLayer failed in Win32: The system cannot find the path specified. (0x3) on Docker Desktop for Windows Community edition 2.3.0.3(45519) (#835)

Run in CMD/PS as admin

docker build -t test:v1 . -m 4GB

with below configs

{ "experimental": false, "storage-opts": [ "size=150GB" ], "data-root": "C:\docker-root" }

base image is mcr.microsoft.com/dotnet/framework/sdk:4.8-windowsservercore-ltsc2022

during intermediate run I think the size crosses 20GB

Gets below error -

re-exec error: exit status 1: output: hcsshim::ImportLayer failed in Win32: The system cannot find the path specified. (0x3)

Very annoying is there a fix for this?

-- Reply to this email directly or view it on GitHub: https://github.com/microsoft/hcsshim/issues/835#issuecomment-1867087055 You are receiving this because you commented.

Message ID: @.***>

[mpashaasu]

\\ is already present in my config, the parser will anyway warn you in the docker desktop UI.

[doctorpangloss]

Does COPY --link interact with this issue?

[4ampro]

I'm having a similar issue with the PrepareLayer but it all started with the msg box, "Docker Service is not running". Using the data-root parameter, I pointed my config.json to a drive with space however, the file not found seems to be "c:\bcartifacts.cache:c:\dl".

The system cannot find the path specified. (0x3). ExitCode: 125

FORGOT TO ADD THIS: --volume "c:\bcartifacts.cache:c:\dl"

Please kindly let me know if I should post separately? Any help is appreciated.

More infos available if needed.

win 10 pro docker 4.28.0 (139021) pse as admin new-bccontainer bc dev license v22

Edit: gave it a proper issue of its own: https://github.com/microsoft/hcsshim/issues/2075

[rafabios]

I had the same issue, for me it was due to memory/cpu usage ( I was running an virtual machine as well). When I drop the vm, it builded good.

[arkadR]

We're still randomly facing this issue when building images based servercore:ltsc2022 images on EC2 build agents. None of the workarounds listed in this thread seem to have helped. What we tried:

• Increasing VM CPU and memory
• Storage-opts up to 150GB
• Sleep at the end finishing the problematic build step
• docker system prune and other system cleanups

This error does pop up randomly (about 30% of times) and we really can't figure out what's causing it.