search

microsoft / hcsshim

Windows - Host Compute Service Shim
MIT License
577 stars 259 forks source link

The hcsshim on Windows 10 1903 always fails to build Docker image #624

Closed eriawan closed 5 years ago

eriawan commented 5 years ago

After I have updated my machine from Windows 10 1809 to 1903, I always experiencing errors when doing image build.

The error always shows that the problem is caused by hcsshim:

F:\dockerimages>docker build --rm -f Dockerfile -t dev/testimage .
Sending build context to Docker daemon  330.6MB
Step 1/11 : FROM mcr.microsoft.com/dotnet/framework/runtime:4.7.2-windowsservercore-ltsc2016
 ---> d529d6d68c79
Step 2/11 : SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
 ---> Running in 3a7c4fcc808c
hcsshim::PrepareLayer failed in Win32: Incorrect function. (0x1) path=C:\ProgramData\Docker\windowsfilter\3a7c4fcc808c8794606111ba9fb67342560aa5bfa40a888d564e52871fbd8840

image

This issue is always reproducible on Windows 10 1903, and on previous version of Windows 10 it doesn't happen. I have tested on other machines that have WIndows 10 1803 and Windows 10 1809, and using the same Docker for Windows Desktop (2.0.0.3), the image build is always finished/done successfully.

This has been reported before on Docker for Windows: docker/for-win#3884 especially this comment and the next comments after: https://github.com/docker/for-win/issues/3884#issuecomment-498939672

eriawan commented 5 years ago

@jhowardmsft @jterry75 @jstarks

Is this repo active? This issue has been 6 days without any reply. Please confirm...

jbolduan commented 5 years ago

I'm also getting this error since upgrading my machine to 1903 when trying to build any dockerfile (that I've tried so far).

It seems to get to the first config step and then fail.

gatsby0121 commented 5 years ago

Was anyone able to fix this? I'm getting it as well.

thaJeztah commented 5 years ago

/cc @ddebroy

xuelix commented 5 years ago

Just updated my Win 10 to 1903. Seeing the same issue.

moo-ecxio commented 5 years ago

got the same issue on win10 release 1903 when trying to copy content from a container to the file system: docker cp [containerName]:[containerPath] [hostFilesystem] resulting in Error response from daemon: hcsshim::PrepareLayer failed in Win32: Incorrect function. (0x1) path=C:\ProgramData\Docker\windowsfilter\6a7019f5530c15951e6a7441dba27ed40a87c560b4ab5befe324aa26956c498d

Docker version 18.09.2, build 6247962

eriawan commented 5 years ago

@jterry75 @jhowardmsft

Guys, please reply/response soon. Is this repo actively monitored for issues too? Is it only active for code changes (commits) only? Because I have seen active commits, but on issues are not having responses at all.

Please confirm, as this issue is now aged more than 3 weeks.... 😞

As for me, I think I'm going to rollback my Windows to be 1809 and I won't update to 1903 to build any docker container on Windows 10 1903 until this issue is solved, and this is quite a showstopper for me and my teams, including other folks that have experienced this issue too.

giniyat202 commented 5 years ago

After debugging for a few hours I stumbled upon a call to AttachVirtualDisk which is used to mount the images.
Some undocumented flags were passed to the function during one of the calls and I found out that removing the undocumented flags resolves the issue for some reason.
I have created a simple program to patch the running dockerd daemon in memory and remove the undocumented flags. It has been tested on 10.0.18362.175, however, I cannot guarantee that it would work on other builds.
USE AT YOUR OWN RISK

antonkhelou commented 5 years ago

After upgrading to 1903, I couldn't build anything. The only thing that fixed it was removing the storage-ops.

thaJeztah commented 5 years ago

@antonkhelou that's storage-ops in the daemon configuration? What storage-ops did you have configured?

thaJeztah commented 5 years ago

Oh, I see a comment in https://github.com/moby/moby/issues/36831#issuecomment-498612392

If you have a different storage location configured, could you try if things work if you disable Windows defender, or exclude the location? https://support.microsoft.com/en-us/help/4028485/windows-10-add-an-exclusion-to-windows-security

giniyat202 commented 5 years ago

I'm using the default storage location, and not using storage-opts Changing the storage location, excluding the storage location and disabling Windows defender won't solve the problem in my case. EDIT: using storage-opts results in a slightly different error Error response from daemon: hcsshim::PrepareLayer failed in Win32: The parameter is incorrect. (0x57) when using the hack in my earlier comment. when not using the hack, the error is not changed.

lowenna commented 5 years ago

This is not an hcsshim issue. I'm closing this. See https://github.com/docker/for-win/issues/3884#issuecomment-510973455

lyling commented 5 years ago

After debugging for a few hours I stumbled upon a call to AttachVirtualDisk which is used to mount the images. Some undocumented flags were passed to the function during one of the calls and I found out that removing the undocumented flags resolves the issue for some reason. I have created a simple program to patch the running dockerd daemon in memory and remove the undocumented flags. It has been tested on 10.0.18362.175, however, I cannot guarantee that it would work on other builds. USE AT YOUR OWN RISK

how to use it?

giniyat202 commented 5 years ago

After debugging for a few hours I stumbled upon a call to AttachVirtualDisk which is used to mount the images. Some undocumented flags were passed to the function during one of the calls and I found out that removing the undocumented flags resolves the issue for some reason. I have created a simple program to patch the running dockerd daemon in memory and remove the undocumented flags. It has been tested on 10.0.18362.175, however, I cannot guarantee that it would work on other builds. USE AT YOUR OWN RISK

how to use it?

First you should compile the program. I created a c++ console application on Visual Studio 2015 but generally any c compiler that targets Win32 should work.
After that, you can use an elevated powershell to run the following command:
\path\to\generated\executable.exe p (ps dockerd).id. You can also revert the changes by running \path\to\generated\executable.exe u (ps dockerd).id.

lyling commented 5 years ago

After debugging for a few hours I stumbled upon a call to AttachVirtualDisk which is used to mount the images. Some undocumented flags were passed to the function during one of the calls and I found out that removing the undocumented flags resolves the issue for some reason. I have created a simple program to patch the running dockerd daemon in memory and remove the undocumented flags. It has been tested on 10.0.18362.175, however, I cannot guarantee that it would work on other builds. USE AT YOUR OWN RISK

how to use it?

First you should compile the program. I created a c++ console application on Visual Studio 2015 but generally any c compiler that targets Win32 should work. After that, you can use an elevated powershell to run the following command: \path\to\generated\executable.exe p (ps dockerd).id. You can also revert the changes by running \path\to\generated\executable.exe u (ps dockerd).id.

@giniyat202 It cant work again. it works follow your workaround a few days ago. but after i do some reset docker desktop, update windows. i found i cant work. and i can`t track what different.

should i compile it again after windows update ? now it is windows 10 1903(18362.418)

giniyat202 commented 5 years ago

After debugging for a few hours I stumbled upon a call to AttachVirtualDisk which is used to mount the images. Some undocumented flags were passed to the function during one of the calls and I found out that removing the undocumented flags resolves the issue for some reason. I have created a simple program to patch the running dockerd daemon in memory and remove the undocumented flags. It has been tested on 10.0.18362.175, however, I cannot guarantee that it would work on other builds. USE AT YOUR OWN RISK

how to use it?

First you should compile the program. I created a c++ console application on Visual Studio 2015 but generally any c compiler that targets Win32 should work. After that, you can use an elevated powershell to run the following command: \path\to\generated\executable.exe p (ps dockerd).id. You can also revert the changes by running \path\to\generated\executable.exe u (ps dockerd).id.

@giniyat202 It cant work again. it works follow your workaround a few days ago. but after i do some reset docker desktop, update windows. i found i cant work. and i can`t track what different.

should i compile it again after windows update ? now it is windows 10 1903(18362.418)

@lyling I have just updated it for version 10.0.18362.175 and it should work as usual once you recompile it

thaJeztah commented 5 years ago

Looks like regression is also tracked in https://github.com/microsoft/hcsshim/issues/708, and a workaround has been merged in https://github.com/microsoft/hcsshim/pull/718

Work is in progress to get that workaround merged into the Docker codebase (see https://github.com/moby/moby/pull/40128)

lyling commented 5 years ago

@giniyat202 you have deleted the simple program ????

giniyat202 commented 5 years ago

@giniyat202 you have deleted the simple program ????

no, its still available

lyling commented 5 years ago

@giniyat202 you have deleted the simple program ????

no, its still available

@giniyat202 , i get the code and i see you do an update for 418. it works great. thank you so much.

dayue-liang commented 5 years ago

hello,

I compiled simple-program using MinGW-w64. I executed a program but return error.

C:\docker\fix>dockerd-patch.exe p 8096 CreateToolhelp32Snapshot failed What is wrong ??

lyling commented 5 years ago

run as administrator

From: dayue-liang notifications@github.com Sent: Saturday, November 2, 2019 8:59:37 PM To: microsoft/hcsshim hcsshim@noreply.github.com Cc: Nicolas Lau nicolaslau@outlook.com; Mention mention@noreply.github.com Subject: Re: [microsoft/hcsshim] The hcsshim on Windows 10 1903 always fails to build Docker image (#624)

I compiled simple-program using MinGW-w64. I executed a program but return error.

C:\docker\fix>dockerd-patch.exe p 8096 CreateToolhelp32Snapshot failed What is wrong ??

― You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fhcsshim%2Fissues%2F624%3Femail_source%3Dnotifications%26email_token%3DAADWQ5IZAYYB5UX2G4PMK43QRV2TTA5CNFSM4HW2SAA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC43LAQ%23issuecomment-549041538&data=02%7C01%7C%7Cf31971694af44ea4969a08d75f948750%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637082963835481233&sdata=CPPxkGQp%2BXiousZ6ARgtVc38phnuFi9COflRUmh91JE%3D&reserved=0, or unsubscribehttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAADWQ5IAV3TXYK3KA7VNIKDQRV2TTANCNFSM4HW2SAAQ&data=02%7C01%7C%7Cf31971694af44ea4969a08d75f948750%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637082963835492887&sdata=rgOt%2BhIaOpKkExrSq2vs6cqgQxnOc5nVR20tuJhikpc%3D&reserved=0.

dayue-liang commented 5 years ago

Thanks! > lyling

I used command prompt by administrator and failed. And then I used powershell by administrator and succeeded.

rioam2 commented 5 years ago

@giniyat202 Any insight on how I can migrate this patch to work for 18362.449? I've noticed that you updated the check_integrity_bytes in your previous update.

giniyat202 commented 5 years ago

@giniyat202 Any insight on how I can migrate this patch to work for 18362.449? I've noticed that you updated the check_integrity_bytes in your previous update.

I use x64dbg for that. The exact steps required to update this patch are as follows:

  1. Load vmcompute.dll inside x64dbg. You can find this file under %systemroot%\system32.
  2. Select Symbols tab, right click vmcompute.dll module and select Download symbols for this module . You also need to make note of Base address next to the module.
  3. Type ?MountVhd in the search bar below the right pane then double click the only result
  4. Hit Ctrl-r
  5. At this point it is required to find the call that passes 376 on edx, In my case it was the 4th reference.
  6. Assuming nothing has changed in the function body, check_integrity_offset should be calculated as by subtracting Base obtained in step 2 from the address of xor r8d, r8d instruction and check_integrity_bytes are the bytes starting from this instruction all the way to the ja instuction slightly below the call

I will update my code once I receive the update (Windows update didn't offer it for some reason)

rioam2 commented 5 years ago

@giniyat202 Thank you! Turns out that your script still works with 18362.449, and my issue was just user error. Assuming Microsoft doesn't fix anytime soon, these instructions will be very handy for the next breaking update 👍

El-Gor-do commented 5 years ago

@giniyat202 I'm trying to update the patch for 18363.449 but I'm having problems following your instructions. After opening vmcompute.dll, the Symbols tab doesn't show vmcompute.dll. The left panel shows dllloader64_3158.exe, apphelp.dll, msvcp_win.dll, ucrtbase.dll, gdi32full.dll, kernelbase.dll, win32u.dll, gdi32.dll, user32.dll, kernel32.dll and ntdll.dll. How do I download symbols for vmcompute.dll?

giniyat202 commented 5 years ago

@giniyat202 I'm trying to update the patch for 18363.449 but I'm having problems following your instructions. After opening vmcompute.dll, the Symbols tab doesn't show vmcompute.dll. The left panel shows dllloader64_3158.exe, apphelp.dll, msvcp_win.dll, ucrtbase.dll, gdi32full.dll, kernelbase.dll, win32u.dll, gdi32.dll, user32.dll, kernel32.dll and ntdll.dll. How do I download symbols for vmcompute.dll?

You might have set your debugger to stop at system breakpoint. Try to resume the program (F9) and see if it becomes visible. also @rioam2 mentions that no modification is required for 18363.449 so you might want to try the current version first.

El-Gor-do commented 5 years ago

@giniyat202 thanks for your help, I managed to figure it out now. It turns out that the check_integrity_bytes and check_integrity_offset for 18363.449 is identical to 18362.418. Just to expand on your instructions for those of us who are not x64dbg experts,

2 - If vmcompute.dll doesn't appear in the modules list then keep pressing F9 until vmcompute.dll appears.

5 - In the References tab, double click on each reference to jump to the call in the CPU tab, then scroll up 3 lines and check if the preceding 3 instructions are "xor r8d,r8d", "mov edx,376" and "mov rcx,qword ptr ss:[rbp-58]"

El-Gor-do commented 5 years ago

After all this, it turns out that I haven't been reading the Docker error message correctly. My hcsshim error is 0x57 rather than 0x1 so this patch isn't applicable to my case. Doh!

giniyat202 commented 5 years ago

After all this, it turns out that I haven't been reading the Docker error message correctly. My hcsshim error is x57 rather than 0x1 so this patch isn't applicable to my case. Doh!

do you have storage-ops in your config? If so try to remove it and see if it works

El-Gor-do commented 5 years ago

I didn't have storage-opts set at all in my daemon.json. I also tried with "storage-opts": [ "size=20479M" ] but still no luck so I guess I just have to wait for Microsoft to fix the bug.

bplasmeijer commented 5 years ago

After debugging for a few hours I stumbled upon a call to AttachVirtualDisk which is used to mount the images. Some undocumented flags were passed to the function during one of the calls and I found out that removing the undocumented flags resolves the issue for some reason. I have created a simple program to patch the running dockerd daemon in memory and remove the undocumented flags. It has been tested on 10.0.18362.175, however, I cannot guarantee that it would work on other builds. USE AT YOUR OWN RISK

how to use it?

First you should compile the program. I created a c++ console application on Visual Studio 2015 but generally any c compiler that targets Win32 should work. After that, you can use an elevated powershell to run the following command: \path\to\generated\executable.exe p (ps dockerd).id. You can also revert the changes by running \path\to\generated\executable.exe u (ps dockerd).id.

Using the patch with storage-opt NOT working;

I did try the patch but from error code hcsshim::PrepareLayer - failed failed in Win32: The parameter is incorrect. (0x01) to hcsshim::PrepareLayer - failed failed in Win32: The parameter is incorrect. (0x57)

without the storage-opt setting, docker build works fine!

(https://github.com/imphasing/dockerPatcher)

@giniyat202 windows version 1903 (18362.418)

mitchcapper commented 5 years ago

0x57 is well documented in: https://github.com/microsoft/hcsshim/issues/708 has been patched in hcsshim with a work around already for windows 1903. The work around being merged into docker can be found in terms of progress at: https://github.com/moby/moby/pull/40128 once merged in a new release will have to be made. 0x01 error has also already been patched and merged: https://github.com/moby/moby/pull/39679 you will just need a binary newer than that for that fix (or wait for the backport release https://github.com/docker/engine/pull/314 binaries).

eriawan commented 5 years ago

Hi guys!

After ensuring these steps:

  1. uninstalling Docker, deleting Docker data,
  2. deleting/uninstalling online storage desktop (such as Box, DropBox)
  3. having fresh install of Docker Desktop 2.1.0.4 with ensuring that I have no storage-opts in the Docker configuration,

Now I have successful Docker build!

I have this configuration:

Here's my Docker about box:

image

I have tried to apply the suggested patch from @giniyat202 , but the patch didn't work for me.

NOTES:

  1. Uninstall all of the online drive storage that permeates as having virtual file system, such as Box, Dropbox on your machine that is used to build Docker image.
  2. The only one online drive storage that has nothing to do with this issue is Microsoft OneDrive. (don't ask me why, I really don't know 😞 )
  3. Apparently before I have Windows 10 1903 updates of build 18362.449, this issue still happened. It seems something from Microsoft has fixed this issue (on Windows 10 side).
  4. I have no problem running Docker build on both Windows 10 builds of 18362,449 and 18362.476, as long as I have at least Docker Desktop 2.1.0.4 (detail in the about box above).

Feel free to tell me if my suggestion doesn't work for some or all of you.

El-Gor-do commented 4 years ago

I've just updated Windows to 18363.476 and I am now able to build Windows images without setting "storage-opts": [ "size=20G" ]

Edit... actually, no. "docker-compose build" still fails with 0x57 but "docker build ." is able to build a Windows image. So my workaround for "docker-compose build" is "docker build -t ." to build my Windows image first, then run "docker-compose build" which will successfully use docker-compose.yml to create my container.

bplasmeijer commented 4 years ago

deleting/uninstalling online storage desktop (such as Box, DropBox) will solve the problem @El-Gor-do

El-Gor-do commented 4 years ago

@bplasmeijer I don't use Box or DropBox but I found that uninstalling OneDrive (which comes preinstalled with Windows 10) fixed the 0x57 error when using docker-compose.

El-Gor-do commented 4 years ago

Looks like I spoke too soon, After uninstalling OneDrive then I was able to docker-compose but after rebooting, 0x57 occurs again despite no longer having OneDrive installed.

wawa0210 commented 4 years ago

After upgrading to 1909 and the docker version is 19.03.5 and the same problem when i run image with storage-opt param,such as

image

Any one have the same problems?

jseixas01 commented 4 years ago

Hello, Always the same problem: "hcsshim::PrepareLayer - failed failed in win32 : Incorrect function. (0x1)"

Docker Engine 19.03.5 Docker Desktop 2.1.0.5 Windows 10 1903 build 18362.535

Configuration file: { "registry-mirrors": [], "insecure-registries": [], "debug": true, "experimental": false, ], "data-root": "E:\Docker" }

I have always the same error with or without "storage-opts" and also when using default location. I am nw on Docker am I need urgently to solve this issue.

Can You help me ?

Thanks.

João

DDieterich commented 4 years ago

Windows just did an update and laid down a new copy of this driver.

  1. Rename the C:\Windows\System32\drivers\cbfs6.sys as something else (or delete it).
  2. Remove the container.
  3. Reboot.

See: https://github.com/docker/for-win/issues/3884#issuecomment-533952921

kaidokert commented 4 years ago

Running on Windows Version 10.0.18362, patch provided by @giniyat202 worked perfectly. Thanks a lot ! Also thanks to @imphasing for putting in a VS solution

Bhodges94 commented 3 years ago

Hi, I tried following the steps above and got as far as compiling this C++ program and then it threw some check integrity error and followed @giniyat202 instructions on x64dbg but when you get to the last step it doesn't actually say what to do with the information.

I'm on Windows 10 20H2, have run windows update and have the latest docker (4.1.1)

The error I get seems the same. I don't know what to do with this check integrity error from the C++ app so bit lost what i can do now.

Sending build context to Docker daemon 5.632kB Step 1/2 : FROM mcr.microsoft.com/windows/nanoserver:1809 ---> 1dba00abbe16 Step 2/2 : USER ContainerAdministrator ---> Running in 7da904ee3667 hcsshim::PrepareLayer - failed failed in Win32: Incorrect function. (0x1)

thanks *update, I have upgraded to the latest windows 10 version (21H1) and still same issue I've just uninstalled box as per @El-Gor-do suggestion and restarted and it now works

xakod commented 2 years ago

Problem still here

docker pull mcr.microsoft.com/windows/nanoserver:1909
1909: Pulling from windows/nanoserver
fe8c39bf0334: Extracting [==================================================>]  102.1MB/102.1MB
failed to register layer: re-exec error: exit status 1: output: hcsshim::ProcessBaseLayer \\?\C:\ProgramData\Docker\windowsfilter\c93bfe644648f253e1aedf8d036ef5ba16436fdb59097fe5c97eba860b6dbe59: The parameter is incorrect.
docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Build with BuildKit (Docker Inc., v0.6.3)
  compose: Docker Compose (Docker Inc., v2.1.1)
  scan: Docker Scan (Docker Inc., 0.9.0)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 20.10.10
 Storage Driver: windowsfilter
  Windows:
 Logging Driver: json-file
 Plugins:
  Volume: local
  Network: ics internal l2bridge l2tunnel nat null overlay private transparent
  Log: awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog
 Swarm: inactive
 Default Isolation: hyperv
 Kernel Version: 10.0 19043 (19041.1.amd64fre.vb_release.191206-1406)
 Operating System: Windows 10 Pro Version 2009 (OS Build 19043.1348)
 OSType: windows
 Architecture: x86_64
 CPUs: 8
 Total Memory: 7.881GiB
 Name: CIBuilder
 ID: ***
 Docker Root Dir: C:\ProgramData\Docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

Is there any fix? dockerd patcher cannot work because it doesn't find vmcompute.dll.

Antonin750 commented 2 years ago

Hi, The patch works fine for Windows 11 with a new version of vmcompute.dll. Obviously you need to correctly find the new edx position. Be careful the check_integrity_bytes changed too.

I don't know why that doesn't work with the 376 value and that must be replace by another one but great job