search

microsoft / dotnet-framework-docker

The repo for the official docker images for .NET Framework on Windows Server Core.
https://hub.docker.com/_/microsoft-dotnet-framework
MIT License
708 stars 335 forks source link

Image stopped working after latest update #521

Closed soorajpudiyadath closed 4 years ago

soorajpudiyadath commented 4 years ago

Steps to reproduce the issue

  1. Use image mcr.microsoft.com/dotnet/framework/sdk:4.8 in AWS Pipeline 2.Use the build command - MSBuild.exe .*.csproj -t:clean,restore,build

Expected behavior

build

Actual behavior

Command did not exit successfully MSBuild.exe *** exit status 3221225785

Additional information (e.g. issue happens only occasionally)

This started happening on 2/21/2020 , docker hub says the image the was updated an hour ago

Output of docker version

(paste your output here)

Output of docker info

(paste your output here)
mthalman commented 4 years ago

The images have not been updated in the last hour. The "Last Modified" column for the SDK images says 2/19/2020.

Regardless, can you confirm whether you're running into this Windows host/container compatibility issue? That was introduced in the latest update. To verify the Windows version of your container, you can run:

docker run --rm mcr.microsoft.com/dotnet/framework/sdk:4.8 cmd /C ver
soorajpudiyadath commented 4 years ago

while i do this, is there away we can use the previous image ?

mthalman commented 4 years ago

Yes, you can use a timestamp tag for that but it's not a multi-arch tag so you'll need to know the version of your Docker host machine. The timestamp you'll want is 20200114, so the tag would look like this: mcr.microsoft.com/dotnet/framework/sdk:4.8-20200114-windowsservercore-<version> where <version> is one of the following Windows versions that matches your host: ltsc2016, ltsc2019, 1903, 1909.

bbrummer commented 4 years ago

Same here, msbuild.exe apparently does not exist in the latest image? At least for Server 2016.

This trashed our AWS CodeBuild pipelines with the same "exit status 3221225785" error and nothing more, taking us days to track the problem down ultimately to this bad MS image, time we didn't have as we had critical code changes we needed built and deployed...blocked by this.

I'm straight up shocked this got through any testing, much less published as the latest? We apparently had far too much faith in MS's release process here. Clearly, tracking latest for these SDK images is absolutely NOT suitable for production work flows. We're reviewing where we go from here, but chances are this complete loss of faith in MS's images will mandate we build a big, ugly, slow review process internally where we keep, maintain, and strictly bless each update ourselves.

And so should every other dev group on earth that uses these images....which is a massive amount of completely redundant work...that makes no sense whatsoever...yet this is where we are with this now DAYS old latest image that's STILL completely FUBAR with NO effective response that MS folks are even aware.

mthalman commented 4 years ago

@bbrummer - I completely understand your frustration. MSBuild does exist in the image and the image will work if your Docker host is updated to the latest Windows update. This issue is not specific to .NET Framework Docker images but is rather a result of a version incompatibility with all Windows images for this month's update and is documented here. That is the official response to this known issue. We are actively working on providing better test coverage for Windows containers to help prevent issues like this from occurring; this worked is tracked in https://github.com/dotnet/docker-tools/issues/387.

tallandtree commented 4 years ago

What to do with the fact that said update is no longer available: https://support.microsoft.com/en-us/help/4524244/security-update-for-windows-10-february-11-2020. Does this mean that the new windows docker images that cause these issues will be removed as well from the docker registry?

mthalman commented 4 years ago

What to do with the fact that said update is no longer available: https://support.microsoft.com/en-us/help/4524244/security-update-for-windows-10-february-11-2020. Does this mean that the new windows docker images that cause these issues will be removed as well from the docker registry?

@tallandtree - The update you're referring to is a standalone update for UEFI boot manager. It is not the latest cumulative updates (LCU) for February 2020 which are still available. See this page for links to all the LCUs: https://support.microsoft.com/help/4540981/windows-server-images-for-february-2020

MichaelSimons commented 4 years ago

Closing as this is caused by the Windows container change that requires the Docker host machine match the container version. See this documentation for additional details.

Selmirrrrr commented 4 years ago

Same here, msbuild.exe apparently does not exist in the latest image? At least for Server 2016.

This trashed our AWS CodeBuild pipelines with the same "exit status 3221225785" error and nothing more, taking us days to track the problem down ultimately to this bad MS image, time we didn't have as we had critical code changes we needed built and deployed...blocked by this.

I'm straight up shocked this got through any testing, much less published as the latest? We apparently had far too much faith in MS's release process here. Clearly, tracking latest for these SDK images is absolutely NOT suitable for production work flows. We're reviewing where we go from here, but chances are this complete loss of faith in MS's images will mandate we build a big, ugly, slow review process internally where we keep, maintain, and strictly bless each update ourselves.

And so should every other dev group on earth that uses these images....which is a massive amount of completely redundant work...that makes no sense whatsoever...yet this is where we are with this now DAYS old latest image that's STILL completely FUBAR with NO effective response that MS folks are even aware.

Hello, may I ask if you did found a solution on AWS CodeBuild for this issue ? Thanks in advance :)

viscardo commented 4 years ago

I just tried to make it work in AWS code build and I am getting the "Reason: exit status 3221225785" Can anyone in MS help me ?

elee1001 commented 4 years ago

I am having the same problem. The image works fine in an Windows 2016 Server instance but when its added to the AWS CodeBuild, I am getting the same error running MSbuild "Reason: exit status 3221225785".

mthalman commented 4 years ago

For people running into this issue, see the earlier comment explaining the cause: https://github.com/microsoft/dotnet-framework-docker/issues/521#issuecomment-591601203.

woodyiii commented 4 years ago

strangely, we just started to encounter this today for our AWS CodeBuild images as well. thanks for the earlier comments and explanation. will try to see if that help us repair things.

eli88fine commented 4 years ago

This just started happening to us today as well. We have the AWS CodeBuild docker image pinned to a specific version...so it's very surprising that all of a sudden it's erroring.

CloudFormation definition of docker image (I also manually confirmed in the online console that the image name in the CodeBuild project still matches): ComputeType: BUILD_GENERAL1_MEDIUM Image: aws/codebuild/windows-base:2.0-20.02.07 Type: WINDOWS_CONTAINER

Build log:

[Container] 2020/05/08 18:31:27 Downloading S3 cache...

14 | [Container] 2020/05/08 18:31:51 Registering with agent 15 | [Container] 2020/05/08 18:31:51 Phases found in YAML: 3 16 | [Container] 2020/05/08 18:31:51 BUILD: 19 commands 17 | [Container] 2020/05/08 18:31:51 INSTALL: 30 commands 18 | [Container] 2020/05/08 18:31:51 PRE_BUILD: 2 commands 19 | [Container] 2020/05/08 18:31:51 Phase complete: DOWNLOAD_SOURCE State: SUCCEEDED 20 | [Container] 2020/05/08 18:31:51 Phase context status code: Message: 21 | [Container] 2020/05/08 18:31:51 Entering phase INSTALL 22 | [Container] 2020/05/08 18:31:51 Running command python --version 23 | Python 3.7.5 24 |   25 | [Container] 2020/05/08 18:31:54 Running command node --version 26 | v10.18.0 27 |   28 | [Container] 2020/05/08 18:31:57 Running command npm --version 29 | 6.13.4 30 |   31 | [Container] 2020/05/08 18:32:01 Running command nvm ls 32 |   33 | [Container] 2020/05/08 18:32:04 Command did not exit successfully nvm ls exit status 3221225794 34 | [Container] 2020/05/08 18:32:07 Phase complete: INSTALL State: FAILED 35 | [Container] 2020/05/08 18:32:07 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: nvm ls. Reason: exit status 3221225794

KristinS-realityi commented 4 years ago

This started happening to us 4-5 days ago. @eli88fine @woodyiii - were you able to resolve?

eli88fine commented 4 years ago

I opened a support ticket with AWS. They claimed they had released a new Windows image back in February to use in CodeBuild that was compatible with the security patch. They hadn't. I pointed that out to them on Sunday----they still haven't replied to that message, but sometime in the last 12 hours they released a new Windows Server image to use in CodeBuild (although they still named it as if they had released it on May 5: aws/codebuild/windows-base:2.0-20.05.05).

I updated my CodeBuild to use the new image and the problem resolved.

cloneluke commented 4 years ago

yes, you need to update your codebuild images to have the feb security updates (AWS just did their hosts on 5/7 in US-EAST-1) so for me, I went from: FROM mcr.microsoft.com/dotnet/framework/runtime:4.7.2-20190611-windowsservercore-ltsc2016 to FROM mcr.microsoft.com/dotnet/framework/runtime:4.7.2-20200414-windowsservercore-ltsc2016 rebuilt docker image and uploaded to ecr and I was GTG.

This was especially frustrating because the new container would not work pre-5/7 and the old container would not work post-5/7 because host and container both had to be at post-Feb levels.

mthalman commented 4 years ago

This was especially frustrating because the new container would not work pre-5/7 and the old container would not work post-5/7 because host and container both had to be at post-Feb levels.

Yes, the February update was the inflection point here. If the Docker host and Docker container had Windows versions that differed such that they were on either side of the February update, then things would not work. So you could have a Docker host and container with a Windows version that there both older or both newer than the February update and things would work. But you couldn't have a Docker host with a Windows version older than the February update and a Docker container with a version equal to or newer than Feb update, or vice versa. This was unintentional behavior on the part of the Windows release.

roysouma commented 4 years ago

yes, you need to update your codebuild images to have the feb security updates (AWS just did their hosts on 5/7 in US-EAST-1) so for me, I went from: FROM mcr.microsoft.com/dotnet/framework/runtime:4.7.2-20190611-windowsservercore-ltsc2016 to FROM mcr.microsoft.com/dotnet/framework/runtime:4.7.2-20200414-windowsservercore-ltsc2016 rebuilt docker image and uploaded to ecr and I was GTG.

This was especially frustrating because the new container would not work pre-5/7 and the old container would not work post-5/7 because host and container both had to be at post-Feb levels.

Hi, Can you please share the docker file used? I'm using the build commands specified in https://aws.amazon.com/blogs/devops/extending-aws-codebuild-with-custom-build-environments-for-the-net-framework/ which is failing now with updated image. I have attached the file that I'm using. Dockerfile_backup.txt

cloneluke commented 4 years ago

yes, you need to update your codebuild images to have the feb security updates (AWS just did their hosts on 5/7 in US-EAST-1) so for me, I went from: FROM mcr.microsoft.com/dotnet/framework/runtime:4.7.2-20190611-windowsservercore-ltsc2016 to FROM mcr.microsoft.com/dotnet/framework/runtime:4.7.2-20200414-windowsservercore-ltsc2016 rebuilt docker image and uploaded to ecr and I was GTG. This was especially frustrating because the new container would not work pre-5/7 and the old container would not work post-5/7 because host and container both had to be at post-Feb levels.

Hi, Can you please share the docker file used? I'm using the build commands specified in https://aws.amazon.com/blogs/devops/extending-aws-codebuild-with-custom-build-environments-for-the-net-framework/ which is failing now with updated image. I have attached the file that I'm using. Dockerfile_backup.txt

whats your error? My dockerfile is a very specialized use case not pertinent to this problem

roysouma commented 4 years ago

Hi Luke,

I could resolve the issue. I had to apply latest updates to the ec2

On Wed, 13 May 2020 at 6:14 PM, Luke Bodeen notifications@github.com wrote:

yes, you need to update your codebuild images to have the feb security updates (AWS just did their hosts on 5/7 in US-EAST-1) so for me, I went from: FROM mcr.microsoft.com/dotnet/framework/runtime:4.7.2-20190611-windowsservercore-ltsc2016 to FROM mcr.microsoft.com/dotnet/framework/runtime:4.7.2-20200414-windowsservercore-ltsc2016 rebuilt docker image and uploaded to ecr and I was GTG. This was especially frustrating because the new container would not work pre-5/7 and the old container would not work post-5/7 because host and container both had to be at post-Feb levels.

Hi, Can you please share the docker file used? I'm using the build commands specified in https://aws.amazon.com/blogs/devops/extending-aws-codebuild-with-custom-build-environments-for-the-net-framework/ which is failing now with updated image. I have attached the file that I'm using. Dockerfile_backup.txt https://github.com/microsoft/dotnet-framework-docker/files/4620974/Dockerfile_backup.txt

whats your error? My dockerfile is a very specialized use case not pertinent to this problem

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/microsoft/dotnet-framework-docker/issues/521#issuecomment-627957764, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOPZTSLPIS2RB3XWXSBINWLRRKI3NANCNFSM4KZK6F5Q .

sowmyajagar commented 4 years ago

I was able to resolve the issue by building custom image with base image mcr.microsoft.com/dotnet/framework/sdk:4.8-20200512-windowsservercore-ltsc2016 . SDK will install build tools and other dependencies for you.