[Azure DevOps] Windows-2016 environment removal postponed until July 31, 2022

miketimofeev commented 2 years ago

Breaking changes

Windows 2016 hosted runners were scheduled for full removal on April 1, 2022. However, we are still seeing significant usage from customers and we want to give them more time to migrate to the new runners. In order to give customers another chance to move to either windows-2019 or windows-2022 we will delay the removal of windows-2016 until July 31, 2022.

To raise awareness of the removal, we will have rolling 24-hour periods where the image will be unavailable for customers. During these "brownout" periods, customers will see an error message indicating that the image is slated for removal on July 31, 2022. The schedule for those brownout periods is as follows for Azure DevOps:

Monday April 18, 00:00 UTC to April 19, 00:00 UTC
Tuesday April 26, 00:00 UTC to April 27, 00:00 UTC
Wednesday May 4, 00:00 UTC to May 5, 00:00 UTC
Thursday May 12, 00:00 UTC to May 13, 00:00 UTC
Friday May 20, 00:00 UTC to May 21, 00:00 UTC
Monday May 23, 00:00 UTC to May 24, 00:00 UTC
Tuesday May 31, 00:00 UTC to June 1, 00:00 UTC
Wednesday June 8, 00:00 UTC to June 9, 00:00 UTC
Thursday June 16, 00:00 UTC to June 17, 00:00 UTC
Friday June 24, 00:00 UTC to June 25, 00:00 UTC
Monday June 27, 00:00 UTC to June 28, 00:00 UTC
Friday July 22, 00:00 UTC to July 23, 00:00 UTC
Wednesday July 27, 00:00 UTC to July 28, 00:00 UTC
Full removal - July 31, 15:00 UTC

See the original announcement at https://github.com/actions/virtual-environments/issues/4312

Target date

July 31, 2022

The motivation for the changes

Windows Server 2016 Active support ends on 11 Jan 2022 and Windows Server 2022 VM image is going out of beta later this year. As part of our ongoing efforts to keep GitHub and Azure Devops hosted runners updated and secure, the Windows 2016 virtual environment will be removed from GitHub Actions and Azure DevOps.

Possible impact

Workflows targeting the Windows 2016 image will fail.

Virtual environments affected

[ ] Ubuntu 18.04
[ ] Ubuntu 20.04
[ ] macOS 10.15
[ ] macOS 11
[ ] macOS 12
[x] Windows Server 2016
[ ] Windows Server 2019
[ ] Windows Server 2022

Mitigation ways

Change your workflows to use windows-latest, windows-2022, or windows-2019

kningyi commented 2 years ago

Noob here, our yml config is already using vmImage: 'windows-latest' but we're still seeing the error. What should we do?

miketimofeev commented 2 years ago

@kningyi please remove everything below the

pool:
    vmImage: 'windows-latest'

(name and demands) and you should be good to go

gene-keys commented 2 years ago

IMPORTANT NOTE FOR THOSE OF YOU STILL STRUGGLING WITH WIN2016 BROWNOUT NOTICE.

CHECK EVERYTHING!

NOT JUST YOUR PIPELINE CODE, CHECK PIPELINE TRIGGERS POOLS. CHECK RELEASES POOLS, CHECK ALL THE POOLS EVERYWHERE ON YOUR DEVOPS ACCOUNT. WHAT WILL HAPPEN IS YOU WILL EVENTUALLY FIND SOMETHING STILL SET TO USE WIN2016 POOL. ONCE YOU FIND ALL OF THEM YOUR ACCOUNT WILL NO LONGER GET THE BROWNOUT NOTICE EVEN DURING THE SCHEDULED TIMES. SO IF YOU ARE GETTING THE NOTICE ITS NOT YOUR WIN2017 POOL, OR WIN-LATEST IN THE YML, ITS SOMETHING ELSE,

AZURE NOTICES ARE DOING A TERRIBLE JOB ON THIS WITH THEIR WORDING AND WHERE THE NOTICE APPEARS. THE NOTICE MEANS YOUR AZURE DEVOPS ACCOUNT IS STILL USING WINDWOS POOL SOMEWHERE, YOU DID NOT CHECK EVEYWHERE.

Aeroverra commented 2 years ago

I have a release pipeline (the one in the Releases tab) that gets triggered after a pipeline from the Pipelines Tab that I use for deployment to azure. This deployment keeps failing because of the 2016 error.

In the normal build pipeline I was able to set the vmImage but I don't see any way to access the yaml or change the image in the release pipeline edit page. How can this be done? I have tried searching this up plenty of times with no luck.

Edit: lol I can't believe that alluded me for so long. In case anyone else is looking. When you edit the release pipeline its under the tasks tab and then the category "run on agent". Looks like 2019 was recently made the default finally.

Ziya137200 commented 2 years ago

Noob here, our yml config is already using vmImage: 'windows-latest' but we're still seeing the error. What should we do?

slonopotamus commented 2 years ago

As part of our ongoing efforts to keep GitHub and Azure Devops hosted runners updated and secure

Isn't Windows Server 2016 going to get security updates until 2027 and this "motivation" is simply misleading?

eli-schwartz commented 2 years ago

As discussed in #5238

During these "brownout" periods, customers will see an error message indicating that the image is slated for removal on June 30, 2022.

The brownout error message still says April 1 and points to an old ticket with out-of-date information.

prajwalar97 commented 2 years ago

We are having a pipeline which runs on 2016 Agent Specification. After changing the Agent Specification to 2019 or windows Latest we are getting below Error 2022-04-21T15:06:27.3818037Z Project file contains ToolsVersion="15.0". This toolset may be unknown or missing, in which case you may be able to resolve this by installing the appropriate version of MSBuild, or the build may have been forced to a particular ToolsVersion for policy reasons. Treating the project as if it had ToolsVersion="4.0". For more information, please see http://go.microsoft.com/fwlink/?LinkId=291333. error CS1519: Invalid token '(' in class, struct, or interface member declaration error CS1519: Invalid token '.' in class, struct, or interface member declaration

robertoandrade commented 2 years ago

As part of our ongoing efforts to keep GitHub and Azure Devops hosted runners updated and secure

Isn't Windows Server 2016 going to get security updates until 2027 and this "motivation" is simply misleading?

I agree 100% with this. According to Microsoft's own documentation, Windows Server 2016 will continue to get security updates until Jan 12, 2027 when it will finally stop receiving it which would justify for Microsoft to remove support for it on pipelines or start these brownouts.

geekzter commented 2 years ago

Noob here, our yml config is already using vmImage: 'windows-latest' but we're still seeing the error. What should we do?

@Ziya137200 @kningyi please remove the line (26) with the reference to pool 'Hosted VS2017'.

afelthe commented 2 years ago

I keep getting this error in my release pipelines and cannot figure out where to go to fix it. I am not using YAML but CLI.

mohitook commented 2 years ago

I get this message the whole day. Could you move your maintenance window out of work-hours next time? We'd appreciate it.

Additionally our systems are not using any windows-2016 related element, so the error itself makes no sense...

miketimofeev commented 2 years ago

@mohitook if the systems don't use then you won't see the message. Maybe some of your jobs are using windows-2016 as it was the default image some time ago?

carvido1 commented 1 year ago

Hello. Is this affecting also self hosted runners ? It is not clear to me

miketimofeev commented 1 year ago

@carvido1 no, it is not

sicklittlemonkey commented 1 year ago

@mohitook if the systems don't use then you won't see the message. Maybe some of your jobs are using windows-2016 as it was the default image some time ago?

Yep, for me it was a copy of an older pipeline. I thought the error was just for the pipeline being run, but it seems to be caused by any other 2016 agent setting you have.

Edit: Oh no, I still have the error. Must be another one ... Wish it listed them!

bobblybit commented 1 year ago

how do i fix this issue

bobblybit commented 1 year ago

@mohitook if the systems don't use then you won't see the message. Maybe some of your jobs are using windows-2016 as it was the default image some time ago?

please how do i fix this

megamingus commented 1 year ago

Hey there!, I was just hit by the brownout in a pipeline working flawlessly for years 😭. when running it on a 2022 image, I get an error. I'm not sure but, it seems that there are missing some UWP SDKs?

##[warning]C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\WindowsXaml\v17.0\8.2\Microsoft.Windows.UI.Xaml.Common.targets(676,5): Warning : Could not find target platform winmds for the SDK specified by [Windows, 10.0, UAP, 10.0.10240.0, 10.0.15063.0]
##[debug]Processed: ##vso[task.logissue type=Warning;sourcepath=C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\WindowsXaml\v17.0\8.2\Microsoft.Windows.UI.Xaml.Common.targets;linenumber=676;columnnumber=5;code=;]Could not find target platform winmds for the SDK specified by [Windows, 10.0, UAP, 10.0.10240.0, 10.0.15063.0]
C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\WindowsXaml\v17.0\8.2\Microsoft.Windows.UI.Xaml.Common.targets(676,5): warning : Could not find target platform winmds for the SDK specified by [Windows, 10.0, UAP, 10.0.10240.0, 10.0.15063.0] 
[...]
##[error]MuralSketch.Core\Components\InkAnalyzerComponent\InkAnalyzerComponentModel.cs(7,31): Error CS0234: The type or namespace name 'Analysis' does not exist in the namespace 'Windows.UI.Input.Inking' (are you missing an assembly reference?)

gsmtcnr commented 1 year ago

I selected windows2022 instead of vs2017-win2016 then It fixed.

JMA-Dv commented 1 year ago

I made a post in stack where I explaned how I fixed this issue. If you're interested https://stackoverflow.com/questions/71493785/error-deploying-release-pipeline-windows-2016-deprecated-azure-devops/72779265#72779265

kapilvagyani commented 1 year ago

Does this means after 30th June 2022 we wont be able to create VM even for window server sku- "2016-Datacenter-Server-Core" using images from Microsoft market place ? pls clarify. Thanks in advance. Kapil

al-cheb commented 1 year ago

Does this means after 30th June 2022 we wont be able to create VM even for window server sku- "2016-Datacenter-Server-Core" using images from Microsoft market place ? pls clarify. Thanks in advance. Kapil

These changes are not related to Azure, only to Hosted Agents.

megamingus commented 1 year ago

is there a way to install missing UWP SDKs in newer images? or are te images going to be updated with the missing ones?

miketimofeev commented 1 year ago

@megamingus what SDKs do you mean?

MelchiorBrandtCorstiusAtABNAMRO commented 1 year ago

Hello, in our organization we still see jobs being run on Microsoft Hosted vs2017-win2016 Agents. I thought these would no longer be available as of 30 June? When can we expect these to be offline permanently?

miketimofeev commented 1 year ago

@melchiorbc we are a bit delayed with some changes to default hosted images. Most likely the image will be fully deprecated in 1-2 weeks.

rushfrisby commented 1 year ago

What other agent can build .NET Framework projects? I've tried windows-latest, windows-2022, and windows-2019 but the solution fails to build on all with an error about not being able to locate the .NET Framework assemblies.

miketimofeev commented 1 year ago

@rushfrisby what error do you have? Windows-2019, for example, has a reduced set of installed .Net frameworks: 4.7.2 & 4.8 vs 4.6.1, 4.6.2, 4.7, 4.7.1, 4.7.2, 4.8 on windows-2016

rushfrisby commented 1 year ago

@miketimofeev ##[error]C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(1221,5): Error MSB3644: The reference assemblies for .NETFramework,Version=v4.6.1 were not found. To resolve this, install the Developer Pack (SDK/Targeting Pack) for this framework version or retarget your application. You can download .NET Framework Developer Packs at https://aka.ms/msbuild/developerpacks

al-cheb commented 1 year ago

@rushfrisby, Could you bump .Net version to 4.6.2?

JMA-Dv commented 1 year ago

@rushfrisby I ran against this issue. 1) Try to point your project to .NET v4.8. 2) Re-publish now with the v4.8 project. (if it doesn't fix it, go to step 3). 3) In your .yaml pipeline check for the pool tag. and change the vmImage: 'windows-2016' to 'windows-2019' or 'windows-latest' 4) Re-run your pipeline.

Those were my steps to correct the issue.

DeeDeeG commented 1 year ago

EDIT: Okay, this comment can be disregarded. The problem was resolved for us. We resolved this by updating one of our native C/C++ dependencies. It builds and passes CI running on the new Windows image.

Original comment

We have a project that started experiencing CI errors in the new image, even when we installed older Visual Studio Build Tools with Chocolatey. So newer Visual Studio isn't the culprit. https://github.com/atom-community/atom/issues/428

Note: _Building_ on the new image caused the problem. We have been testing on the `windows-2019` image for over a year. If we take the app built on the newer image (`windows-2019`) and test it on `vs2017-win2016`, it does _not_ cause the tests to pass. I can only conclude _building_ on the newer image is the problem.

Does anyone have any idea what else is different about the new CI image environment that would have caused this? `windows-2019` vs `vs2017-win2016`? _(Other than Visual Studio being newer?)_ Any hints of where to zero in our troubleshooting? Thank you. Right now it's looking like a subtle difference of behavior in some native C++ addon code we build for a Node module or two. (Async code that never returns, if I understand correctly.)

miketimofeev commented 1 year ago

The new retirement date is set — July, 22

miketimofeev commented 1 year ago

The retirement date is shifted to July, 31

MelchiorBrandtCorstiusAtABNAMRO commented 1 year ago

LOL, can you guys please, please, please stop shifting the retirement date :) every time we need to tell our developers to stop using Win2016 as Agent type, and each time you guys postpone the deadline slowly people start using it again (I know their own fault), and we need to start warning them again, but we (and also Microsoft) are loosing our credibility on when they are actually decommissioned.

miketimofeev commented 1 year ago

@melchiorbc sorry for the inconvenience, hopefully, this is the last one 🤞

miketimofeev commented 1 year ago

Two more brownout dates are set:

Friday July 22, 00:00 UTC to July 23, 00:00 UTC
Wednesday July 27, 00:00 UTC to July 28, 00:00 UTC

miketimofeev commented 1 year ago

Windows 2016 environment is finally deprecated in ADO.

ddeveza commented 1 year ago

I already updated the agent spec but still encountering the error of 2016 deprecation.

Bondarenko1990 commented 1 year ago

Windows 2016 hosted runners were scheduled for full removal on April 1, 2022. However, we are still seeing significant usage from customers and we want to give them more time to migrate to the new runners. In order to give customers another chance to move to either windows-2019 or windows-2022 we will delay the removal of windows-2016 until July 31, 2022.

same error. I use windows-2019

MelchiorBrandtCorstiusAtABNAMRO commented 1 year ago

Windows 2016 environment is finally deprecated in ADO.

Hi Mike, are you sure of this? Up to today we still see jobs using the "vs2017-win2016" Microsoft Hosted Agent image.

miketimofeev commented 1 year ago

@melchiorbc could you share a few links, please? I don't need access to pipelines, will take a look at telemetry by those urls

MelchiorBrandtCorstiusAtABNAMRO commented 1 year ago

@miketimofeev I checked and I do see now the jobs trying to use 'vs2017-win2016' do get cancelled by Microsoft stating this is a deprecated environment, only people are still able to pick the 'vs2017-win2016' in their pipelines from the dropdown with VM types (which is confusing).

miketimofeev commented 1 year ago

@melchiorbc this is fixed now, sorry for the inconvenience

actions / runner-images