Ubuntu 16.04 environment will be removed on September 20, 2021

[AlenaSviridenko]

Traditional 5-years support of Ubuntu 16.04 by Canonical ends in April, 2021. To keep our environment updated and secured, we will remove Ubuntu 16.04 from GitHub on September 20, 2021, and from Azure DevOps on October 18, 2021.

You will need to migrate ubuntu-16.04 workflows to ubuntu-18.04 or ubuntu-latest which will run on Ubuntu 20.04 LTS.

To make sure everyone is aware of this change, we scheduled 2 short "brownouts". Ubuntu 16.04 builds will fail during the brownout period. Therefore, it is recommended to migrate your workflows prior to September 6, 2021.

The brownouts are scheduled for the following dates and times:

• September 6 September 7, 2021 4:00pm UTC – 10:00pm UTC 1:00pm UTC – 7:00pm UTC
• September 14, 2021 4:00pm UTC – 10:00pm UTC (GitHub only)
• October 11, 2021 4:00pm UTC – 10:00pm UTC (Azure DevOps only)

Please, use this announcement issue to provide any feedback or ask questions regarding the upcoming change. Thank you.

[tomato42]

The brownouts are scheduled for the following dates and times:

* September 6, 2021 5:00pm UTC – 10:00pm UTC

* September 14, 2021 5:00pm UTC – 10:00pm UTC

So anybody that works a 9 to 5 job in EMEA or APAC has no chance of hitting them?

[hugovk]

https://www.timeanddate.com/worldclock/meetingtime.html?iso=20210906&p1=136&p2=195&p3=101&p4=176&p5=33&p6=248&p7=152

[sergeyklay]

To keep our environment updated and secured, we will remove Ubuntu 16.04 on September 20, 2021.

Removing the support of distro version does not make the environment updated, it makes the environment less rich in features. There are many cases where you need to make sure that your project sill works on a previous version of the OS/distro. For example, your project depends on libraries that can only be installed on the Ubuntu 16.04 or even 14.04. The less choice you give, the less wide the range of systems where a particular project has been tested will be. Ultimately, this is an affect on the end user.

[Wohlstand]

I do builds on older Ubuntu as these builds are able to run in a portable mode on a wide number of Linux distros. Building with newer Ubuntu will cause these binaries will not work in many cases:

• Ubuntu 20 builds will work on Fedora as I tested
• Ubuntu 18 builds at least works on OpenSUSE, but they won't work on Debian
• Ubuntu 16 builds works on Debian and works on CentOS 7

[AlenaSviridenko]

Hi there, please, consider using self-hosted runners for such specific cases https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners

[catthehacker]

I do builds on older Ubuntu as these builds are able to run in a portable mode on a wide number of Linux distros. Building with newer Ubuntu will cause these binaries will not work in many cases:

• Ubuntu 20 builds will work on Fedora as I tested
• Ubuntu 18 builds at least works on OpenSUSE, but they won't work on Debian
• Ubuntu 16 builds works on Debian and works on CentOS 7

Isn't that just based on GLIBC compatibility? Build static binary and it will work on all machines.

[Wohlstand]

Isn't that just based on GLIBC compatibility? Build static binary and it will work on all machines.

Yeah, however, even I do link -static-libgcc, it still depends on the external libc. The most of failure reasons are the libm and pthread which I can't link statically as they causes me other challenges

[catthehacker]

To keep our environment updated and secured, we will remove Ubuntu 16.04 on September 20, 2021.

Removing the support of distro version does not make the environment updated, it makes the environment less rich in features. There are many cases where you need to make sure that your project sill works on a previous version of the OS/distro. For example, your project depends on libraries that can only be installed on the Ubuntu 16.04 or even 14.04. The less choice you give, the less wide the range of systems where a particular project has been tested will be. Ultimately, this is an affect on the end user.

Ubuntu 16.04 LTS is supported until 2024 through Canonical’s Extended Security Maintenance (ESM) product. Xenial enters the ESM period in April 2021, with security patches provided for an additional three years beyond the traditional five-year standard support.

It's also possible to build artifacts/binaries in Docker container using jobs.<job_id>.container and ubuntu:16.04 image (https://hub.docker.com/_/ubuntu)

[DeeDeeG]

(Note: It turns out we can build our app on Ubuntu 18.04, and the app will still run on Ubuntu 16.04 and Debian 9. Debian 8 and Ubuntu 14.04 were already too old to run our app, even when we built the app on Ubuntu 16.04. So I don't think upgrading to building on Ubuntu 18.04 makes any backward-compatibility issues for us. This question is out of curiosity and out of my personal interest in having a backup plan in case something new comes up that still requires building on Ubuntu 16.04.)

I was curious to try the Docker container approach for still building on Ubuntu 16.04. But Azure Pipelines makes it so the user running commands in the Docker container is not root, and yet the sudo command isn't installed in ubuntu Docker images newer than 14.04, and seccomp appears to block creation of the apt cache directories. So apt-get update and apt-get install or sudo apt-get update etc. all fail. So you can't easily update or install packages in an Ubuntu 16.04 Docker image under Azure Pipelines. (Or I couldn't find a way to).

If the Azure Pipelines folks know a way to make this work, I'd be interested to know. It's hard to take advantage of Ubuntu Docker images if you can't install packages in them. (The base images are pretty sparse when it comes to preinstalled packages). We wouldn't be able to finish the build with just the few packages included in the base ubuntu:16.04 Docker image.

Anyhow, thanks for the heads-up advisory, and for making this CI service available.

P.S. I have found elsewhere on the internet that "using a container with sudo preinstalled" is a recommended solution, but it's hard to find trustworthy and fit-for-purpose images based on Ubuntu 16.04 without creating and hosting one myself, which would be a lot of effort, would probably get out of date somewhat quickly, and could maybe even be expensive. So making it work with the official ubuntu:16.04 Docker image would be fantastic. Any hints/tips/tricks appreciated. But otherwise my "Plan A" is just to build on Ubuntu 18.04 for now.

[catthehacker]

@DeeDeeG There are better suitable images for building/compiling software such as buildpack-deps but they don't include sudo unfortunately. I'm also not aware of any such images, I usually have to make them myself (https://github.com/catthehacker/docker_images/blob/fc531ee59cd7a1f56422d42b9ec92993ed834471/linux/ubuntu/act/Dockerfile). The nice thing about Docker is that you could have Dockerfile(s) in your repository for creating development environment but also used for CI.

As for the non-root user you could try running container with options. Both GitHub Actions and Azure DevOps support standard docker run flags via options:

resources:
  containers:
  - container: tests
    image: ubuntu:18.04
    options: --user 0 # or --user root but using 0 or 0:0 guarantees that root-privileged user is used if the username was changed

P.S. I have found elsewhere on the internet that "using a container with sudo preinstalled" is a recommended solution, but it's hard to find trustworthy and fit-for-purpose images based on Ubuntu 16.04 without creating and hosting one myself, which would be a lot of effort, would probably get out of date somewhat quickly, and could maybe even be expensive. So making it work with the official ubuntu:16.04 Docker image would be fantastic. Any hints/tips/tricks appreciated. But otherwise my "Plan A" is just to build on Ubuntu 18.04 for now.

I've been figuring that out and so far I have pretty well working solution for auto building containers that are up-to-date (if you don't count need to change packages, typos, etc.). https://github.com/catthehacker/docker_images/blob/master/.github/workflows/build-ubuntu.yml If you need any help regarding Docker/files, you can ping me, preferably not in this thread to not go full off topic 😸

[MaksimZhukov]

Hello everyone! The first Ubuntu 16.04 brownout has been moved from September 6 to September 7 (4:00pm UTC – 10:00pm UTC). Thanks!

[miketimofeev]

The first brownout time has also switched to 1:00pm UTC – 7:00pm UTC to cover more users.

[hugovk]

Updated timezone chart for September 7 (1:00pm UTC – 7:00pm UTC):

https://www.timeanddate.com/worldclock/meetingtime.html?day=7&month=9&year=2021&p1=136&p2=195&p3=101&p4=176&p5=33&p6=248&p7=152&iv=0

[felfa01]

We are not specifying our agent pool selection in our azure pipeline yaml hierarchy. We noticed now during brownout that it defaults to ubuntu-16.04 (causing outage for us). Wouldn't it make more sense for hosted agent pool selection to default to ubuntu-latest?

[nick-anykeydesigns]

There is a major bug, we have our environment set to ubuntu-latest and this is still bricking us with the brownout!!!

`variables:

Agent VM image name

vmImageName: "ubuntu-latest"`

[knordum]

Noticed that using "Ubuntu-latest" (not sure if the capital U makes the difference) specified at the top level on the pool node fixed it for us (Using ubuntu latest) `trigger:

• main

  pool: vmImage: 'Ubuntu-latest'

  steps:

• task: Maven@3 inputs: mavenPomFile: 'pom.xml' mavenOptions: '-Xmx3072m' javaHomeOption: 'JDKVersion' jdkVersionOption: '1.8' jdkArchitectureOption: 'x64' publishJUnitResults: false testResultsFiles: '*/surefire-reports/TEST-.xml' goals: 'package'`

According to this documentation: https://docs.microsoft.com/en-us/azure/devops/pipelines/customize-pipeline?view=azure-devops

[nick-anykeydesigns]

Thanks for the idea, but I just switched to 'Ubuntu-latest' and got the same error @knordum

[miketimofeev]

@nick-anykeydesigns @PaulGarnerUk could you please share the sample pipeline? Something like this should work

trigger:
- master

pool:
  vmImage: 'Ubuntu-latest'

steps:
- script: java -version
  displayName: 'Run a one-line script'

[nick-anykeydesigns]

I just updated my build process to use ubuntu-18.04 w/ no luck

[RobClenshaw]

We're specifying ubuntu-20.04 and getting hit by the brownout.

[mrcmoresi]

Noticed that using "Ubuntu-latest" (not sure if the capital U makes the difference) specified at the top level on the pool node fixed it for us (Using ubuntu latest) `trigger:

• main

pool: vmImage: 'Ubuntu-latest'

steps:

• task: Maven@3 inputs: mavenPomFile: 'pom.xml' mavenOptions: '-Xmx3072m' javaHomeOption: 'JDKVersion' jdkVersionOption: '1.8' jdkArchitectureOption: 'x64' publishJUnitResults: false testResultsFiles: '*/surefire-reports/TEST-.xml' goals: 'package'`

According to this documentation: https://docs.microsoft.com/en-us/azure/devops/pipelines/customize-pipeline?view=azure-devops

Tried this one in a Azure CD pipeline and worked perfectly!

[Azokah]

Noticed that using "Ubuntu-latest" (not sure if the capital U makes the difference) specified at the top level on the pool node fixed it for us (Using ubuntu latest) `trigger:

• main

pool: vmImage: 'Ubuntu-latest' steps:

• task: Maven@3 inputs: mavenPomFile: 'pom.xml' mavenOptions: '-Xmx3072m' javaHomeOption: 'JDKVersion' jdkVersionOption: '1.8' jdkArchitectureOption: 'x64' publishJUnitResults: false testResultsFiles: '*/surefire-reports/TEST-.xml' goals: 'package'`

According to this documentation: https://docs.microsoft.com/en-us/azure/devops/pipelines/customize-pipeline?view=azure-devops

Tried this one in a Azure CD pipeline and worked perfectly!

I was having the same issue and adding Ubuntu-latest pool to the YAML file solved it up for me too.

[javissimo]

Tried with the capital 'U' with no luck. I noticed that we have a multi-staged pipeline with every stage using ubuntu-latest. The first stage succeeded, showing with the following logs:

Pool: Azure Pipelines
Image: Ubuntu-latest

The second one picked Ubuntu 16 regardless having specified ubuntu-latest or Ubuntu-latest

Pool: Azure Pipelines
Image: Ubuntu16

Hope this helps to identify the issue!

[froboy]

ditto here. It seems that no pool definition will lead to this error, but adding:

pool:
  vmImage: 'Ubuntu-latest'

to any pipeline files that it's missing from should resolve the issue.

[nick-anykeydesigns]

through the UX we had this set

[skoladich]

I have hundreds of YAML builds in Azure DevOps, none of which specify a VM image. Changing all of those YAML builds is not an acceptable solution. Please change the default VM image.

[miketimofeev]

@skoladich the default image will be changed in the nearest future.

[skoladich]

@skoladich the default image will be changed in the nearest future.

How near? This is preventing me from doing my job today.

[benc-uk]

What's going on!? My pipelines refer to ubuntu-latest pool, but I'm getting this error about Ubuntu 16.04

The docs state that ubuntu-latest is an alias for ubuntu-20.04 https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml#software

EDIT: I changed to ubuntu-20.04 and still getting the brownout and my job cancelled

Azure DevOps Pipelines is effectively down right now

[thepoynt]

My pipeline is using windows-latest and is bricked due to the brownout. Is that intended? I'm trying to fix a critical production issue and can't for another 4 hours?

[andrii-lundiak]

My work related Pipeline using only 'ubuntu-latest' (lowercase u), and seems to be OK. Azure DevOps. Will be testing more these days.

[rfontijn]

Same here. Brownout when using ubuntu-latest. I need to deploy!

[larsrinn]

through the UX we had this set

Where can I find this?

[SimonAlling]

There is a major bug, we have our environment set to ubuntu-latest and this is still bricking us with the brownout!!!

`variables:

Agent VM image name

vmImageName: "ubuntu-latest"`

Not sure about the context here, but in Azure DevOps, the YAML key is vmImage, not vmImageName. It's not usually under variables though, so it's a bit of a long shot …

:bulb: General heads-up to everyone: If you're not successful in specifying a VM image on the top level of your Azure DevOps YAML pipeline and you need a quick fix, you can always do it for each job:

 - job:
   displayName: Do something
+  pool:
+    vmImage: ubuntu-20.04
   steps:
     - task: CmdLine@2
       inputs:
         script: |
           echo "Hello World!"

[MOURIK]

Same for me ! I'm using ubuntu-latest. i tried also with "ubuntu-18.04" but still same issue.

[error]This is a scheduled Ubuntu 16.04 brownout. Ubuntu 16.04 LTS environment will be removed on September 20, 2021. For more details, see https://github.com/actions/virtual-environments/issues/3287

,##[error]The remote provider was unable to process the request.

[benc-uk]

It's strange it's affecting a pipeline I run manually, but not one triggered from a PR

[benc-uk]

@SimonAlling - I have exactly that set in all my jobs, and I'm still getting the issue

[SimonAlling]

@SimonAlling - I have exactly that set in all my jobs, and I'm still getting the issue

Could you please share a YAML snippet representing a job that doesn't work?

[benc-uk]

AHHH HA! I had one job (which doesn't have any steps, there's a good reason, long story) where I'd not set a pool on it, because it has no steps. But this seems to fall back to the internal default of 16.04 and trigger the brownout

I've set a pool on this dummy job now, and it runs OK

[MOURIK]

I defined vmImage pool globally (vmImage: ubuntu-20.04) instead of declaring inside jobs. it works fine now!

[derekwilliamsliquidx]

I have also mitigated this issue with a global image definition. This was, in a few words, a charlie foxtrot of an event.

[miketimofeev]

We have stopped the brownout to resolve the issue with the default pools

[yosefe]

We have stopped the brownout to resolve the issue with the default pools

Does it mean the default pool is still going to work after Sep 20 (with newer ubuntu)?

[miketimofeev]

@yosefe yes, we won't make any new brownouts until the default pool is switched from Ubuntu 16

[nick-anykeydesigns]

through the UX we had this set

Where can I find this?

1. Edit the build
2. From the menu select triggers
3. Select the YAML tab

[nick-anykeydesigns]

FIX: https://stackoverflow.com/questions/69103955/azure-ado-build-pipeline-failed-due-to-brown-out-of-ubuntu-16-04/69103956#69103956

[minnieshi]

I defined vmImage pool globally (vmImage: ubuntu-20.04) instead of declaring inside jobs. it works fine now!

how did you do it at global level? (meaning on project or org level?)

[Dustone25]

Nada man just doing wassup!!

[SimonAlling]

how did you do it at global level? (meaning on project or org level?)

@MOURIK meant for an entire pipeline, not just for individual jobs:

pool:
  vmImage: ubuntu-20.04

jobs:
  - job:
    […] 

[miketimofeev]

The next brownout (September 14, 2021 4:00pm UTC – 10:00pm UTC) will only affect GitHub. For Azure DevOps we are working on switching the default vmimage from Ubuntu-16 to Ubuntu-20 thus the retirement date for Azure DevOps is shifted to October.