The Ubuntu 18.04 Actions runner image will begin deprecation on 2022/08/08 and will be fully unsupported by 2023/04/03

[miketimofeev]

Breaking changes

We have started the deprecation process for Ubuntu 18.04. While the image is being deprecated, You may experience longer queue times during peak usage hours. Deprecation will begin on 2022/08/08 and the image will be fully unsupported by 2023/04/03

To raise awareness of the upcoming removal, we will temporarily fail jobs using Ubuntu 18.04. Builds that are scheduled to run during the brownout periods will fail. The brownouts are scheduled for the following dates and times:

GitHub Actions\Azure DevOps::

• October 3, 12:00 UTC – October 3, 14:00 UTC
• October 18, 14:00 UTC – October 18, 16:00 UTC
• November 15, 18:00 UTC – November 15, 20:00 UTC
• November 30, 20:00 UTC – November 30, 22:00 UTC
• December 15, 20:00 UTC – December 16 00:00 UTC
• January 5, 10.00 UTC – January 5, 14.00 UTC
• January 13, 12.00 UTC – January 13, 16.00 UTC
• January 18, 14.00 UTC – January 18, 18.00 UTC
• January 24, 16.00 UTC – January 24, 20.00 UTC
• February 7, 16.00 UTC – February 7, 22.00 UTC
• February 21, 10.00 UTC – February 21, 22.00 UTC
• March 6, 00.00 UTC – March 7, 00.00 UTC
• March 13, 00.00 UTC – March 14, 00.00 UTC
• March 21, 00.00 UTC – March 22, 00.00 UTC
• March 28, 00.00 UTC – March 29, 00.00 UTC

Target date

April 1st, 2023

The motivation for the changes

We maintain the latest two stable versions of any given OS version. Ubuntu 22.04 is going GA on 8/8/22 thus we start deprecating the oldest image.

Possible impact

Workflows using the ubuntu-18.04 image label should be updated to ubuntu-latest, ubuntu-20.04, or ubuntu-22.04

Platforms affected

• [X] Azure DevOps
• [X] GitHub Actions

Virtual environments affected

• [X] Ubuntu 18.04
• [ ] Ubuntu 20.04
• [ ] Ubuntu 22.04
• [ ] macOS 10.15
• [ ] macOS 11
• [ ] macOS 12
• [ ] Windows Server 2019
• [ ] Windows Server 2022

Mitigation ways

N\A

[willaaam]

Don't get me wrong, I fully understand that you're deprecating 18.04 - but incorporating brownout periods to raise awareness?

If my team needs to get a build out for a critical bugfix on the 22nd of August... I'm just going to have to wait for 4 hours? If my understanding of that is true, I find that very odd behavior in an enterprise environment. Some businesses actually have SLAs with their clients. Would kindly request the team to find another way to "raise awareness". I came here because of the red banner when using the image, that worked fine as well and is a lot less invasive.

You're giving us two weeks to migrate away before brownouts start. Migrating is an action that requires testing and validation.

[fbrosseau]

Does this impact hosted agents only, or does this also mean the agent tooling/infra (for self hosted pools, but also for 1ES pools for internal folks) will become EOL for ubuntu18? My team still needs to keep shipping ubuntu18 versions of our product for years.

[panekj]

@fbrosseau it's hosted agent only

[smistad]

I am sad to see this removal of ubuntu-18.04 and hope you will reconsider. I use github to build C++ software on ubuntu-18.04, essentially making it possible to run it on 18.04 and newer releases. Due to the lack of backward compatability when compiling with the C++ libraries on linux (GLIBC), binaries compiled on 20.04 will not work on 18.04.

[mikhailkoliada]

I am sad to see this removal of ubuntu-18.04 and hope you will reconsider. I use github to build C++ software on ubuntu-18.04, essentially making it possible to run it on 18.04 and newer releases. Due to the lack of backward compatability when compiling with the C++ libraries on linux (GLIBC), binaries compiled on 20.04 will not work on 18.04.

@smistad as per the policy we only support the two latest LTS

[panekj]

I use github to build C++ software on ubuntu-18.04, essentially making it possible to run it on 18.04 and newer releases. Due to the lack of backward compatability when compiling with the C++ libraries on linux (GLIBC), binaries compiled on 20.04 will not work on 18.04.

You can use container: option to build with ubuntu:18.04 image or statically link binaries (or install Ubuntu 18.04 sysroot on new runners and build with that).

[smistad]

I use github to build C++ software on ubuntu-18.04, essentially making it possible to run it on 18.04 and newer releases. Due to the lack of backward compatability when compiling with the C++ libraries on linux (GLIBC), binaries compiled on 20.04 will not work on 18.04.

You can use container: option to build with ubuntu:18.04 image or statically link binaries.

Thanks for the tip @panekj

[ojwb]

Deprecation will begin on 8/8/2022 and the image will be fully unsupported by 12/1/2022

Should be 12/1/2023 or is it a US format date?

[aaronfranke]

@smistad as per the policy we only support the two latest LTS

@mikhailkoliada Is there any way we can convince GitHub to change this policy to the latest three LTS releases? We actually switched from 20.04 to 18.04 a few months ago because we found it worked better for our use cases.

[deathaxe]

What about projects which still need python 3.3 support for a reasonable time?

see: https://github.com/wbond/package_control/runs/7884696802?check_suite_focus=true

You must not simply break existing workflows!!!!

[scottshambaugh]

Ubuntu 18.04 does not reach the end of its standard support phase until April 2023. If the policy here is to support the last two LTS releases, I would suggest it be amended to align dropping support when Ubuntu does.

And another vote for 2 weeks before the brownouts start being very short.

[probonopd]

Ubuntu 18.04 does not reach the end of its standard support phase until April 2023. The AppImage project is relying on it to be available on GitHub Actions for the whole timeframe doring which it is supported.

[dbussink]

There seems to be quite a discovery issue with the brown out. I hadn't seen this issue originally which isn't the biggest deal, but seeing CI jobs fail with no context whatsoever is very confusing.

All I saw was a "This check was cancelled" with no context whatsoever about why it was cancelled. No message that there's a brown out, no link to this issue etc.

Somehow I was able to navigate to a different page of a status check that finally showed that information. Is it possible to make this more visible and prominent so it can be found easier?

It seems to not be visible if you click "Details" from a pull request. Only when manually navigating from the "Actions" tab can this be found.

[danfairs]

So yeah, we've just found out that 18.04 is deprecated. Considering I just upgraded to Enterprise at the cost of the order of thousands of dollars a month, I'm not massively pleased to see you just breaking our whole company's jobs for four hours.

Couldn't this be done in a different way? A banner on jobs that use the image? An email to org admins? It's pretty bad behaviour.

[panekj]

A banner on jobs that use the image?

[danfairs]

A banner on jobs that use the image?

What can I say? A whole company of 20 people didn't notice it.

[panekj]

There is also https://github.blog/changelog/2022-08-09-github-actions-the-ubuntu-18-04-actions-runner-image-is-being-deprecated-and-will-be-removed-by-12-1-22

[danfairs]

There is also https://github.blog/changelog/2022-08-09-github-actions-the-ubuntu-18-04-actions-runner-image-is-being-deprecated-and-will-be-removed-by-12-1-22

I'm sure there is. But we pay for this. You (as in GH, not you personally) need to be proactive about reaching out to customers when you're planning to break something. The rest of Microsoft is pretty good at this.

[m0ar]

This brownout scheme is the worst way to "raise awareness for deprecation" and has put a dark tint over our otherwise happy relationship with GitHub. Who figured that this would be a good solution? We sure disagree.

[covidium]

Why not send some emails? It's easier than screwing peoples builds :)

[amacgregorhumi]

@miketimofeev Brownout is pretty extreme, you are getting in the way of people shipping and being able to work. And stopping for service we are paying for.

This is huge loss of trust, and pushing us to consider alternatives.

[Steve-Glass]

👋 Thank you all for expressing your concerns with the deprecation schedule and feedback on preferred availability. We will revise the deprecation schedule and are will immediately stop the brownouts.

[j4mb0ti]

We are working on a complex migration with a pretty tight deadline (tomorrow). We have noticed the deprecation and planned for it accordingly. Running into a random brownout is the worst thing any tool could do to us at the moment. Yes, we are aware of the deprecation. Yes, we have planned to make the necessary changes when we are able to. You are just adding insult to an injury - not raising awareness. Much disappointment. I thought microsoft would have learned from its passed mistakes - but apparently we are still in the Windows SE area. That's the kind of awareness you are raising. And now I am pretty sure I'll keep my other projects over at GitLab.

[mullermp]

This is impacting AWS SDK For Ruby CI - I agree with others that this is a poor way to raise awareness, as this is the first time I've heard of this campaign. We can make steps to migrate.

[rtek]

Please come up with a more reasonable way to handle deprecation of the runners.

Consider how AWS handles lambda runtime deprecations as an example. They distribute emails when old runtimes are being used. You have access to all the workflows so you know whos using them.

[ijansch]

With a three week notice period, introducing these brownouts is like pulling the deprecation date forward. Because you can’t rely on a production system that breaks deployments every week for 6 hours. Not being able to perform an action during this time is Ill-advised and a serious risk for anyone relying on GitHub actions for production deployments.

It’s like putting a house on fire to raise awareness that playing with matches is dangerous.

See https://www.linkedin.com/posts/jansch_github-cicd-activity-6967561507354120192-084T?utm_source=share&utm_medium=member_ios for how this lead to a serious production incident at one of our customers when we needed to roll back a release because of a bug.

At least make this a setting so we can only do this ‘Forced awareness’ on test/acceptance builds. A feature like this on production is just killing reliability or GitHub actions for serious production work.

[mattlewis92]

We got hit by this brownout yesterday but nothing told us our jobs were getting cancelled because of the brownout, I found this issue purely by chance. If you're going to do something like this you really need to give an obvious error message and explain why jobs are being cancelled, just seeing the job being cancelled made me think there was an outage. Even our github account manager wasn't aware of this brownout that was happening.

[fcjz]

Also, last night, I also experienced this brownout , but there is no explanation, I feel a little baffling.

Too sloppy.

[ghost]

Don't get me wrong, I fully understand that you're deprecating 18.04 - but incorporating brownout periods to raise awareness?

If my team needs to get a build out for a critical bugfix on the 22nd of August... I'm just going to have to wait for 4 hours? If my understanding of that is true, I find that very odd behavior in an enterprise environment. Some businesses actually have SLAs with their clients. Would kindly request the team to find another way to "raise awareness". I came here because of the red banner when using the image, that worked fine as well and is a lot less invasive.

You're giving us two weeks to migrate away before brownouts start. Migrating is an action that requires testing and validation.

We had the exact same problem when windows-2017 was deprecated, and it was extremely painful to have this transition forced unto us.

IMHO this one is particularly bad since there are legitimate reasons to build on older versions of Ubuntu; namely, so that the linker won't pull in symbols from newer versions of glibc which the customer's systems don't have.

[miketimofeev]

The deprecation date has shifted to March 1st, 2023 to give more time to adapt the workflows to Ubuntu20 or Ubuntu22. The brownout schedule is also adjusted and now has shortened periods.

[dbussink]

The deprecation date has shifted to March 1st, 2023 to give more time to adapt the workflows to Ubuntu20 or Ubuntu22. The brownout schedule is also adjusted and now has shortened periods.

@miketimofeev Is there also a plan to make it much more obvious in the UI that the brownout is happening? The lack of any information in the view linked from pull requests made the user experience significantly worse here.

[miketimofeev]

The deprecation date has shifted to March 1st, 2023 to give more time to adapt the workflows to Ubuntu20 or Ubuntu22. The brownout schedule is also adjusted and now has shortened periods.

@miketimofeev Is there also a plan to make it much more obvious in the UI that the brownout is happening? The lack of any information in the view linked from pull requests made the user experience significantly worse here.

it's on our radar but I'm afraid we can't guarantee that better UI notifications will be implemented during this deprecation.

[miketimofeev]

We have decided to prolong the deprecation until the official end of life — April 2023. Brownouts schedule has been adjusted as well to reflect the prolonged deprecation

[JimmyBanks]

We have decided to prolong the deprecation until the official end of life — April 2023. Brownouts schedule has been adjusted as well to reflect the prolonged deprecation

Our project is using 'ubuntu-latest', but the pipelines continually default to 18.04 giving us this alert/error notice.

[miketimofeev]

@JimmyBanks could you share a few links to your pipelines with these warning and the screenshot of the Initialize job step with the runner image?

[JimmyBanks]

@JimmyBanks could you share a few links to your pipelines with these warning and the screenshot of the Initialize job step with the runner image?

Turns out this was a mistake on our end and the pipeline was set to 18.04, not latest. We mixed up two different projects settings, and assumed both were updated at the same time to latest.

Sorry for wasting your time

[rene-hermenau]

To raise awareness of the upcoming removal, we will temporarily fail jobs using Ubuntu 18.04.

This will explain why our jobs suddenly fail without doing any modifications. We are paying you a lot of money for CI minutes, so you should inform your customers properly in advance, keep the deprecated services running until expiration date instead of making your clients tests fail and rising their costs for extra CI minutes and development time.

[jkoooo4456]

Tlustoch

[sean-mcmanus]

Our data shows Ubuntu 18 usage is still close to Ubuntu 22 usage. Can you delay the deprecation/removal until more users have migrated off Ubuntu 18? The support for only 2 OS versions seems arbitrary and not driven by usage data?

[miketimofeev]

@sean-mcmanus we will support Ubuntu 18 until it reaches its end of life in April 2023

[awaters-pa]

This is happening today and today is not on your list....any advice? We are very aware of the deprecation and don't need our productivity on the needed changes to be blocked by an artificial hindrance.

[patrickcavli]

This incident occurred while running CI pipeline of our platform. At least they could have mentioned this before the scheduled brownout. Any way nice way to change deprecated dependencies!

[HSPDev]

Yeah wtf, this was not scheduled and on a friday... I was about to go home with a nice bugfixed shipped out....

[GeoFro]

October 14th is not on your brownout list.

[gibz104]

🤡

[awaters-pa]

Pretty slimy to update the schedule after you get complaints. This brownout should have been cancelled and not spring on everyone for a six hour window. This is completely irresponsible on your part @github

[OlegKarasik]

The best moment here is that jobs are get canceled and ... no notifications are send by email. We have just lost a few hours before realised that our build is not running and not propagating to environments. Thanks you very much, you increased my awareness of other platforms.

[alemuntoni]

Could you please at least change the error message and update to the right removal date?? Is it December 1st or April 1st? I hope you can confirm that it will be April 1st, because EOL of Ubuntu 18.04 is April 2023.

This is totally unprofessional.

[ollielowson-wcbs]

For posterity, @ddobranic / @github.

Before: https://web.archive.org/web/20221012145059/https://github.com/actions/runner-images/issues/6002

After:

[awaters-pa]

And the blog still isn't updated... :roll_eyes: https://github.blog/changelog/2022-08-09-github-actions-the-ubuntu-18-04-actions-runner-image-is-being-deprecated-and-will-be-removed-by-12-1-22/