search

actions / runner-images

GitHub Actions runner images
MIT License
9.36k stars 2.9k forks source link

The Ubuntu 18.04 Actions runner image will begin deprecation on 2022/08/08 and will be fully unsupported by 2023/04/03 #6002

Closed miketimofeev closed 1 year ago

miketimofeev commented 1 year ago

Breaking changes

We have started the deprecation process for Ubuntu 18.04. While the image is being deprecated, You may experience longer queue times during peak usage hours. Deprecation will begin on 2022/08/08 and the image will be fully unsupported by 2023/04/03

To raise awareness of the upcoming removal, we will temporarily fail jobs using Ubuntu 18.04. Builds that are scheduled to run during the brownout periods will fail. The brownouts are scheduled for the following dates and times:

GitHub Actions\Azure DevOps::

Target date

April 1st, 2023

The motivation for the changes

We maintain the latest two stable versions of any given OS version. Ubuntu 22.04 is going GA on 8/8/22 thus we start deprecating the oldest image.

Possible impact

Workflows using the ubuntu-18.04 image label should be updated to ubuntu-latest, ubuntu-20.04, or ubuntu-22.04

Platforms affected

Virtual environments affected

Mitigation ways

N\A

miketimofeev commented 1 year ago

@ollielowson-wcbs sorry, I'm not part of the team anymore and am not aware of the changes to the schedule. Please ping @ddobranic for clarification

ddobranic commented 1 year ago

@awaters-pa sorry, my bad. reverted to original dates.

awaters-pa commented 1 year ago

@awaters-pa sorry, my bad. reverted to original dates.

So when should we expect this unexpected brownout to end? @ddobranic

redaER7 commented 1 year ago

brownouts are starting... of course, with no notifications or alert...

pierskarsenbarg commented 1 year ago

Information on brownouts here: https://github.com/actions/runner-images/issues/6002

Also here: https://github.blog/changelog/2022-08-09-github-actions-the-ubuntu-18-04-actions-runner-image-is-being-deprecated-and-will-be-removed-by-12-1-22/

awaters-pa commented 1 year ago

Information on brownouts here: #6002

Why link back to this issue? The dates are wrong. They were changed and then changed back.

pierskarsenbarg commented 1 year ago

Because when I first saw the link to this issue it said that the brownouts were due today 🤷

mihaylovg commented 1 year ago

@ddobranic Only registered to let you know what happened today was not professional at all. My client was not able to deploy changes to production so they are not happy too.

manxorist commented 1 year ago

Please use a non-broken date format in the issue title: https://en.wikipedia.org/wiki/ISO_8601

aibaars commented 1 year ago

Out of curiosity is 4/1/2023 an English or an American date? Could you update the title to be unambiguous, please?

aalbacetef commented 1 year ago

Don't get me wrong, I fully understand that you're deprecating 18.04 - but incorporating brownout periods to raise awareness?

If my team needs to get a build out for a critical bugfix on the 22nd of August... I'm just going to have to wait for 4 hours? If my understanding of that is true, I find that very odd behavior in an enterprise environment. Some businesses actually have SLAs with their clients. Would kindly request the team to find another way to "raise awareness". I came here because of the red banner when using the image, that worked fine as well and is a lot less invasive.

You're giving us two weeks to migrate away before brownouts start. Migrating is an action that requires testing and validation.

Literally just ran into this. CI started failing due to the brownout while working on a bugfix.

dalboris commented 1 year ago

Pretty please, GitHub, would it be possible to stop with this "brownout periods" insanity? Some people are trying to work.

In order to raise awareness, I would recommend sending emails. As is, Ubuntu 18.04 is still supported and breaking builds is poor practice. At the very least add a button "Yes, I'm aware of the deprecation, please always run my Ubuntu 18.04 builds anyway from now on".

Note that the reason we're still using Ubuntu 18.04 is not that our organization is slow to migrate to newer versions. The reason is that we need to generate AppImages, and these AppImages must be generated on Ubuntu 18.04 until at least April 2023, and ideally even after that. Our build scripts work perfectly fine onubuntu-latest, but we have to build them on Ubuntu 18.04, otherwise the built binaries would link to versions of libc which are too new.

phlax commented 1 year ago

sorry - but this has to take the prize for unnecessarily alienating your customers

niagol commented 1 year ago

Can you at least adhere to your own timelines?

rossvz commented 1 year ago

Can you at least adhere to your own timelines?

Agreed. At the time of this comment, the original post states November 15, 18:00 UTC – November 15, 20:00 UTC but this brownout seemed to start at 20:00 UTC

dalboris commented 1 year ago

It's 21:02 UTC, builds are still failing. Oh, and they did fail already soon after 18:00 UTC. The only reasonable course of action for GitHub is to cancel all brownouts periods until an opt-out mechanism is implemented, this is simply not acceptable.

sjaeckel commented 1 year ago

IMO your decision logic on the helpfulness of these brownout tests is flawed.

The assumption that this leads to fewer projects breaking sometime in the future when a specific runner is deprecated won't work, as this brownout mechanism is only triggered for "very active" projects. If a project is e.g. feature complete and has not many rebuilds, chances are high that this will not help at all and builds will only break sometime in the future after the runner was deprecated.

Instead you're hurting all projects that maybe have a good reason to still build against "soon to be deprecated runners" as already indicated by many users.

[...] Ubuntu 18.04 is still supported and breaking builds is poor practice. At the very least add a button "Yes, I'm aware of the deprecation, please always run my Ubuntu 18.04 builds anyway from now on".

^ this

The only reasonable course of action for GitHub is to cancel all brownouts periods until an opt-out mechanism is implemented, this is simply not acceptable.

^ and this

hosseinmirzae commented 1 year ago

Breaking changes

We have started the deprecation process for Ubuntu 18.04. While the image is being deprecated, You may experience longer queue times during peak usage hours. Deprecation will begin on 2022/08/08 and the image will be fully unsupported by 2023/04/01

To raise awareness of the upcoming removal, we will temporarily fail jobs using Ubuntu 18.04. Builds that are scheduled to run during the brownout periods will fail. The brownouts are scheduled for the following dates and times:

GitHub Actions\Azure DevOps::

  • October 3, 12:00 UTC – October 3, 14:00 UTC
  • October 18, 14:00 UTC – October 18, 16:00 UTC
  • November 15, 18:00 UTC – November 15, 20:00 UTC
  • November 30, 20:00 UTC – November 30, 22:00 UTC
  • December 15, 20:00 UTC - December 16 00:00 UTC
  • January 5, 10.00 UTC - January 5, 14.00 UTC
  • January 13, 12.00 UTC - January 13, 16.00 UTC
  • January 18, 14.00 UTC - January 18, 18.00 UTC
  • January 24, 16.00 UTC - January 24, 20.00 UTC
  • February 1, 18.00 UTC - February 1, 22.00 UTC
  • February 7, 16.00 UTC - February 7, 22.00 UTC
  • February 13, 14.00 UTC - February 13, 22.00 UTC
  • February 21, 10.00 UTC - February 21, 22.00 UTC
  • February 28, 10.00 UTC - February 28, 22.00 UTC
  • March 6, 00.00 UTC - March 7, 00.00 UTC
  • March 13, 00.00 UTC - March 14, 00.00 UTC
  • March 21, 00.00 UTC - March 22, 00.00 UTC

Target date

April 1st, 2023

The motivation for the changes

We maintain the latest two stable versions of any given OS version. Ubuntu 22.04 is going GA on 8/8/22 thus we start deprecating the oldest image.

Possible impact

Workflows using the ubuntu-18.04 image label should be updated to ubuntu-latest, ubuntu-20.04, or ubuntu-22.04

Platforms affected

  • [x] Azure DevOps
  • [x] GitHub Actions

Virtual environments affected

  • [x] Ubuntu 18.04
  • [x] Ubuntu 20.04
  • [x] Ubuntu 22.04
  • [x] macOS 10.15
  • [x] macOS 11
  • [x] macOS 12
  • [x] Windows Server 2019
  • [x] Windows Server 2022

Mitigation ways

N\A

gh pr checkout 81

dalboris commented 1 year ago

For anyone interested, here is a workaround to keep being able to run actions on Ubuntu 18.04 during the brownout periods, or after the deprecation. The idea is to run the action within a Ubuntu 18.04 Docker container, within a ubuntu-latest GitHub self-hosted runner.

Here is a minimal example compiling a C++/CMake project (see ubuntu-18.04-docker-minimal.yml for a live working example).

name: Ubuntu 18.04 (Docker Minimal)
on: [push, pull_request]
jobs:
  build:
    runs-on: ubuntu-latest
    container: ubuntu:18.04
    steps:

    - uses: actions/checkout@v2

    - name: Install Dependencies
      run: |
        apt-get update
        apt-get install -y cmake build-essential

    - name: Build
      working-directory: ${{github.workspace}}
      run: |
        mkdir build
        cd build
        cmake ..
        cmake --build .

Note that actions/checkout@v2 requires git 2.18+ to "properly work", and the default version from the ubuntu:18.04 Docker image is only 2.17. So unless you manually install a newer git, actions/checkout@v2 will simply download the code as is, instead of doing a git clone, and you won't have access to the repo history. In order to install a newer git and get a real git clone, you add the following before the actions/checkout@v2 step:

    - name: Install and Configure Recent Git
      run: |
        apt-get update
        apt-get install -y software-properties-common
        add-apt-repository ppa:git-core/ppa
        apt-get update
        apt-get install -y git
        git config --global --add safe.directory `pwd` # Fix "fatal: detected dubious ownership in repository at '/__w/<user>/<repo>'""

Also, you probably want a newer version of CMake (default is 3.10), which you can do with:

    - name: Install Recent CMake
      run: |
        apt-get install -y gpg wget
        wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | gpg --dearmor - | tee /etc/apt/trusted.gpg.d/kitware.gpg >/dev/null
        apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 6AF7F09730B3F0A4
        apt-add-repository "deb https://apt.kitware.com/ubuntu/ bionic main"
        apt-get update
        apt-get install -y cmake

I hope this helps.

By the way, @mikhailkoliada @ddobranic , since this comment will soon be buried in the middle of the 700+-folded comments, feel free to copy this workaround (or link to it) in the original message: it will probably frustrate GitHub users less if they see a workaround at the top of this thread. I sure would have loved to see it instead of spending hours getting something to work.

sgorugantu commented 1 year ago

Even though we are using "windows-latest" as our Vm Image, we are still getting error as "This is a scheduled Ubuntu-18.04 brownout" and this is impacting our pipelines.

kyrofa commented 1 year ago

Interesting strategy to make sure folks were notified! May I suggest, however, a less hair-pulling message on this page? I re-triggered several times and checked the github status before I finally visited the summary page, which actually contained a very helpful message.

Screenshot from 2023-02-07 08-46-57

phlax commented 1 year ago

as if to emphasize what an incredibly bad idea this is without any kind of opt out you have done this in the middle of an openssl patch release when projects need to run CI to publish their projects

aligajani commented 1 year ago

I do not think it is a good idea to prevent Github actions from running due to a brownout. You could notify us in a different way, or warn us using some alerts, but preventing a critical push due to this issue isn't a pleasing Developer Experience.

joshforbes commented 1 year ago

I do not think a "brownout" is a good strategy for this. Quite frustrating to be honest. We don't want to move off of 18.04, our plan is to move to a container before the deadline. However, it would have been nice to continue operations uninterrupted until the date.

AleksandarTokarev commented 1 year ago

I agree with the comments above - brownouts are not the best way to do things - it would be better to have notifications or emails sent with warnings instead.

probonopd commented 1 year ago

the image will be fully unsupported by 2023/04/0

Will it become unsupported or unavailable? I hope the former, since I'd like to continue building for it.

CALIFAS95 commented 1 year ago

Breaking changes

We have started the deprecation process for Ubuntu 18.04. While the image is being deprecated, You may experience longer queue times during peak usage hours. Deprecation will begin on 2022/08/08 and the image will be fully unsupported by 2023/04/01

To raise awareness of the upcoming removal, we will temporarily fail jobs using Ubuntu 18.04. Builds that are scheduled to run during the brownout periods will fail. The brownouts are scheduled for the following dates and times:

GitHub Actions\Azure DevOps::

  • October 3, 12:00 UTC – October 3, 14:00 UTC
  • October 18, 14:00 UTC – October 18, 16:00 UTC
  • November 15, 18:00 UTC – November 15, 20:00 UTC
  • November 30, 20:00 UTC – November 30, 22:00 UTC
  • December 15, 20:00 UTC – December 16 00:00 UTC
  • January 5, 10.00 UTC – January 5, 14.00 UTC
  • January 13, 12.00 UTC – January 13, 16.00 UTC
  • January 18, 14.00 UTC – January 18, 18.00 UTC
  • January 24, 16.00 UTC – January 24, 20.00 UTC
  • February 7, 16.00 UTC – February 7, 22.00 UTC
  • February 21, 10.00 UTC – February 21, 22.00 UTC
  • March 6, 00.00 UTC – March 7, 00.00 UTC
  • March 13, 00.00 UTC – March 14, 00.00 UTC
  • March 21, 00.00 UTC – March 22, 00.00 UTC
  • March 28, 00.00 UTC – March 29, 00.00 UTC

Target date

April 1st, 2023

The motivation for the changes

We maintain the latest two stable versions of any given OS version. Ubuntu 22.04 is going GA on 8/8/22 thus we start deprecating the oldest image.

Possible impact

Workflows using the ubuntu-18.04 image label should be updated to ubuntu-latest, ubuntu-20.04, or ubuntu-22.04

Platforms affected

  • [x] Azure DevOps
  • [x] GitHub Actions

Virtual environments affected

  • [x] Ubuntu 18.04
  • [ ] Ubuntu 20.04
  • [ ] Ubuntu 22.04
  • [ ] macOS 10.15
  • [ ] macOS 11
  • [ ] macOS 12
  • [ ] Windows Server 2019
  • [ ] Windows Server 2022

Mitigation ways

N\A

dirkf commented 1 year ago

How do we run tests against old versions of packages not offered in later Ubuntu images?

kyrofa commented 1 year ago

Consider using docker, @dirkf.

GKOLKO commented 1 year ago

Hello, I am facing issue with the new versions of Ubunto (20, 22 and latest) I have this error each time: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-resources-plugin:3.3.0:resources (default-resources) on project ebx-technip-module: filtering /home/vsts/work/1/s/mdm-technip/ebx-technip-module/src/main/resources/resources.lnk to /home/vsts/work/1/s/mdm-technip/ebx-technip-module/src/main/webapp/WEB-INF/classes/resources.lnk failed with MalformedInputException: Input length = 1 -> [Help 1] How can I solve this issue?

MikeMcC399 commented 1 year ago

@GKOLKO

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-resources-plugin:3.3.0:resources How can I solve this issue?

This does not sound like an error with the GitHub runner. I suggest you search instead on https://stackoverflow.com/.

nahimmedvent commented 1 year ago

Ubuntu 20.04 and 22.04 don't work at all for me, I keep getting the error /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /home/site/wwwroot/.python_packages/lib/site-packages/pyodbc.cpython-37m-x86_64-linux-gnu.so)
Because these versions run on 2.29 glibc

mikhailkoliada commented 1 year ago

@nahimmedvent software on Ubuntu 20.04 and 22.04 is compiled with newer version of libc, which makes the software compiled with older version of libs simply incompatible. There is no straightforward way tp solve this problem as it is a bit non-trivial. The easiest solution would be to switch to ubuntu:18.04 in docker.

lakshyagithub commented 1 year ago

This will happen on April fools 🤣

mikhailkoliada commented 1 year ago

Ubuntu-18.04 is deprecated and disabled for all the customers

Abduljakulsalsal commented 1 year ago

✌️

eyalroz commented 1 year ago

Well, you could at least provide a proper notice about Ubuntu 18.04 runners no longer being available, instead of having actions fail mysteriously...