Closed miketimofeev closed 1 year ago
@ollielowson-wcbs sorry, I'm not part of the team anymore and am not aware of the changes to the schedule. Please ping @ddobranic for clarification
@awaters-pa sorry, my bad. reverted to original dates.
@awaters-pa sorry, my bad. reverted to original dates.
So when should we expect this unexpected brownout to end? @ddobranic
brownouts are starting... of course, with no notifications or alert...
Information on brownouts here: #6002
Why link back to this issue? The dates are wrong. They were changed and then changed back.
Because when I first saw the link to this issue it said that the brownouts were due today 🤷
@ddobranic Only registered to let you know what happened today was not professional at all. My client was not able to deploy changes to production so they are not happy too.
Please use a non-broken date format in the issue title: https://en.wikipedia.org/wiki/ISO_8601
Out of curiosity is 4/1/2023
an English or an American date? Could you update the title to be unambiguous, please?
Don't get me wrong, I fully understand that you're deprecating 18.04 - but incorporating brownout periods to raise awareness?
If my team needs to get a build out for a critical bugfix on the 22nd of August... I'm just going to have to wait for 4 hours? If my understanding of that is true, I find that very odd behavior in an enterprise environment. Some businesses actually have SLAs with their clients. Would kindly request the team to find another way to "raise awareness". I came here because of the red banner when using the image, that worked fine as well and is a lot less invasive.
You're giving us two weeks to migrate away before brownouts start. Migrating is an action that requires testing and validation.
Literally just ran into this. CI started failing due to the brownout while working on a bugfix.
Pretty please, GitHub, would it be possible to stop with this "brownout periods" insanity? Some people are trying to work.
In order to raise awareness, I would recommend sending emails. As is, Ubuntu 18.04 is still supported and breaking builds is poor practice. At the very least add a button "Yes, I'm aware of the deprecation, please always run my Ubuntu 18.04 builds anyway from now on".
Note that the reason we're still using Ubuntu 18.04 is not that our organization is slow to migrate to newer versions. The reason is that we need to generate AppImages, and these AppImages must be generated on Ubuntu 18.04 until at least April 2023, and ideally even after that. Our build scripts work perfectly fine onubuntu-latest
, but we have to build them on Ubuntu 18.04, otherwise the built binaries would link to versions of libc which are too new.
sorry - but this has to take the prize for unnecessarily alienating your customers
Can you at least adhere to your own timelines?
Can you at least adhere to your own timelines?
Agreed.
At the time of this comment, the original post states November 15, 18:00 UTC – November 15, 20:00 UTC
but this brownout seemed to start at 20:00 UTC
It's 21:02 UTC, builds are still failing. Oh, and they did fail already soon after 18:00 UTC. The only reasonable course of action for GitHub is to cancel all brownouts periods until an opt-out mechanism is implemented, this is simply not acceptable.
IMO your decision logic on the helpfulness of these brownout tests is flawed.
The assumption that this leads to fewer projects breaking sometime in the future when a specific runner is deprecated won't work, as this brownout mechanism is only triggered for "very active" projects. If a project is e.g. feature complete and has not many rebuilds, chances are high that this will not help at all and builds will only break sometime in the future after the runner was deprecated.
Instead you're hurting all projects that maybe have a good reason to still build against "soon to be deprecated runners" as already indicated by many users.
[...] Ubuntu 18.04 is still supported and breaking builds is poor practice. At the very least add a button "Yes, I'm aware of the deprecation, please always run my Ubuntu 18.04 builds anyway from now on".
^ this
The only reasonable course of action for GitHub is to cancel all brownouts periods until an opt-out mechanism is implemented, this is simply not acceptable.
^ and this
Breaking changes
We have started the deprecation process for Ubuntu 18.04. While the image is being deprecated, You may experience longer queue times during peak usage hours. Deprecation will begin on 2022/08/08 and the image will be fully unsupported by 2023/04/01
To raise awareness of the upcoming removal, we will temporarily fail jobs using Ubuntu 18.04. Builds that are scheduled to run during the brownout periods will fail. The brownouts are scheduled for the following dates and times:
GitHub Actions\Azure DevOps::
- October 3, 12:00 UTC – October 3, 14:00 UTC
- October 18, 14:00 UTC – October 18, 16:00 UTC
- November 15, 18:00 UTC – November 15, 20:00 UTC
- November 30, 20:00 UTC – November 30, 22:00 UTC
- December 15, 20:00 UTC - December 16 00:00 UTC
- January 5, 10.00 UTC - January 5, 14.00 UTC
- January 13, 12.00 UTC - January 13, 16.00 UTC
- January 18, 14.00 UTC - January 18, 18.00 UTC
- January 24, 16.00 UTC - January 24, 20.00 UTC
- February 1, 18.00 UTC - February 1, 22.00 UTC
- February 7, 16.00 UTC - February 7, 22.00 UTC
- February 13, 14.00 UTC - February 13, 22.00 UTC
- February 21, 10.00 UTC - February 21, 22.00 UTC
- February 28, 10.00 UTC - February 28, 22.00 UTC
- March 6, 00.00 UTC - March 7, 00.00 UTC
- March 13, 00.00 UTC - March 14, 00.00 UTC
- March 21, 00.00 UTC - March 22, 00.00 UTC
Target date
April 1st, 2023
The motivation for the changes
We maintain the latest two stable versions of any given OS version. Ubuntu 22.04 is going GA on 8/8/22 thus we start deprecating the oldest image.
Possible impact
Workflows using the
ubuntu-18.04
image label should be updated toubuntu-latest
,ubuntu-20.04
, orubuntu-22.04
Platforms affected
- [x] Azure DevOps
- [x] GitHub Actions
Virtual environments affected
- [x] Ubuntu 18.04
- [x] Ubuntu 20.04
- [x] Ubuntu 22.04
- [x] macOS 10.15
- [x] macOS 11
- [x] macOS 12
- [x] Windows Server 2019
- [x] Windows Server 2022
Mitigation ways
N\A
gh pr checkout 81
For anyone interested, here is a workaround to keep being able to run actions on Ubuntu 18.04 during the brownout periods, or after the deprecation. The idea is to run the action within a Ubuntu 18.04 Docker container, within a ubuntu-latest
GitHub self-hosted runner.
Here is a minimal example compiling a C++/CMake project (see ubuntu-18.04-docker-minimal.yml for a live working example).
name: Ubuntu 18.04 (Docker Minimal)
on: [push, pull_request]
jobs:
build:
runs-on: ubuntu-latest
container: ubuntu:18.04
steps:
- uses: actions/checkout@v2
- name: Install Dependencies
run: |
apt-get update
apt-get install -y cmake build-essential
- name: Build
working-directory: ${{github.workspace}}
run: |
mkdir build
cd build
cmake ..
cmake --build .
Note that actions/checkout@v2
requires git 2.18+ to "properly work", and the default version from the ubuntu:18.04
Docker image is only 2.17. So unless you manually install a newer git, actions/checkout@v2
will simply download the code as is, instead of doing a git clone
, and you won't have access to the repo history. In order to install a newer git and get a real git clone, you add the following before the actions/checkout@v2
step:
- name: Install and Configure Recent Git
run: |
apt-get update
apt-get install -y software-properties-common
add-apt-repository ppa:git-core/ppa
apt-get update
apt-get install -y git
git config --global --add safe.directory `pwd` # Fix "fatal: detected dubious ownership in repository at '/__w/<user>/<repo>'""
Also, you probably want a newer version of CMake (default is 3.10), which you can do with:
- name: Install Recent CMake
run: |
apt-get install -y gpg wget
wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | gpg --dearmor - | tee /etc/apt/trusted.gpg.d/kitware.gpg >/dev/null
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 6AF7F09730B3F0A4
apt-add-repository "deb https://apt.kitware.com/ubuntu/ bionic main"
apt-get update
apt-get install -y cmake
I hope this helps.
By the way, @mikhailkoliada @ddobranic , since this comment will soon be buried in the middle of the 700+-folded comments, feel free to copy this workaround (or link to it) in the original message: it will probably frustrate GitHub users less if they see a workaround at the top of this thread. I sure would have loved to see it instead of spending hours getting something to work.
Even though we are using "windows-latest" as our Vm Image, we are still getting error as "This is a scheduled Ubuntu-18.04 brownout" and this is impacting our pipelines.
Interesting strategy to make sure folks were notified! May I suggest, however, a less hair-pulling message on this page? I re-triggered several times and checked the github status before I finally visited the summary page, which actually contained a very helpful message.
as if to emphasize what an incredibly bad idea this is without any kind of opt out you have done this in the middle of an openssl patch release when projects need to run CI to publish their projects
I do not think it is a good idea to prevent Github actions from running due to a brownout. You could notify us in a different way, or warn us using some alerts, but preventing a critical push due to this issue isn't a pleasing Developer Experience.
I do not think a "brownout" is a good strategy for this. Quite frustrating to be honest. We don't want to move off of 18.04, our plan is to move to a container before the deadline. However, it would have been nice to continue operations uninterrupted until the date.
I agree with the comments above - brownouts are not the best way to do things - it would be better to have notifications or emails sent with warnings instead.
the image will be fully unsupported by 2023/04/0
Will it become unsupported or unavailable? I hope the former, since I'd like to continue building for it.
Breaking changes
We have started the deprecation process for Ubuntu 18.04. While the image is being deprecated, You may experience longer queue times during peak usage hours. Deprecation will begin on 2022/08/08 and the image will be fully unsupported by 2023/04/01
To raise awareness of the upcoming removal, we will temporarily fail jobs using Ubuntu 18.04. Builds that are scheduled to run during the brownout periods will fail. The brownouts are scheduled for the following dates and times:
GitHub Actions\Azure DevOps::
- October 3, 12:00 UTC – October 3, 14:00 UTC
- October 18, 14:00 UTC – October 18, 16:00 UTC
- November 15, 18:00 UTC – November 15, 20:00 UTC
- November 30, 20:00 UTC – November 30, 22:00 UTC
- December 15, 20:00 UTC – December 16 00:00 UTC
- January 5, 10.00 UTC – January 5, 14.00 UTC
- January 13, 12.00 UTC – January 13, 16.00 UTC
- January 18, 14.00 UTC – January 18, 18.00 UTC
- January 24, 16.00 UTC – January 24, 20.00 UTC
- February 7, 16.00 UTC – February 7, 22.00 UTC
- February 21, 10.00 UTC – February 21, 22.00 UTC
- March 6, 00.00 UTC – March 7, 00.00 UTC
- March 13, 00.00 UTC – March 14, 00.00 UTC
- March 21, 00.00 UTC – March 22, 00.00 UTC
- March 28, 00.00 UTC – March 29, 00.00 UTC
Target date
April 1st, 2023
The motivation for the changes
We maintain the latest two stable versions of any given OS version. Ubuntu 22.04 is going GA on 8/8/22 thus we start deprecating the oldest image.
Possible impact
Workflows using the
ubuntu-18.04
image label should be updated toubuntu-latest
,ubuntu-20.04
, orubuntu-22.04
Platforms affected
- [x] Azure DevOps
- [x] GitHub Actions
Virtual environments affected
- [x] Ubuntu 18.04
- [ ] Ubuntu 20.04
- [ ] Ubuntu 22.04
- [ ] macOS 10.15
- [ ] macOS 11
- [ ] macOS 12
- [ ] Windows Server 2019
- [ ] Windows Server 2022
Mitigation ways
N\A
How do we run tests against old versions of packages not offered in later Ubuntu images?
Consider using docker, @dirkf.
Hello, I am facing issue with the new versions of Ubunto (20, 22 and latest) I have this error each time: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-resources-plugin:3.3.0:resources (default-resources) on project ebx-technip-module: filtering /home/vsts/work/1/s/mdm-technip/ebx-technip-module/src/main/resources/resources.lnk to /home/vsts/work/1/s/mdm-technip/ebx-technip-module/src/main/webapp/WEB-INF/classes/resources.lnk failed with MalformedInputException: Input length = 1 -> [Help 1] How can I solve this issue?
@GKOLKO
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-resources-plugin:3.3.0:resources How can I solve this issue?
This does not sound like an error with the GitHub runner. I suggest you search instead on https://stackoverflow.com/.
Ubuntu 20.04 and 22.04 don't work at all for me, I keep getting the error /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /home/site/wwwroot/.python_packages/lib/site-packages/pyodbc.cpython-37m-x86_64-linux-gnu.so)
Because these versions run on 2.29 glibc
@nahimmedvent software on Ubuntu 20.04 and 22.04 is compiled with newer version of libc, which makes the software compiled with older version of libs simply incompatible. There is no straightforward way tp solve this problem as it is a bit non-trivial. The easiest solution would be to switch to ubuntu:18.04 in docker.
This will happen on April fools 🤣
Ubuntu-18.04 is deprecated and disabled for all the customers
✌️
Well, you could at least provide a proper notice about Ubuntu 18.04 runners no longer being available, instead of having actions fail mysteriously...
Breaking changes
We have started the deprecation process for Ubuntu 18.04. While the image is being deprecated, You may experience longer queue times during peak usage hours. Deprecation will begin on 2022/08/08 and the image will be fully unsupported by 2023/04/03
To raise awareness of the upcoming removal, we will temporarily fail jobs using Ubuntu 18.04. Builds that are scheduled to run during the brownout periods will fail. The brownouts are scheduled for the following dates and times:
GitHub Actions\Azure DevOps::
Target date
April 1st, 2023
The motivation for the changes
We maintain the latest two stable versions of any given OS version. Ubuntu 22.04 is going GA on 8/8/22 thus we start deprecating the oldest image.
Possible impact
Workflows using the
ubuntu-18.04
image label should be updated toubuntu-latest
,ubuntu-20.04
, orubuntu-22.04
Platforms affected
Virtual environments affected
Mitigation ways
N\A