actions / runner-images

GitHub Actions runner images
MIT License
9.17k stars 2.84k forks source link

windows-latest workflows will use Windows Server 2022 #4856

Closed miketimofeev closed 2 years ago

miketimofeev commented 2 years ago

Breaking changes

Windows Server 2022 is ready to be the default version for the "windows-latest" label in GitHub Actions and Azure DevOps.

Target date

This change will be rolled out over a period of several weeks beginning on January, 17. We plan to complete the migration by March, 6.

The motivation for the changes

Starting from November 2021 Windows Server 2022 with Visual Studio 2022 image is generally available for all customers. We have monitored customer feedback to improve the Windows Server 2022 image stability and now we are ready to set it as the latest.

Possible impact

⚠️ CodeQL does not currently support analysis of compiled languages on Windows Server 2022 ⚠️ Support for Windows Server 2022 has been added.

The Windows Server 2022 image has a different set of software than Windows Server 2019. The most significant changes are listed in the table below: Tool name Windows-2019 Windows-2022 Notes
Go lang 1.15.* 1.16.* Go lang version bumped to the more recent one
Python 3.7.* 3.9.* Python version bumped to the more recent one
Ruby 2.5.* 3.0.* Ruby version bumped to the more recent one
Google Cloud SDK latest - The software was removed due to maintenance concerns, consider using the official action https://github.com/google-github-actions/setup-gcloud
Innosetup latest - The software was removed due to its low popularity and maintenance concerns
Parcel latest - The software was removed due to its low popularity and maintenance concerns
Cloud Foundry CLI latest - The software was removed due to its low popularity and maintenance concerns
Java 8(default)
11
13
17
8(default)
11
17
Due to our software guidelines we left only LTS versions on the image. If you need to use other versions of Java, consider using setup-java and Java tool installer tasks
Python toolcache 2.7
3.5
3.6
3.7
3.8
3.9
3.10
3.7
3.8
3.9
3.10
EOL python versions were removed
PyPy toolcache 2.7
3.6
3.7
3.8
2.7
3.7
3.8
Version 3.6 was deprecated as it won't receive any updates
Ruby toolcache 2.4
2.5(default)
2.6
2.7
3.0
2.7
3.0(default)
Left only two recent versions
MySQL 5.7.* 8.0.* The version was updated to the latest major one
.NET 2.1
3.1
5.0
3.1
5.0
Version 2.1 is EOL and was removed
Windows Driver Kit 10.0.* - VS2022 is not yet supported by the WDK
Az powershell module 1.0.0.zip
1.6.0.zip
2.3.2.zip
2.6.0.zip
3.1.0.zip
3.5.0.zip
3.8.0.zip
4.3.0.zip
4.4.0.zip
4.7.0.zip
5.5.0.zip
5.9.0.zip
6.5.0
6.5 Left only the most recent 6.* release
Android SDK build-tools and platforms 19 - 32 27 - 32 Android Platforms and build tools < 27.0 were removed from image. Consider installation in runtime using sdkmanager

Virtual environments affected

Mitigation ways

If you see any issues with your workflows during this transition period:

tt2468 commented 2 years ago

Are there any known actions that we can use to install innosetup?

iAkashPattnaik commented 2 years ago

Are there any known actions that we can use to install innosetup?

Nah, I don't think that's possible from actions.

miketimofeev commented 2 years ago

@tt2468 innosetup can be installed relatively fast in runtime. Please feel free to ask for help if needed

MellyDawgKY commented 2 years ago

Why this was not done on the day of formally releasing visual studio 2022?

I always assumed Microsoft would follow the literal definition of 'latest' - silly me.

neman commented 2 years ago
image

What about LTS .NET 6.0?

miketimofeev commented 2 years ago

@neman 6.0 is bundled in VS2022, so it's available on the image but not installed via part of .NET installation script

remyjette commented 2 years ago

I'm seeing this warning

[warning]windows-latest pipelines will use windows-2022 soon. For more details, see https://github.com/actions/virtual-environments/issues/4856

In all of my Windows pipelines, but none of my pipelines use windows-latest - they all use windows-2019.

mikeKuester commented 2 years ago

Why this was not done on the day of formally releasing visual studio 2022?

Same for me! mairaw mentioned the issue I opend. I think that the name "windows-latest" is totally misleading. Something like "..-stable" would fit much better to the reality.

miketimofeev commented 2 years ago

I'm seeing this warning

[warning]windows-latest pipelines will use windows-2022 soon. For more details, see #4856

In all of my Windows pipelines, but none of my pipelines use windows-latest - they all use windows-2019.

Sorry for the inconvenience, this is a known issue and will be fixed next week.

catthehacker commented 2 years ago

Something like "..-stable" would fit much better to the reality.

-stable implies that it is more "stable" than other windows-* labels, which is not and would be even more misleading

mikeKuester commented 2 years ago

Ok, good point, do you have a better idea?

With "-latest" I expect the "latest bits and bytes". Not that several month after the release of the actual .NET 6 it's not included. :(

catthehacker commented 2 years ago

the label windows-* pertains to Windows version, not software it includes since windows-2019 is Windows Server 2019 and windows-2022 is Windows Server 2022, naturally, windows-latest is latest GA version of Windows Server

stap123 commented 2 years ago

FWIW

I agree windows-latest implies the most recent GA versions of stuff. This would mean that as soon as windows-2022 was GA then the -latest version should have switched over the same day (or close to it) not 6 months later (windows 2022 was GA in Sep).

When you take into consideration that .NET 6 was not available on the -2019 image on release day either it was a bit frustrating being encouraged to update to the latest .NET asap and then having to find out that it actually required changes to pipelines because -latest doesn't mean latest 🥲 Acknowledging that the solution was to pin to -2022 so not the end of the world it still meant confusion and frustration when the expectation was for the latest bits to be available.

There was a similar thing with Azure Functions last year with a many month lag between .NET 5 and support for it in functions. They worked to ensure that .NET 6 was supported on day 1 when that came out in Nov. Maybe there's a lesson to be learned there?

@mikeKuester Another possible option would be to simply not offer a windows-latest and only offer pinned versions then people would have to take control of their own versions. I accept that makes it slightly more painful for people managing windows versions and many pipelines but it does remove the ambiguity.

catthehacker commented 2 years ago

I agree windows-latest implies the most recent GA versions of stuff. This would mean that as soon as windows-2022 was GA then the -latest version should have switched over the same day (or close to it) not 6 months later (windows 2022 was GA in Sep).

It takes time to work out all the quirks of new software when it's released to not break environment. Switching over to windows-2022 on day 0 would break a lot of pipelines/workflows as well as it would require for GHA team to have it available on that time in Azure environment. I'll admit it is my fault that I didn't put * with clarification that latest OS requires additional work, thorough testing and customer feedback before it can be released "in the wild" and all that combined takes time. What I originally meant by GA is GA in sense of OS vendor and GitHub Actions.

When you take into consideration that .NET 6 was not available on the -2019 image on release day either it was a bit frustrating being encouraged to update to the latest .NET asap and then having to find out that it actually required changes to pipelines because -latest doesn't mean latest

The correct way is to use dotnet installation task (ADO | GHA) and not rely on specific bits installed on the host as much as possible.

stap123 commented 2 years ago

It takes time to work out all the quirks of new software when it's released to not break environment. Switching over to windows-2022 on day 0 would break a lot of pipelines/workflows as well as it would require for GHA team to have it available on that time in Azure environment. I'll admit it is my fault that I didn't put * with clarification that latest OS requires additional work, thorough testing and customer feedback before it can be released "in the wild" and all that combined takes time. What I originally meant by GA is GA in sense of OS vendor and GitHub Actions.

Totally understand it takes time to ensure everything works it was more an observation of the fact that -latest implies a little risk because you're asking for the latest 😃 Thanks for the clarification though.

The correct way is to use dotnet installation task (ADO | GHA) and not rely on specific bits installed on the host as much as possible.

That's a fair point, on the flip side would it not make sense to have no .NET versions available on the image by default so that it's not possible for people to do it wrong? This would force people down the happy path when creating pipelines.

Keep up the good work 🍾

MarcoWesterink commented 2 years ago

@neman 6.0 is bundled in VS2022, so it's available on the image but not installed via part of .NET installation script

Still might be good to add to the table. It confused me too.

catthehacker commented 2 years ago

That's a fair point, on the flip side would it not make sense to have no .NET versions available on the image by default so that it's not possible for people to do it wrong? This would force people down the happy path when creating pipelines.

Some versions of software that can be installed through actions/setup-* is kept in the image because of popularity (of version and not only) and (not sure if all since I didn't check all actions but) setup-* tasks will match against requested version and if it matches, it will skip installation (e.g. task requires version '5.0.x', '5.0.100' is installed - skips installation). I also don't want to speak as something given due to the fact I'm just a random contributor, so it's better to take GH Actions team words for mine but I do remember .NET being nightmare to maintain due to various issues (like installation time which I tried to help mitigate but for Ubuntu, ref: https://github.com/actions/virtual-environments/discussions/2367).

Totally understand it takes time to ensure everything works it was more an observation of the fact that -latest implies a little risk because you're asking for the latest

Another possible option would be to

use windows-current, it doesn't imply any (in)stability and/or version of software installed on the runner although, I'm sure if that was or were to be used as the name it would confuse new/old users "why it's not latest anymore" and some could find current as inadequate choice of word for the tag (somehow), nature of language and conveying message in as few words as possible

stap123 commented 2 years ago

use windows-current, it doesn't imply any (in)stability and/or version of software installed on the runner although, I'm sure if that was or were to be used as the name it would confuse new/old users "why it's not latest anymore" and some could find current as inadequate choice of word for the tag (somehow), nature of language and conveying message in as few words as possible

Naming is hard for sure 🤣

Tyler887 commented 2 years ago

@tt2468 innosetup can be installed relatively fast in runtime. Please feel free to ask for help if needed

Would have run choco install innosetup.

DominusExult commented 2 years ago

@tt2468 innosetup can be installed relatively fast in runtime. Please feel free to ask for help if needed

Would have run choco install innosetup.

Is this what @tt2468 meant?

jedieaston commented 2 years ago

There are Windows SDKs missing from the VS 2022 installation that were present in the 2019 image, are we to assume these won't be added to this image? I could install them at runtime but that takes a while :(

jsoref commented 2 years ago

Personally, I'd suggest -default, although I'd be totally happy w/ -current.

-latest is clearly a misnomer.

micdenny commented 2 years ago

Can we disable the warning without the need to update the pipeline yml to stick on windows-2022?

miketimofeev commented 2 years ago

@micdenny do you use AzureDevOps? If yes — I'm afraid it's not possible at the moment.

micdenny commented 2 years ago

@miketimofeev yes we use azuredevops, btw thank you for the reply.

drwill-ms commented 2 years ago

Our existing pipeline, updated specifically to windows-2022 and VS 2022 now has an error that sn.exe can't be found trying to disable strong name validation. Suggestions?

##[debug]INPUT_SCRIPT: 'call "C:\\Program Files (x86)\\Microsoft Visual Studio\\2022\\Enterprise\\Common7\\Tools\\VsDevCmd.bat"
##[debug]sn -Vr *,31bf3856ad364e35'
##[debug]INPUT_WORKINGDIRECTORY: 'D:\a\1\s'
##[debug]Asserting container path exists: 'D:\a\1\s'
Generating script.
##[debug]AGENT_VERSION: '2.198.2'
##[debug]AGENT_TEMPDIRECTORY: 'D:\a\_temp'
##[debug]Asserting container path exists: 'D:\a\_temp'
##[debug]Asserting leaf path exists: 'C:\Windows\system32\cmd.exe'
========================== Starting Command Output ===========================
##[debug]Entering Invoke-VstsTool.
##[debug] Arguments: '/D /E:ON /V:OFF /S /C "CALL "D:\a\_temp\304ba344-c007-433b-8fad-d7ce519fe037.cmd""'
##[debug] FileName: 'C:\Windows\system32\cmd.exe'
##[debug] WorkingDirectory: 'D:\a\1\s'
"C:\Windows\system32\cmd.exe" /D /E:ON /V:OFF /S /C "CALL "D:\a\_temp\304ba344-c007-433b-8fad-d7ce519fe037.cmd""
The system cannot find the path specified.
'sn' is not recognized as an internal or external command,
operable program or batch file.
##[debug]Exit code: 1
##[debug]Leaving Invoke-VstsTool.
##[error]Cmd.exe exited with code '1'.
ssbarnea commented 2 years ago

Can we please get the upcoming change warnings disabled for distributions with -latest suffix? Anyone that uses -latest already made a decision about what they want to use.

I find perfectly normal to warn people that do not use latest about changes that may impact them.

Current behavior add noise to pull requests and distracts from other work.

MatthewSteeples commented 2 years ago

@ssbarnea Arguably the message only wants to be shown to people that are using -latest as that's the only people that it affects. In our case it was a useful prompt as we then tested our builds against -2022 on a branch, discovered that they didn't work, and then pinned our builds to -2019. Without this warning we'd have been caught off guard when the change happened, and it would have need to have been a rushed and unscheduled change rather than part of a planned sprint.

ssbarnea commented 2 years ago

@MatthewSteeples I am inclined to believe that most people will not do anything about it until the change is happening. Testing before change is happening requires human time, and based on what I see on ~100 project which I am watching/contributing is that it is quite uncommon for someone to go an "let's see what happens if I try to update to newer distro".

The reality is that yesterday I seen my builds failing just because Microsoft decided to use pipx to pre-install ansible-core using pipx instead of using the outdated ansible from Ubuntu. That was a change on the image and I do not blame them for doing that change, jut is an example that this warning is questionable and we should have a way to disable it.

It should not be so hard to add an entry to yaml to declare that you don't want to be notified about new releases. In the end, I am not aware of subscribing to a changelog to build images either, and I would find that far more useful than "asking user to test our new stuff". At least with email, i would be able to decide if latest changes worth testing or not and do not pile up on each CI run.

Should I also mention that the ... soon is extremely vague, is can be today, next week, next month on or even later. Nobody knows and we are all forced to see that "advertisement" annotation that we cannot control.

Just this morning I had to dig on the issue tracker to find an undocumented feature to disable warning when archive artifact does not have anything to archive. if-no-files-found: ignore was the solution, nice and simple.

I am only asking not to punish those that want to use/support/test with the latest version of a specific platform. They already made that statement by picking a label like that. Those that value stability more, can use specific versions.

Keep in mind that the fact that a build generated warnings (annotations) does change the UI in other places too. I don't want to be distracted by that icons for false warnings.

These annotations are important and usually require user to do something to address them. Sadly for this particular one, there is no escape.

jsoref commented 2 years ago

The workaround today is to pin forward or pin backward. This is of course annoying because in either case one gets commit churn and in either case one has to unpin at some later date.

It'd certainly be nice if GitHub supported a repository/org secret that says DO_NOT_WARN_IF_WINDOWS_LATEST_IS_2022 with a specific cookie value (to be announced in a task like this one).

Relying on a cookie value would be necessary to prevent people from proactively setting DO_NOT_WARN_IF_WINDOWS_LATEST_IS_2023, DO_NOT_WARN_IF_WINDOWS_LATEST_IS_2024, ... DO_NOT_WARN_IF_WINDOWS_LATEST_IS_2038.

Joren-Thijs-KasparSolutions commented 2 years ago

Are there any issues tracking the update progress on the ubuntu-latest image?

I agree that the label -latest should alweays be the latest version available and not something we will update 4 to 5 months down the line...

For example my team has been having a lot of trouble moving our microservices from Net 5 to Net 6 because it's currently unsupported in the "latest" images. Forcing us to run our CI steps in containers which brings it's own problems.

Furthermore i just don't find it acceptable making us wait 4 to 5 months before we can upgrade to a new version when github has it available on day one. especially with Net 6 being an LTS. The same went for XCode 13 support on MacOS, we just went months without building or testing our IOS version of our flutter app because we simply couldn't, while again github had it on day one.

Please introduce a actual "latest" image so people who want to move forward can move forward.

ssbarnea commented 2 years ago

I feel @Joren-Thijs-KasparSolutions pain but I think the reason why we are on this unfortunate situation is because of a past naming mistake that was done.

My guess is that the team producing the system images imagined that they can easily release a fully working image soon after an operating system was released. The reality is that these images are, for good reasons, a kitchensink of tools and it would be close to impossible to get all of them compatible with newer operating system in time.

You cannot really expect to pass all the test suites on a new operating system, some changes will break some of the (build) tools on these platform, and I think that is something we should only get used to.

I do remember seeing delays of even more than 6 months in getting updates in the past but I seen some speed improvements too, it gets better.

Still, how about just updating the documentation in order to state that -latest counts as unstable and that many features are not guaranteed to work. If you want you can even add a new set of labels with -stable suffix which would point to versions of the operating system that passed the test of time.

Example of less confusing setup:

This is also a common naming pattern for container images so nobody should be confused by it.

rsouthworth commented 2 years ago

The following step, described here, might be causing an error:

miketimofeev commented 2 years ago

@rsouthworth the path has changed to the C:\Program Files\Microsoft SQL Server\160\DAC\bin. Sorry for the inconvenience. I'll take a look at updating the docs.

drakkan commented 2 years ago

Hi,

Inno Setup removal will break my Windows workflows, see for example here.

I guess I can download and install it as part of my workflows. The latest version of Inno Setup appears to have nearly 800,000 downloads, so its popularity doesn't seem that low. I'm just wondering if you could reconsider this decision, thank you.

EDIT: will something like this still work?

dechamps commented 2 years ago

I also use Inno Setup in my workflows. It's not clear to me how the "low popularity" of Inno Setup was ascertained or what the "maintenance concerns" are, but consider this another vote to keep it alive.

miketimofeev commented 2 years ago

@drakkan @dechamps would you mind opening an issue to add InnoSetup to the windows-2022 image?

drakkan commented 2 years ago

@drakkan @dechamps would you mind opening an issue to add InnoSetup to the windows-2022 image?

done, #5006 thanks

bormm commented 2 years ago

Choosing "-latest" for something means all your builds are not reproducible/deterministic at all. (https://reproducible-builds.org/) This will maybe not even possible with "windows-", because things might are updated even on these machines.

But I agree with @ssbarnea: Everyone selecting "latest" chose the hottest available way already. It does not make a big difference if this is more or less tested by Microsoft or wait some months to allow people to prepare. Because people that wanted to prepare the switch already have a 2022 branch running and switch by merge that to their productive branch if working. All others maybe also chose "latest" because they don't care or never thought about these things. At the end its the responsibility of the development team to choose the right ingredient for the software.

jcwchen commented 2 years ago

Hi, thank you for the awesome work.

I would like to confirm: Is windows-latest workflows now using Windows Server 2022 on GitHub Actions? (I saw it was Windows Server 2019 yesterday, but it changes to use Windows Server 2022 just now)

drwill-ms commented 2 years ago

Our existing pipeline, updated specifically to windows-2022 and VS 2022 now has an error that sn.exe can't be found trying to disable strong name validation. Suggestions?

##[debug]INPUT_SCRIPT: 'call "C:\\Program Files (x86)\\Microsoft Visual Studio\\2022\\Enterprise\\Common7\\Tools\\VsDevCmd.bat"
##[debug]sn -Vr *,31bf3856ad364e35'
##[debug]INPUT_WORKINGDIRECTORY: 'D:\a\1\s'
##[debug]Asserting container path exists: 'D:\a\1\s'
Generating script.
##[debug]AGENT_VERSION: '2.198.2'
##[debug]AGENT_TEMPDIRECTORY: 'D:\a\_temp'
##[debug]Asserting container path exists: 'D:\a\_temp'
##[debug]Asserting leaf path exists: 'C:\Windows\system32\cmd.exe'
========================== Starting Command Output ===========================
##[debug]Entering Invoke-VstsTool.
##[debug] Arguments: '/D /E:ON /V:OFF /S /C "CALL "D:\a\_temp\304ba344-c007-433b-8fad-d7ce519fe037.cmd""'
##[debug] FileName: 'C:\Windows\system32\cmd.exe'
##[debug] WorkingDirectory: 'D:\a\1\s'
"C:\Windows\system32\cmd.exe" /D /E:ON /V:OFF /S /C "CALL "D:\a\_temp\304ba344-c007-433b-8fad-d7ce519fe037.cmd""
The system cannot find the path specified.
'sn' is not recognized as an internal or external command,
operable program or batch file.
##[debug]Exit code: 1
##[debug]Leaving Invoke-VstsTool.
##[error]Cmd.exe exited with code '1'.

Any idea how I solve the problem of missing sn.exe?

LVMVRQUXL commented 2 years ago

Hi, thank you for the awesome work.

I would like to confirm: Is windows-latest workflows now using Windows Server 2022 on GitHub Actions? (I saw it was Windows Server 2019 yesterday, but it changes to use Windows Server 2022 just now)

As @miketimofeev said in the issue description, Window Server 2022 is available progressively on GitHub Actions since January, 17. They plan to complete this migration by March, 6. My workflows are still using Windows Server 2019, so you are from the lucky ones. 😉

BatmanGamingYT commented 2 years ago

Screenshot 2022-02-14 10 54 07 AM how do i fix this?

alkino commented 2 years ago

Should be close by #5093

Y-LyN-10 commented 2 years ago

It looks-like Windows 2022 is already the windows-latest version. Our builds started to fail with the latest available on GitHub version of Node.js (v16.13.2) at npm install command.

...
gyp ERR! find VS msvs_version not set from command line or npm config
gyp ERR! find VS VCINSTALLDIR not set, not running in VS Command Prompt
gyp ERR! find VS unknown version "undefined" found at "C:\Program Files\Microsoft Visual Studio\2022\Enterprise"
gyp ERR! find VS could not find a version of Visual Studio 2017 or newer to use
gyp ERR! find VS looking for Visual Studio 2015
gyp ERR! find VS - not found
gyp ERR! find VS not looking for VS2013 as it is only supported up to Node.js 8
gyp ERR! find VS 
gyp ERR! find VS **************************************************************
gyp ERR! find VS You need to install the latest version of Visual Studio
gyp ERR! find VS including the "Desktop development with C++" workload.
gyp ERR! find VS For more information consult the documentation at:
gyp ERR! find VS https://github.com/nodejs/node-gyp#on-windows
gyp ERR! find VS **************************************************************
gyp ERR! find VS 
gyp ERR! configure error 
gyp ERR! stack Error: Could not find any Visual Studio installation to use
The issue could be in the Node.js version

Here is the successful output with Windows-2019:

...
gyp info find VS using VS2019 (16.11.32106.194) found at:
gyp info find VS "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise"
gyp info find VS run with --verbose for detailed information
gyp info spawn C:\hostedtoolcache\windows\Python\3.7.9\x64\python3.exe
...

Looking at the Readme, Microsoft Visual Studio 2022 should be there in the image 🤔

jedieaston commented 2 years ago

The version of npm in the image (8.1.2) includes a version of node-gyp that doesn't support vs2022 yet, so it looks for a maximum of 2019.

If you upgrade npm to v8.1.4 or later it will work, so I presume you could just do npm upgrade -g npm@latest (presuming npm is separate from Node on the image). If you want to upgrade node-gyp separately of npm, it's a bit more involved.

bormm commented 2 years ago

It looks-like Windows 2022 is already the windows-latest version.

Why you highlight the already? The post mentions clearly "This change will be rolled out [...] beginning on January, 17. We plan to complete the migration by March, 6." We have 17 February.

Regarding the "Microsoft Visual Studio 2022 should be there in the image" I see a "C:\Program Files\Microsoft Visual Studio\2022\Enterprise" in your message also. So yes it should and is there.

For me looks like related to https://github.com/nodejs/node-gyp/issues/2552 but not the the image itself. Of course its maybe a bad idea to have a npm from 28 Oct 2021 bundled with a VS 2022 incompatible node-gyp if the windows-2022 only contains a VS 2022. Especially if the working version is available since 2021-11-18.

Y-LyN-10 commented 2 years ago

Thank you for the quick responses @jedieaston and @bormm! Unfortunately, we can not update to the latest NPM due to other NPM-related issues, but we will continue testing explicitly on windows-2019 for now and wait on node-gyp for the support of 2022.

acmnpv commented 2 years ago

This seems to have broken the compiler lookup in some cases, see here for a failing build: https://github.com/gromacs/gromacs/runs/5229802627?check_suite_focus=true

working before: https://github.com/gromacs/gromacs/runs/5229248652?check_suite_focus=true

ssbarnea commented 2 years ago

After discovering about a week ago that my WSL pipeline gets stuck I was able to narrow down the culprit: windows-2022 runner is no longer able to use WSL1, while windows-2019 was able to run in few minutes. execution-example

The way it behaves is very weird, it gets so slow that it effectively times-out. That might have something to do with the fact that my codebase is on the /mnt/c and that this has a performance impact.

To get an idea about what kind of slowdown we are talking about, the 2019 jobs runs in less than 3 minutes, while 2022 times out after 30 minutes. Even increasing the timeouts does not help because it will likely take 2-3h+ to finish at its current speed.

The stuck happens while running tests with python (tox and a little bit of ansible syntax check as part of the tests).

Do we need a separated bug for this or we already have one?