search

actions / runner-images

GitHub Actions runner images
MIT License
9.17k stars 2.84k forks source link

Add Python 3.9 #1740

Closed hugovk closed 3 years ago

hugovk commented 3 years ago

Tool information

Area for Triage:

Question, Bug, or Feature?:

Feature

Virtual environments affected

Can this tool be installed during the build?

https://github.com/actions/virtual-environments/issues/912 said:

Only on Windows via a Nuget package. You can build from source during build for other platforms, which I assume only takes 3-4 minutes (based on our CI running on the same images).

Tool installation time in runtime

https://github.com/actions/virtual-environments/issues/912 said:

Should be under 1 minute

Are you willing to submit a PR?

Yes, I can give it a go, with pointers.

PS It took 24 days for 3.8: https://github.com/microsoft/azure-pipelines-image-generation/issues/1317 :)

dibir-magomedsaygitov commented 3 years ago

Hello, @hugovk! Thank you for your request. We are already working on adding python 3.9 on our images. I will post update message when python 3.9 is available to use.

maxim-lobanov commented 3 years ago

Hello everyone, Python 3.9 is already available on GitHub Actions via setup-python task (It is not pre-cached on image yet, so it will be installed on-flight with minor latency):

- uses: actions/setup-python@v2
  with:
    python-version: '3.9'

We also going to deploy new image with pre-cached Python 3.9 next week.

hugovk commented 3 years ago

Confirmed it's now available on GHA, thank you!

Took ~10s to install on Linux, ~60s on macOS and ~51s on Windows, not a problem for me, and good to hear the pre-cached images is coming soon.

https://github.com/hugovk/test/runs/1215290584?check_suite_focus=true

Run actions/setup-python@v2
Version 3.9 was not found in the local cache
Version 3.9 is available for downloading
Download from "actions/python-versions@3.9.0-83838 python-3.9.0-linux-20.04-x64.tar.gz (download)"
Extract downloaded archive
/usr/bin/tar xz --warning=no-unknown-keyword -C /home/runner/work/_temp/e8e512d5-ecbb-4563-9374-a1796267419e -f /home/runner/work/_temp/e9bdf138-7f69-49ec-8166-59a47b1d3ef4
Execute installation script
Check if Python hostedtoolcache folder exist...
Create Python 3.9.0 folder
Copy Python binaries to hostedtoolcache folder
Create additional symlinks (Required for the UsePythonVersion Azure Pipelines task and the setup-python GitHub Action)
Upgrading PIP...
Looking in links: /tmp/tmpiitvwvjj
Requirement already satisfied: setuptools in /opt/hostedtoolcache/Python/3.9.0/x64/lib/python3.9/site-packages (49.2.1)
Requirement already satisfied: pip in /opt/hostedtoolcache/Python/3.9.0/x64/lib/python3.9/site-packages (20.2.3)
Collecting pip
Downloading pip-20.2.3-py2.py3-none-any.whl (1.5 MB)
Installing collected packages: pip
Successfully installed pip-20.2.3
Create complete file
Successfully setup CPython (3.9.0)
charris commented 3 years ago

Commenting for followup postings.

hugovk commented 3 years ago

@charris tip for next time: no need to comment to subscribe to followup postings, there's a Subscribe button, which also avoids notifying dozens of issue and repo watchers :)

laike9m commented 3 years ago

Hello everyone, Python 3.9 is already available on GitHub Actions via setup-python task (It is not pre-cached on image yet, so it will be installed on-flight with minor latency):

- uses: actions/setup-python@v2
  with:
    python-version: '3.9'

We also going to deploy new image with pre-cached Python 3.9 next week.

Hi Maxim, what about Azure pipelines?

maxim-lobanov commented 3 years ago

@laike9m , Unfortunately, Azure Pipelines UsePython task can't install versions on-flight like setup-python so we have to wait for next week until new image with Python 3.9 is deployed. (I suggest to upvote feature request for UsePython to work the same way like setup-python)

laike9m commented 3 years ago

@maxim-lobanov Got it. Upvoted.

webknjaz commented 3 years ago

@maxim-lobanov All my jobs hit API rate limit when I set them to 3.9 :(

Is there a problem with it? https://github.com/ansible/pylibssh/pull/152/checks?check_run_id=1232497424

maxim-lobanov commented 3 years ago

@webknjaz , Hm, I have never seen such error. setup-python is really use system GitHub API token for downloading Python on-flight but it shouldn't expire. We use the same approach in setup-node too and didn't see this issue for it too. Also failed to reproduce it on my repo, it works as expected.

A few thoughts from my side: 1) Do you have any other workflows in your repo that could abuse API silently somehow? I guess github.token has same rate limit for all workflows in repo so any other workflow abuse rate limit hardly, it could affect this workflow. 2) You can try to enable debug logging and look at the logs. 3) I think it makes sense to report an issue in https://github.com/actions/setup-python

@webknjaz , I think I know root cause: your workflow job stale-artifacts-cleanup.yml abuse rate limit. See the logs of the build: https://github.com/ansible/pylibssh/runs/1232157025?check_suite_focus=true It does 3398! requests to GitHub API during 10 minutes and then fail by timeout. I strongly suggest to disable this workflow (at least temporary), wait for a few hours and then try setup-python again.

webknjaz commented 3 years ago

@maxim-lobanov thanks for the hint! Looks like that cleanup workflow can be the root of the issue. I'll try to make it use less requests...

laike9m commented 3 years ago

May I ask the ETA for 3.9 support?

webknjaz commented 3 years ago

@zooba promised to look into it a few days back: https://twitter.com/zooba/status/1315192397144489984

hugovk commented 3 years ago

I see Python 3.9.0 is now cached in these pre-releases, presumably queued up for deploy?

But 3.9 not yet in:

maxim-lobanov commented 3 years ago

We are expecting MacOS and Ubuntu images with Python 3.9 to be deployed tomorrow. On Windows, Python 3.9 will come a bit later with next image update.

hugovk commented 3 years ago

On Windows, Python 3.9 will come a bit later with next image update.

Will the next image update be next week?

LeonidLapshin commented 3 years ago

Hey, @hugovk Windows images will be updated next week, but we'll try our best to make this time period as short as possible. Thank you for patience!

ihnorton commented 3 years ago

It looks like Python 3.9 is now deployed as the default on (at least) linux, at very least for azure pipelines UsePythonVersion?

Many important packages do not have binary wheels available yet - for any platform (eg numpy, scipy, cython, tensorflow, pandas).

Perhaps everyone should be pinning the version, but bumping so quickly like this means that any CI setups using un-pinned python versions are going to do build-from-source for those dependencies above -- and will often fail due to missing build requirements. Ideally the default bump would wait 1-2 more weeks while the ecosystem updates, and let people opt-in to 3.9 for now.

maxim-lobanov commented 3 years ago

@ihnorton , we didn't change default Python at all. Ubuntu 16 still uses Python 3.5 as default, Ubuntu 18 uses 3.6, Ubuntu 20 uses 3.8.

Python 3.9 is available only via UsePythonVersion (for Azure DevOps) and setup-python (for GitHub Actions)

ihnorton commented 3 years ago

Yes, as I mentioned, the default for UsePythonVersion with versionSpec: '3.x' (in other words, un-pinned) is now 3.9. My suggestion is that un-pinned UsePythonVersion/setup-python should not default to 3.9 for a while, to reduce random breakage due to unavailable binary wheels.

maxim-lobanov commented 3 years ago

Oh, got it. That makes sense. cc: @konradpabjan as a setup-python owner to evaluate if it makes sense

As for the UsePython, we don't own this task so I suggest creating an issue in https://github.com/microsoft/azure-pipelines-tasks

jpivarski commented 3 years ago

When this issue is successfully closed, will Python 3.9 be available on Azure Pipelines? I raised an issue on what I thought was the right repo (microsoft/azure-pipelines-tool-lib#77), but then it looked like the question was raised on a different repo (microsoft/azure-pipelines-image-generation#1530), and a bot on that one pointed me here.

But this is for GitHub Actions. GitHub Actions and Azure Pipelines have different interfaces, though they're both owned by Microsoft; does that mean it's the same engine underneath? Does this issue thread represent work that will bring Python 3.9 to GitHub Actions and Azure Pipelines?

(In other words, can I just wait for it, or do I have to run all my testing code through a Docker image to test and deploy my package for Python 3.9?)

maxim-lobanov commented 3 years ago

Hello @jpivarski , Python 3.9 is already available on GitHub Actions. Azure DevOps deployment is in progress. Ubuntu images should be deployed today. MacOS and Windows will come early next week.

jpivarski commented 3 years ago

Hello @jpivarski , Python 3.9 is already available on GitHub Actions. Azure DevOps deployment is in progress. Ubuntu images should be deployed today. MacOS and Windows will come early next week.

Great—thank you. I just needed to know if I had to take some action.

laike9m commented 3 years ago

MacOS and Windows will come early next week.

May I ask if they've been deployed already?

jpivarski commented 3 years ago

May I ask if they've been deployed already?

I tried my builds this morning, and they're still failing due to the lack of Python 3.9 (MacOS and Windows).

miketimofeev commented 3 years ago

@laike9m @jpivarski updated macOS image will be available pretty soon, here is updated readme https://github.com/actions/virtual-environments/pull/1851/files And for windows, I'm afraid we're a bit delayed due to found bugs and going to start image deployment tomorrow, apologize for the inconvenience.

AraHaan commented 3 years ago

Oh please I cant wait for 3.9 to be added by default.

Also btw since my build also uses choco install vcexpress2010 any way to somehow cache the actual install of the Visual Studio 2010 just to be able to build my repository? I do not like having to wait like 6 minutes or so just for it to install much every invoke of the ci.

https://github.com/AraHaan/pthmgr/actions/runs/319111689 I did some testing and it also seems that setup-python does not know about python 3.0.x all the way to 3.4.x leading to build failures I wanted to test to see if my c extension code can compile or is compatible with the version of the python c api on those versions of python 3.

MaksimZhukov commented 3 years ago

Hello everyone! As @miketimofeev mentioned above, Python 3.9 is already available on Ubuntu and macOS images. Unfortunately, we have a delay with Windows images. Thank you for your patience!

As a temporary workaround, you can use the following code to download and install Python 3.9.0 in runtime on Windows images:

# You can also specify the x86 architecture like ".../python-3.9.0-win32-x86.zip"
$url = "https://github.com/actions/python-versions/releases/download/3.9.0-83838/python-3.9.0-win32-x64.zip"
(New-Object System.Net.WebClient).DownloadFile($url, "$($env:TEMP)\python.zip")
7z.exe x "$($env:TEMP)\python.zip" -o"$($env:TEMP)\python" -y | Out-Null
Push-Location -Path "$($env:TEMP)\python"
Invoke-Expression .\setup.ps1

Python 3.9.0 will be available for the UsePythonVersion task after installation described above.

AraHaan commented 3 years ago

Or just use setup-python @MaksimZhukov, However it lacks python 3.0.x all the way up to 3.4.x for x86 and x64 though that of which I also filed an issue for.

charris commented 3 years ago

Any update on the Windows availability timeline?

miketimofeev commented 3 years ago

@charris we're going to finish windows images deployment tomorrow.

laike9m commented 3 years ago

BTW it seems there's some problem with Mac + Python 3.9: https://dev.azure.com/laike9m/laike9m/_build/results?buildId=315&view=logs&j=0ff66468-69a5-508b-6ca1-fbfd9e851960&t=6505030a-b067-588a-292f-b15686a87159

remyjette commented 3 years ago

@laike9m That looks like an issue with your script, which is trying to use add-apt-repository and apt-get on a macOS agent.

As mentioned above, Python 3.9 is already available on the macOS agents. Use the UsePythonVersion task to add the binaries to the PATH.

I've been using Python 3.9 on the macOS and Ubuntu Azure Pipelines agents with no issues.

hugovk commented 3 years ago

🎉 The rollout progress is now 100% for Windows at https://github.com/actions/virtual-environments#available-environments:

image

Does this mean Python 3.9 is now available everywhere on Azure Pipelines?

hugovk commented 3 years ago

PS It took 24 days for 3.8: microsoft/azure-pipelines-image-generation#1317 :)

This year it took 22 days! 🎉

mattpackwood commented 3 years ago

Just adding myself so I can monitor...

ThiefMaster commented 3 years ago

@mattpackwood: There's a "subscribe" button on the side ;)

mattpackwood commented 3 years ago

@ThiefMaster Duh, sorry!

MaksimZhukov commented 3 years ago

Hello everyone! Python 3.9 has been successfully deployed to the all images and currently is available on the Windows 2019 and 2016 as well. Thanks for you patience! I'm closing this issue, but feel free to contact us if you have any concerns.

laike9m commented 3 years ago

Thanks for the hard work on this.

joaander commented 3 years ago

@MaksimZhukov Some of my pipelines requesting Python 3.9 on windows-2019 randomly fail (https://dev.azure.com/glotzerlab/gsd/_build/results?buildId=6977&view=logs&j=45d08669-c843-5a21-c5fb-d107fba53589&t=4b9eac60-fda1-5fb4-16ab-3474dd2a30bb&l=9). When will this change be deployed to all windows hosts?

maxim-lobanov commented 3 years ago

@joaander , Hm, that deployment has been finished a long time ago (and a few more deployments after that). But it looks like some pools were not updated. We will take care about fixing it Doesn't you mind to create the separate issue to track it? (it is not the issue with Python itself, it is more about the fact that some builds are run on outdated image)

joaander commented 3 years ago

@joaander , Hm, that deployment has been finished a long time ago (and a few more deployments after that). But it looks like some pools were not updated. We will take care about fixing it Doesn't you mind to create the separate issue to track it? (it is not the issue with Python itself, it is more about the fact that some builds are run on outdated image)

Certainly: Open in #2039