Python 3.10 is not supported for running the Pants script itself

[slsyy]

 ~/example-golang$ ./pants

 No valid Python interpreter found. For `pants_version = "2.9.0.dev3"`, Pants requires Python 3.7, 3.8, or 3.9 to run. Please check that a valid interpreter is installed and on your $PATH.

$ python --version                                                                      
Python 3.10.1

Maintainer edit:

Note that Pants can still run your own code using Python 3.10, this is only about what Pants uses to run itself. See https://www.pantsbuild.org/docs/python-interpreter-compatibility.

[stuhood]

Thanks for the report! For now though, the message is accurate: Pants includes native code, and is only built for 3.7, 3.8, and 3.9. I'll keep this open to track Pants being published for more versions.

[benjyw]

And, to clarify, this just means that Pants requires 3.7, 3.8 or 3.9 in order to run itself. But it can build your Python code (if you have any) at any version of Python.

[phlax]

Im wondering if there is a timeline for 3.10 support - we would like to upgrade to 3.10 but need to be able to test code with the correct version

[benjyw]

To clarify - Pants should be able to run your own code on 3.10. It requires 3.7-3.9 to run itself, but you can think of that as an implementation detail.

If you set your interpreter constraints to require 3.10, and there is a 3.10 interpreter Pants can discover, Pants will use it to run your tests.

[phlax]

great, that worked, thanks!

[michael-go]

Note that Ubuntu 22.04 LTS ships with Python 3.10

sudo apt update
sudo apt install python3
python3 --version
> Python 3.10.4

so to run Pants one needs to install older Python from an alternative channel, like in example https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa

in case it affects priority of supporting it. thanks :pray:

[jriddy]

Chiming in that current active Fedora versions 35 and 36 ship with Python 3.10 as stock too

This means that for users of these distro versions (as well as Ubuntu 22.04 mentioned above), Pants is not usable "out of the box". It requires a separate step to install an older version of Python that Pants can discover.

So yes, it's an implementation detail, but it's one that users must deal with.

[Eric-Arellano]

@stuhood @benjyw @chrisjrn I added this to the "onboarding" project. We should re-evaluate today how this compares to other priorities to make it easier to get started with using Pants.

[ghost]

Per a slack thread with @Eric-Arellano, the reasons why python 3.10 is not supported are:

1. Releasing on 3.7, 3.8, 3.9, and 3.10 will slow down the CI even more.
2. Adding 3.10 will increase the amount of storage space used on PyPi and Pants is already in the top 20 projects for storage space usage

As such, to add 3.10 they would want to drop support for Python 3.7, which is semi-controversial (understandably so)

[benjyw]

To that great summary I would just add the word "yet". There will come a natural point where we will support 3.10 and drop 3.7, but we just may not be there yet.

[Eric-Arellano]

Largely thanks to @twunderlich-grapl bumping this issue in Slack, I started a poll yesterday:

Pants will release for the most recent Python version + 2 before. So, Pants 2.14 will be 3.8-3.10. When Python 3.11 is released, the corresponding Pants release series will be 3.9-3.11

Pants release series will support their interpreter range indefinitely. Meaning, Pants 2.13 will always be 3.7-3.9, for example

13 upvotes, 0 downvotes. I'll give it until Monday, but think there's enough consensus for us to target Pants 2.14 being released with Python 3.8-3.10 🎉

[slsyy]

2. Adding 3.10 will increase the amount of storage space used on PyPi and Pants is already in the top 20 projects for storage space usage

@twunderlich-grapl Off-topic: why do you include debug symbols in the lib, if space is an issue?

Some experiments:

Current options

237M    target/release/libengine.so
45M     libengine.zip

Without debug symbols

diff --git a/src/rust/engine/Cargo.toml b/src/rust/engine/Cargo.toml
index 0fd564204..c631abf49 100644
--- a/src/rust/engine/Cargo.toml
+++ b/src/rust/engine/Cargo.toml
@@ -8,7 +8,7 @@ publish = false
-debug = true
+#debug = true

23M     target/release/libengine.so
7.9M    libengine.zip

Without debug symbols + lto (faster and smaller, but compiles longer)

diff --git a/src/rust/engine/Cargo.toml b/src/rust/engine/Cargo.toml
index 0fd564204..3cfe90331 100644
--- a/src/rust/engine/Cargo.toml
+++ b/src/rust/engine/Cargo.toml
@@ -8,8 +8,9 @@ publish = false
-debug = true
+#debug = true
+lto = true

19M     target/release/libengine.so
6.9M    libengine.zip

If the only reason is a:

 # Enable debug symbols in the `release` profile: doesn't add a noticeable overhead in size/speed,
 # but enables profiling of optimized builds.

then you can use force-frame-pointers for the same purpose

[stuhood]

2. Adding 3.10 will increase the amount of storage space used on PyPi and Pants is already in the top 20 projects for storage space usage

@twunderlich-grapl Off-topic: why do you include debug symbols in the lib, if space is an issue?

...

If the only reason is a:

 # Enable debug symbols in the `release` profile: doesn't add a noticeable overhead in size/speed,
 # but enables profiling of optimized builds.

then you can use force-frame-pointers for the same purpose

Interesting! So it is still possible to do useful debugging/profiling of a release build with force-frame-pointers? In that case, a patch to disable debug symbols and enable force-frame-pointers in release builds would be very welcome! I don't expect that it will impact how many versions of Python we're able to support, but a good improvement regardless.

[kamalmarhubi]

There seems to be a tradeoff between this and switching to PyOxidizer (#7369). Both result in some out-of-the-box dysfunction depending on the environment they're run in. Switching the distribution model seems like the better "end state" overall. If ease-of-installation is being considered at all, it might be worth revisiting PyOxidizer and learning what the most recent constraints are.

[benjyw]

I agree that "rust embeds python" (aka standalone binary) is the end state we want to get to. There is still work to be done there though.

[Eric-Arellano]

Update: we are still thinking about this and have the political will (per the Slack vote) to formalize a policy that we release with the 3 most recent Python versions.

The only reason I haven't moved forward on this is that @chrisjrn has made exciting progress on https://github.com/pantsbuild/pants/issues/7369, which has a big impact on what would happen: presumably, we will only release Pants for one Python version, the most recent one, given that it will include the interpreter. So, I wanted to wait to see what happens and invest my time in other project priorities.

But if someone were interested, I think it would be great for Pants 2.14+ to implement this new policy and release for 3.8-3.10. Anyone interested? This PR is probably stale, but see https://github.com/pantsbuild/pants/pull/11858 for when I added 3.9 support.

[danxmoran]

@benjyw @Eric-Arellano @stuhood I haven't been following the "rust embeds python" work to know the current progress, but this has started to bite us at Color. Is it too late to add 3.10 support for 2.15.x? If so, could we set it up for 2.16.x? Especially now that 3.11 has an official release 😅

(I'm happy / available to take on the work if you're 👍 to it)

[benjyw]

We are pretty close to having "rust embeds python" working! It should be out for code review today.

[Kaelten]

With the new scie-pants distributable, how does that impact this task?

[benjyw]

Good question! I think we can close this issue. scie-pants embeds an interpreter that is entirely an implementation detail of Pants.

Of course if your repo contains python code you still need an appropriate python interpreter to run tests on etc, but that's expected. Pants itself doesn't need an external interpreter to run.

[blimmer]

For those looking to decouple their system Python from the version that pants uses, you'll want to move to scie-pants. The new installation documentation talks about how to do this.