Bootstrap python by xrepo

[da-tubi]

Is your feature request related to a problem? Please describe. There are no Python 3.8/3.9 in Ubuntu 22.04.

I tried to use pyenv to install a Python 3.9 interpreter for Pants:

pyenv install 3.9.16

But it keeps failing because I didn't install:

tk8.6-dev
liblzma-dev
...

Describe the solution you'd like I use xrepo to install python.

$ xrepo install "python 3.9.10"
$ xrepo env -b "python 3.9.10" which python
/home/da/.xmake/packages/p/python/3.9.10/d06fa80222094670880bb9934b285f43/bin/python

I added /home/da/.xmake/packages/p/python/3.9.10/d06fa80222094670880bb9934b285f43/bin to PATH, and it works well for Pants.

I hope there is a <XREPO> just like <PYENV> available for xrepo installed Python, so that I do not need to tune the PATH.

Describe alternatives you've considered Further more, if pants can not find the suitable Python interpreter, and if xrepo is available as a command line, pants could invoke xrepo to install the corresponding Python interpreter.

Additional context

• python packages in xrepo: https://xrepo.xmake.io/#/packages/linux?id=python-linux

Notice that

xrepo search python

can only find the latest support python version. And all supported python can be found in the above link.

xrepo users could install python in the following ways:

xrepo install "python 3.x"
xrepo install "python 3.8"
xrepo install "python >=3.11"

[kaos]

Hi @da-tubi,

It would be interesting to learn if scie-pants would work for you here.

Could you please try:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/pantsbuild/scie-pants/main/get-pants.sh)"

This will download and install a pants binary to ~/bin which you should be able to use instead of ./pants. The scie-pants comes with a Python interpreter embedded for use with Pants itself.

[da-tubi]

As you described here, I think scie-pants would only work for me on solving the python bootstrap interpreter.

xrepo (just like conda) will help me install any Python interpreters (theoretically) in a open source and reproducible way.

[kaos]

Yes, I realize it doesn't solve the issue with providing a Python interpreter for your project after Pants has been bootstrapped.

[da-tubi]

Here is my solution without Pants' support of XREPO:

 ln -s /home/da/.xmake/packages/p/python/3.9.10/d06fa80222094670880bb9934b285f43/ $HOME/.pyenv/versions/3.9.10

Just create a symbol link from the xrepo-installed python to pyenv.

[da-liii]

Here is my better solution: https://github.com/da-tubi/pants-airflow/pull/1/files

Step 1: Install Python 3.10.6 via xrepo.

xrepo install "python 3.10.6"

Step 2: Add "%(homedir)s/.xmake/packages/p/python/3.10.6/e357df3d007a461995be04740991afd2/bin/", to the search_path.

[da-liii]

On my Debian sid, if I use the Python interpreter installed via apt, it will failed due to:

1. apache-airflow 2.4.3 from https://files.pythonhosted.org/packages/fe/e1/4978f6f8fa109815cfec77eb0c7ab398e89466c64fe62a1486c83a05ae90/apache_airflow-2.4.3-py3-none-any.whl
    ModuleNotFoundError: No module named 'distutils.util'
2. alembic 1.10.2 from https://files.pythonhosted.org/packages/2a/e2/3b8d29bcc3ac56d579148dbbeee4d5a452c77da82a4051c48785eac89b8b/alembic-1.10.2-py3-none-any.whl
    ModuleNotFoundError: No module named 'distutils.util'

Using the Python interpreter installed by xrepo, the problem is solved.

[da-tubi]

Here is a better way to install Python and make it visible to Pants:

#!/bin/bash

set -x
xrepo install "python 3.10"
PATH_TO_PYTHON=$(xrepo env -b "python 3.10" python -c "import sys; print('/'.join(sys.executable.split('/')[:-2]))")
ln -s $PATH_TO_PYTHON/bin/python3.10 $HOME/.local/bin/python3.10
$HOME/.local/bin/python3.10 --version

If $HOME/.local/bin/ is not in $PATH, we could also add %(homedir)s/.local/bin to search_path.

[jsirois]

On my Debian sid, if I use the Python interpreter installed via apt, it will failed due to:

@darcy-shen FWIW that style error is "well-known" with Debian Python packaging practices. They separate Python into multiple packages. As a developer you generally need pythonX.Y-dev, pythonX.Y-distutils and pythonX.Y-venv.