Create pylance build for conda environment

[eddyxu]

Problem Statement

Make a build (.whl) that can be installed in conda environment (python 3.8-3.11)

[LiWeiJie]

pylance-0.3.7-cp38-abi3-manylinux_2_24_x86_64.whl is not a supported wheel on this platform

python version: 3.8.9 linux: Linux wli-lance-riderlearner-zll8l 5.15.0-46-generic #49~20.04.1-Ubuntu SMP Thu Aug 4 19:15:44 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux maturin version： 0.14.13 pip version： 23.0.1

root@wli-lance-riderlearner-zll8l:~/lance/python# maturin build --release
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.8
🐍 Not using a specific python interpreter
...
...
    Finished release [optimized] target(s) in 1m 02s
📦 Built wheel for abi3 Python ≥ 3.8 to /root/lance/python/target/wheels/pylance-0.3.7-cp38-abi3-manylinux_2_24_x86_64.whl

root@wli-lance-riderlearner-zll8l:~/lance/python# pip install /root/lance/python/target/wheels/pylance-0.3.7-cp38-abi3-manylinux_2_24_x86_64.whl -vvv
Using pip 23.0.1 from /usr/local/lib/python3.8/dist-packages/pip (python 3.8)
Non-user install because site-packages writeable
Created temporary directory: /tmp/pip-build-tracker-k6gt7h84
Initialized build tracking at /tmp/pip-build-tracker-k6gt7h84
Created build tracker: /tmp/pip-build-tracker-k6gt7h84
Entered build tracker: /tmp/pip-build-tracker-k6gt7h84
Created temporary directory: /tmp/pip-install-9t1ngdw9
Created temporary directory: /tmp/pip-ephem-wheel-cache-b7n90aa1
ERROR: pylance-0.3.7-cp38-abi3-manylinux_2_24_x86_64.whl is not a supported wheel on this platform.
Exception information:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper
    status = run_func(*args)
  File "/usr/local/lib/python3.8/dist-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
    return func(self, options, args)
  File "/usr/local/lib/python3.8/dist-packages/pip/_internal/commands/install.py", line 419, in run
    requirement_set = resolver.resolve(
  File "/usr/local/lib/python3.8/dist-packages/pip/_internal/resolution/resolvelib/resolver.py", line 73, in resolve
    collected = self.factory.collect_root_requirements(root_reqs)
  File "/usr/local/lib/python3.8/dist-packages/pip/_internal/resolution/resolvelib/factory.py", line 491, in collect_root_requirements
    req = self._make_requirement_from_install_req(
  File "/usr/local/lib/python3.8/dist-packages/pip/_internal/resolution/resolvelib/factory.py", line 452, in _make_requirement_from_install_req
    self._fail_if_link_is_unsupported_wheel(ireq.link)
  File "/usr/local/lib/python3.8/dist-packages/pip/_internal/resolution/resolvelib/factory.py", line 138, in _fail_if_link_is_unsupported_wheel
    raise UnsupportedWheel(msg)
pip._internal.exceptions.UnsupportedWheel: pylance-0.3.7-cp38-abi3-manylinux_2_24_x86_64.whl is not a supported wheel on this platform.
Remote version of pip: 23.0.1
Local version of pip:  23.0.1
Was pip installed by pip? True
Removed build tracker: '/tmp/pip-build-tracker-k6gt7h84'

[LiWeiJie]

pylance-0.3.7-cp38-abi3-manylinux_2_24_x86_64.whl is not a supported wheel on this platform

python version: 3.8.9 linux: Linux wli-lance-riderlearner-zll8l 5.15.0-46-generic #49~20.04.1-Ubuntu SMP Thu Aug 4 19:15:44 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux maturin version： 0.14.13 pip version： 23.0.1

root@wli-lance-riderlearner-zll8l:~/lance/python# maturin build --release
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.8
🐍 Not using a specific python interpreter
...
...
    Finished release [optimized] target(s) in 1m 02s
📦 Built wheel for abi3 Python ≥ 3.8 to /root/lance/python/target/wheels/pylance-0.3.7-cp38-abi3-manylinux_2_24_x86_64.whl

root@wli-lance-riderlearner-zll8l:~/lance/python# pip install /root/lance/python/target/wheels/pylance-0.3.7-cp38-abi3-manylinux_2_24_x86_64.whl -vvv
Using pip 23.0.1 from /usr/local/lib/python3.8/dist-packages/pip (python 3.8)
Non-user install because site-packages writeable
Created temporary directory: /tmp/pip-build-tracker-k6gt7h84
Initialized build tracking at /tmp/pip-build-tracker-k6gt7h84
Created build tracker: /tmp/pip-build-tracker-k6gt7h84
Entered build tracker: /tmp/pip-build-tracker-k6gt7h84
Created temporary directory: /tmp/pip-install-9t1ngdw9
Created temporary directory: /tmp/pip-ephem-wheel-cache-b7n90aa1
ERROR: pylance-0.3.7-cp38-abi3-manylinux_2_24_x86_64.whl is not a supported wheel on this platform.
Exception information:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper
    status = run_func(*args)
  File "/usr/local/lib/python3.8/dist-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
    return func(self, options, args)
  File "/usr/local/lib/python3.8/dist-packages/pip/_internal/commands/install.py", line 419, in run
    requirement_set = resolver.resolve(
  File "/usr/local/lib/python3.8/dist-packages/pip/_internal/resolution/resolvelib/resolver.py", line 73, in resolve
    collected = self.factory.collect_root_requirements(root_reqs)
  File "/usr/local/lib/python3.8/dist-packages/pip/_internal/resolution/resolvelib/factory.py", line 491, in collect_root_requirements
    req = self._make_requirement_from_install_req(
  File "/usr/local/lib/python3.8/dist-packages/pip/_internal/resolution/resolvelib/factory.py", line 452, in _make_requirement_from_install_req
    self._fail_if_link_is_unsupported_wheel(ireq.link)
  File "/usr/local/lib/python3.8/dist-packages/pip/_internal/resolution/resolvelib/factory.py", line 138, in _fail_if_link_is_unsupported_wheel
    raise UnsupportedWheel(msg)
pip._internal.exceptions.UnsupportedWheel: pylance-0.3.7-cp38-abi3-manylinux_2_24_x86_64.whl is not a supported wheel on this platform.
Remote version of pip: 23.0.1
Local version of pip:  23.0.1
Was pip installed by pip? True
Removed build tracker: '/tmp/pip-build-tracker-k6gt7h84'

This is a glibc version problem, my glibc version is 2.23. But the constructed whl requires 2.24. Specifying compatibility can solve

[rockpunk]

FWIW, I think the problem with this is that you need to use a conda virtualenv to call maturin (so maturin links to the conda versions of python). I did this manually over the weekend in a manylinux 2_28 container and could then install the resulting manylinux wheel in conda 3.8-3.11 env running on ubuntu:20.04 (and the manylinux-2_28 container). However, when I tried installing the wheel on stock python in ubuntu:20.04 or manylinux-2_28, it will fail with ABI error as above. Not sure why.

I didn't see an obvious way to force conda to be used in the pyo3/maturin action, so you might need to remove the maturin GHA and move the build steps to the custom container with miniconda installed.

[changhiskhan]

remove the maturin GHA and move the build steps to the custom container with miniconda installed.

but then the resulting wheels won't work in vanilla python right? that'd be a bummer too. how does like pyarrow work both in stock python and conda? or is it just that we need a conda forge thing and have ppl do conda install -c conda-forge pylance in conda environments?

[weiji14]

or is it just that we need a conda forge thing and have ppl do conda install -c conda-forge pylance in conda environments?

Would it be possible to upload the .tar.gz source distribution (sdist) of pylance to PyPI? I'm interested in packaging pylance for conda-forge, but ideally, it would require the sdist to be on PyPI first, see https://conda-forge.org/docs/maintainer/adding_pkgs.html#step-by-step-instructions. We can then handle building pylance from source on the conda-forge infrastructure on different platforms (x86, aarch64, arm64, ppcle64, etc).

[yhori991]

The wheels and installable artifacts are hosted by conda-forge and are available now.

The build recipe/feedstock is here https://github.com/conda-forge/pylance-feedstock

conda-forge supports the automatic upgrade from tarballs released from GitHub releases and it's been working well so far.