search

pantsbuild / scie-pants

Protects your Pants from the elements.
https://www.pantsbuild.org/docs/installation
Apache License 2.0
19 stars 20 forks source link

Pants 3rdparty non-repo-local plugins with non-wheel deps cannot be used. #71

Open jsirois opened 1 year ago

jsirois commented 1 year ago

This is a gedanken issue and I've labelled it a bug, but its unclear how to treat this.

The problem:

If a Pants user has a [BOOTSTRAP] plugins = [ ... ] entry that requires compilation of an sdist, they cannot use scie-pants, since scie-pants uses Python Build Standalone interpreters and these have an unuseable sysconfig / cannot build C extensions.

To work around this it seems you'd either:

  1. Need to pre-build wheels for such sdists and make them available via a find-links repo or an alternative index.
  2. Need Pants to grow the capability of discovering a local interpreter different from its own to resolve plugins with.

The latter is a horrible solution since it requires you have a local interpreter exactly matching the "hidden" Pants interpreter major / minor version (Generally 3.9 currently). This is strictly worse than the status quo of the ./pants script today, which has the same requirements, but with more lenient Python major / minor version strictures, accepting any of 3.{7,8,9}.

jsirois commented 1 year ago

Another way to phrase the problem at a higher level is that if Pants ships as a binary, plugins must too.

That is a new requirement on Pants plugin authors and translates to ensuring their complete dep set is available as pre-built wheels for all the platforms Pants supports.

stuhood commented 1 year ago

2. Need Pants to grow the capability of discovering a local interpreter different from its own to resolve plugins with.

Although horrible, I think that this might involve a one line change like:

--- a/src/python/pants/init/plugin_resolver.py
+++ b/src/python/pants/init/plugin_resolver.py
@@ -71,3 +71,3 @@ async def resolve_plugins(
             PythonExecutable.fingerprinted(
-                sys.executable, ".".join(map(str, sys.version_info[:3])).encode("utf8")
+                sys.executable, ".".join(map(str, sys.version_info[:2])).encode("utf8")
             ),
jsirois commented 1 year ago

Yeah, seems likely. It truly is horrible though. I think the best path forward is to squash this right away - IIUC there are 0 known 3rdparty plugins today with sdist-only deps for platforms Pants supports - and allow this back in later when we can support it in a non-horrible way. I don't think that day will come, but either way, restrict early, expand later is ~always a good safe bet.

stuhood commented 1 year ago

IIUC there are 0 known 3rdparty plugins today with sdist-only deps for platforms Pants supports

Wouldn't the transitive deps of in-repo plugins also be affected?

jsirois commented 1 year ago

Not really. For an in-repo plugin its always easy to arrange find-links with pre-built wheels. That is slightly clunky but not horrible. For a public 3rdparty repo you now ask someone to support hosting for the world. Of course in-repo assumes in-repo and private. You have the same problem again if the plugin is in-repo and public.

jsirois commented 1 year ago

Another left field angle that probably doesn't work is to have the PBS Python re-write its sysconfig in an install step. I'm not sure that even works though - I have not experimented. I think that the bad paths are all in there though. If this did work it would likely be complicated / titchy code. You have to be ~/.configure and go find these paths on the current machine, if they exist (users would need gcc / clang / etc and you'd need to detect where they all were. Basically, if you could get that to work, it would properly belong in PBS or its suite. A tool to localize a PBS Python when possible.

jsirois commented 1 year ago

Option #18 is to have scie-pants grow the ability to:

  1. Clone the pyenv repo (either demand git be on PATH or link the Git crate)
  2. Build an interpreter on the fly.

This 18th option would have to be opt-in somehow, since it will fail unless pre-reqs for pyenv are met; so you'd only want to use it if you were forced to by plugins with sdist-only deps.

jsirois commented 9 months ago

Modern PBS seems to work with everything I throw at it, including psutil, hdrhistogram 0.10.1 (see #69), numpy, PyYAML and pandas, using pip install <thing> --no-binary :all: from a venv I created with it (I used 3.12.1, but this is nominally fixed for 3.{8,9,10,11} too in the 20240107 release.

Scie pants probably wants to check that out: https://github.com/indygreg/python-build-standalone/releases/tag/20240107 See also https://github.com/pantsbuild/scie-pants/issues/52#issuecomment-1901007164 for another reason to upgrade.

jsirois commented 9 months ago

I'm not sure if the gedanken issue is already fixed over in Pants, but regardless, #52 indicates an PBS upgrade is warranted.

jsirois commented 9 months ago

Modern PBS seems to work with everything I throw at it, including psutil, hdrhistogram 0.10.1 ...

Ok, that turned out to be because I happened to have clang installed. With just gcc, those two fail under the last 2 years worth of PBS releases, and the hard-coded clang from the PBS build process is not one of those issues solved in the 20240107 release.

jsirois commented 9 months ago

And to confirm the gedanken bug is real, in the scie-pants repo itself, without clang installed, just gcc, add an sdist-only platform specific requirement:

diff --git a/pants.toml b/pants.toml
index 5d8d7f7..cb68c18 100644
--- a/pants.toml
+++ b/pants.toml
@@ -8,6 +8,10 @@ backend_packages = [
     "pants.backend.python.typecheck.mypy",
 ]

+plugins = [
+    "hdrhistogram==0.10.1",
+]
+
 [anonymous-telemetry]
 enabled = true
 repo_id = "e0b99427-9bc2-4f6a-b197-f5f378849b15"

And try to run Pants:

jsirois@Gill-Windows:~/dev/pantsbuild/scie-pants (main *) $ pants -V
Bootstrapping Pants 2.18.0
Installing pantsbuild.pants==2.18.0 into a virtual environment at /home/jsirois/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.18.0
Found existing installation: setuptools 58.1.0
Uninstalling setuptools-58.1.0:
  Successfully uninstalled setuptools-58.1.0
New virtual environment successfully created at /home/jsirois/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.18.0.
14:41:44.88 [INFO] Starting: Resolving plugins: hdrhistogram==0.10.1
14:41:52.26 [INFO] Completed: Resolving plugins: hdrhistogram==0.10.1
14:41:52.26 [ERROR] 1 Exception encountered:

ProcessExecutionFailure: Process 'Resolving plugins: hdrhistogram==0.10.1' failed with exit code 1.
stdout:

stderr:
pid 1485817 -> /home/jsirois/.cache/pants/named_caches/pex_root/venvs/ebe283be6c4e568db7b331850fb421d7e39a2f30/1c4ceb7496c56f0c5a2a8e185e772484dd2c4f2d/bin/python -sE /home/jsirois/.cache/pants/named_caches/pex_root/venvs/ebe283be6c4e568db7b331850fb421d7e39a2f30/1c4ceb7496c56f0c5a2a8e185e772484dd2c4f2d/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /home/jsirois/.cache/pants/named_caches/pex_root/pip_cache wheel --no-deps --wheel-dir /home/jsirois/.cache/pants/named_caches/pex_root/built_wheels/sdists/hdrhistogram-0.10.1.tar.gz/542a10f166ea07229cd85a6c09229936749ed7840018993537340fa9ada78566/cp39-cp39-manylinux_2_35_x86_64.21d0e2c1bb5e417b8c85a39d5186aca6.work /home/jsirois/.cache/pants/named_caches/pex_root/downloads/resolver_download.uew9dcp9/home.jsirois..cache.nce.3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e.bindings.venvs.2.18.0.bin.python3.9/hdrhistogram-0.10.1.tar.gz --index-url https://pypi.org/simple/ --retries 5 --timeout 15 exited with 1 and STDERR:
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [152 lines of output]
...
      [pbr] Reusing existing SOURCES.txt
      running build_ext
      building 'pyhdrh' extension
      creating build/temp.linux-x86_64-cpython-39
      creating build/temp.linux-x86_64-cpython-39/src
      clang -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fdebug-default-version=4 -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -I/tools/deps/libedit/include -fPIC -I/home/jsirois/.cache/pants/named_caches/pex_root/venvs/ebe283be6c4e568db7b331850fb421d7e39a2f30/1c4ceb7496c56f0c5a2a8e185e772484dd2c4f2d/include -I/home/jsirois/.cache/nce/f3ff38b1ccae7dcebd8bbf2e533c9a984fac881de0ffd1636fbb61842bd924de/cpython-3.9.18+20231002-x86_64-unknown-linux-gnu-install_only.tar.gz/python/include/python3.9 -c src/python-codec.c -o build/temp.linux-x86_64-cpython-39/src/python-codec.o
      error: command 'clang' failed: No such file or directory
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for hdrhistogram
ERROR: Failed to build one or more wheels

Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.

Use --print-stacktrace for more error details and/or -ldebug for more logs.
See https://www.pantsbuild.org/v2.18/docs/troubleshooting for common issues.
Consider reaching out for help: https://www.pantsbuild.org/v2.18/docs/getting-help

Maybe a ~simple solution - Have scie-pants figure out CC and then plumb it via env vars? Here I try that manually:

diff --git a/pants.toml b/pants.toml
index 5d8d7f7..d4653c7 100644
--- a/pants.toml
+++ b/pants.toml
@@ -8,6 +8,10 @@ backend_packages = [
     "pants.backend.python.typecheck.mypy",
 ]

+plugins = [
+    "hdrhistogram==0.10.1",
+]
+
 [anonymous-telemetry]
 enabled = true
 repo_id = "e0b99427-9bc2-4f6a-b197-f5f378849b15"
@@ -39,7 +43,8 @@ known_versions = [

 [subprocess-environment]
 env_vars = [
-    "BUILDROOT=%(buildroot)s"
+    "BUILDROOT=%(buildroot)s",
+    "CC"
 ]

 [source]
$ CC=gcc pants -V
14:47:30.51 [INFO] Starting: Resolving plugins: hdrhistogram==0.10.1
14:47:33.35 [INFO] Completed: Resolving plugins: hdrhistogram==0.10.1
14:47:33.35 [ERROR] 1 Exception encountered:

ProcessExecutionFailure: Process 'Resolving plugins: hdrhistogram==0.10.1' failed with exit code 1.
stdout:

stderr:
pid 1487232 -> /home/jsirois/.cache/pants/named_caches/pex_root/venvs/ebe283be6c4e568db7b331850fb421d7e39a2f30/1c4ceb7496c56f0c5a2a8e185e772484dd2c4f2d/bin/python -sE /home/jsirois/.cache/pants/named_caches/pex_root/venvs/ebe283be6c4e568db7b331850fb421d7e39a2f30/1c4ceb7496c56f0c5a2a8e185e772484dd2c4f2d/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /home/jsirois/.cache/pants/named_caches/pex_root/pip_cache wheel --no-deps --wheel-dir /home/jsirois/.cache/pants/named_caches/pex_root/built_wheels/sdists/hdrhistogram-0.10.1.tar.gz/542a10f166ea07229cd85a6c09229936749ed7840018993537340fa9ada78566/cp39-cp39-manylinux_2_35_x86_64.59d589e13bda458ca8616eeffc6d81a6.work /home/jsirois/.cache/pants/named_caches/pex_root/downloads/resolver_download.nzzxlou4/home.jsirois..cache.nce.3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e.bindings.venvs.2.18.0.bin.python3.9/hdrhistogram-0.10.1.tar.gz --index-url https://pypi.org/simple/ --retries 5 --timeout 15 exited with 1 and STDERR:
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [153 lines of output]
...
      [pbr] Reusing existing SOURCES.txt
      running build_ext
      building 'pyhdrh' extension
      creating build/temp.linux-x86_64-cpython-39
      creating build/temp.linux-x86_64-cpython-39/src
      gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fdebug-default-version=4 -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -I/tools/deps/libedit/include -fPIC -I/home/jsirois/.cache/pants/named_caches/pex_root/venvs/ebe283be6c4e568db7b331850fb421d7e39a2f30/1c4ceb7496c56f0c5a2a8e185e772484dd2c4f2d/include -I/home/jsirois/.cache/nce/f3ff38b1ccae7dcebd8bbf2e533c9a984fac881de0ffd1636fbb61842bd924de/cpython-3.9.18+20231002-x86_64-unknown-linux-gnu-install_only.tar.gz/python/include/python3.9 -c src/python-codec.c -o build/temp.linux-x86_64-cpython-39/src/python-codec.o
      gcc: error: unrecognized command-line option ‘-fdebug-default-version=4’
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for hdrhistogram
ERROR: Failed to build one or more wheels

Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.

Use --print-stacktrace for more error details and/or -ldebug for more logs.
See https://www.pantsbuild.org/v2.18/docs/troubleshooting for common issues.
Consider reaching out for help: https://www.pantsbuild.org/v2.18/docs/getting-help

So, apparently no easy way out.