search

pantsbuild / scie-pants

Protects your Pants from the elements.
https://www.pantsbuild.org/docs/installation
Apache License 2.0
19 stars 20 forks source link

Option to avoid download python exe from https://github.com/indygreg/python-build-standalone #154

Open da-liii opened 1 year ago

da-liii commented 1 year ago

For example, in Github Action, we'd prefer to install the python needed by Pants itself by actions/setup-python.

jsirois commented 1 year ago

@darcy-shen that's a no-go. The whole point of scie-pants is that it ships with a Python interpreter you, as the consumer, are unaware of. It is absolutely not intended to work with an external interpreter. Do you actually care about the interpreter it uses, or just that it doesn't download the interpreter dynamically? If its the latter, that can be accomodated. Right now we just ship a "thin" scie that does not include the Python interpreter - to save space - but downloads it as needed instead. We can also ship "fat" scies that include the interpreter. That said, downloads will still happen (of wheels) when a new Pants version is installed.

I guess what I really need here is more explanation of why you want to avoid https://github.com/indygreg/python-build-standalone - due to the technology, due to the runtime use of the network, or some other reason?

da-liii commented 1 year ago

due to the technology

I have found a better way to install Python, see https://github.com/pantsbuild/pants/issues/17962

due to the runtime use of the network

Yes, it can be unstable for me. The network is not guaranteed to be stable.

da-liii commented 1 year ago

We can also ship "fat" scies that include the interpreter.

That's a great idea.

jsirois commented 1 year ago

With scie-pants now converted to science for building its scie, creating fat scies for release is as easy as adding --invert-lazy cpython39 --app-name scie-pants-fat to the science command here: https://github.com/pantsbuild/scie-pants/blob/14ed78482a6e39d7f01a2d01e1b169c862c104ba/package/src/scie_pants.rs#L37-L61

The question that needs to be considered is whether to make the fat scie fully fat (Also include --invert-lazy cpython38). Although this question is probably easy to answer "no" to since 3.8 is only needed for Pants < 2.5 which is quite old at this point, it gets harder to answer when Pants switches to using CPython 3.11. At that point there will be very recent versions of Pants that need 3.9; so the fat variant including both cpython39 and cpython311 distributions (but still probably not cpython38) may make more sense.

I'll leave all this for others to kvetch over and implement.

cognifloyd commented 2 months ago

I like to use termux on my android phone, but it is a quirky environment. Most apps require patching to support the prefixed/relocated location for libs and other lookup paths. I can't use scie-pants because the underlying crates can't find resolve.conf and related networking bits, so dns lookups fail.

Even if the networking bits were resolved, PBS does not have a build for termux (https://github.com/indygreg/python-build-standalone/issues/176 and https://github.com/indygreg/python-build-standalone/issues/87). However, I finally found a pre-built python 3.9 interpreter for termux, so I would love to make scie-pants just use that. Relevant links:

To install these deb files I run pkg install tur-repo and pkg install python3.9 where pkg wraps apt.

kaos commented 2 months ago

I like to use termux on my android phone, but it is a quirky environment. Most apps require patching to support the prefixed/relocated location for libs and other lookup paths. I can't use scie-pants because the underlying crates can't find resolve.conf and related networking bits, so dns lookups fail.

Even if the networking bits were resolved, PBS does not have a build for termux (indygreg/python-build-standalone#176 and indygreg/python-build-standalone#87). However, I finally found a pre-built python 3.9 interpreter for termux, so I would love to make scie-pants just use that. Relevant links:

To install these deb files I run pkg install tur-repo and pkg install python3.9 where pkg wraps apt.

@cognifloyd could PANTS_BOOTSTRAP_URLS in combination with file:/// urls work for this? https://github.com/pantsbuild/scie-pants?tab=readme-ov-file#firewall-support

cognifloyd commented 2 months ago

@cognifloyd could PANTS_BOOTSTRAP_URLS in combination with file:/// urls work for this? https://github.com/pantsbuild/scie-pants?tab=readme-ov-file#firewall-support

Sweet. That's a helpful knob I didn't realize existed. So, I can now bypass the networking bits, but I think termux has to patch locking in Python to work with Android's libc, so I run into this error with the vanilla PBS:

Downloading file:///data/data/com.termux/files/home/downloads/ptex/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tar.gz...
Error: Failed to establish atomic directory /data/data/com.termux/files/home/.cache/nce/f4a11f9bb68736c1c58003a35df7d459dab771227cb6777ec70921f83e0cb95d/locks/configure-97635d72a1cd54a03e580cfb08efc421bd3a0ed9078f43a25e4e739885f2e937. Population of work directory failed: Failed to launch boot binding: Failed to spawn "/data/data/com.termux/files/home/.cache/nce/7d19e1ecd6e582423f7c74a0c67491eaa982ce9d5c5f35f0e4289f83127abcb8/cpython-3.9.18+20240107-aarch64-unknown-linux-gnu-install_only.tar.gz/python/bin/python3.9" ["/data/data/com.termux/files/home/.cache/nce/ad8ed7b8c9b0e05355f776cd229a3b2c18840e000dd0c610048c92c3a910d317/tools.pex", "configure-pants", "--ptex-path", "/data/data/com.termux/files/home/.cache/nce/b71bea2deda31242e8d5d8749d173f4fe166182e8ebbef3ef605da06320bf24a/ptex-linux-aarch64", "--pants-version", "2.20.3", "--pants-config", "", "--github-api-bearer-token", "", "--pants-bootstrap-urls", "/data/data/com.termux/files/home/pants_urls.json", "/data/data/com.termux/files/home/.cache/nce/f4a11f9bb68736c1c58003a35df7d459dab771227cb6777ec70921f83e0cb95d/bindings"]: No such file or directory (os error 2)

This is the python compile/configure option that I think PBS is missing: https://github.com/termux-user-repository/tur/blob/master/tur%2Fpython3.9%2Fbuild.sh#L30

edit: no it is probably not that patch. I think the issue is that the python binary can't find libc as it is linked against glibc not the android libc. I can't even run the PBS python binary that scie unpacked.)

$ ldd /data/data/com.termux/files/home/.cache/nce/7d19e1ecd6e582423f7c74a0c67491eaa982ce9d5c5f35f0e4289f83127abcb8/cpython-3.9.18+20240107-aarch64-unknown-linux-gnu-install_only.tar.gz/python/bin/python3.9
        libpython3.9.so.1.0 => not found
        libpthread.so.0 => not found
        libdl.so.2 => not found
        libutil.so.1 => not found
        libm.so.6 => not found
        librt.so.1 => not found
        libc.so.6 => not found
$ /data/data/com.termux/files/home/.cache/nce/7d19e1ecd6e582423f7c74a0c67491eaa982ce9d5c5f35f0e4289f83127abcb8/cpython-3.9.18+20240107-aarch64-unknown-linux-gnu-install_only.tar.gz/python/bin/python3.9
bash: /data/data/com.termux/files/home/.cache/nce/7d19e1ecd6e582423f7c74a0c67491eaa982ce9d5c5f35f0e4289f83127abcb8/cpython-3.9.18+20240107-aarch64-unknown-linux-gnu-install_only.tar.gz/python/bin/python3.9: cannot execute: required file not found

Vs the termux build:

$ ldd $(which python3.9)
        libandroid-support.so => /data/data/com.termux/files/usr/lib/libandroid-support.so
        libpython3.9.so.1.0 => /data/data/com.termux/files/usr/lib/libpython3.9.so.1.0
        libc.so => /system/lib64/libc.so
        ld-android.so => /system/lib64/ld-android.so
        libdl.so => /system/lib64/libdl.so
        libm.so => /system/lib64/libm.so
$ python3.9
Python 3.9.18 (main, Sep  8 2023, 11:55:57)
[Clang 14.0.7 (https://android.googlesource.com/toolchain/llvm-project 4c603efb on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
kaos commented 2 months ago

You point it at a .tar.gz, doesn't have to be a archive containing the PBS dist, right? What if you package up your termux build and name that like the PBS one instead? (very hackish, indeed, but you are running this on your phone too... :P )

cognifloyd commented 2 months ago

You point it at a .tar.gz, doesn't have to be a archive containing the PBS dist, right? What if you package up your termux build and name that like the PBS one instead? (very hackish, indeed, but you are running this on your phone too... :P )

SCIE checks a hash of the file, so that won't work. 😞

https://github.com/pantsbuild/scie-pants/blob/d336b8fe86f086942cabc00ab81d062af266f9cf/README.md#L96-L99

kaos commented 2 months ago

You point it at a .tar.gz, doesn't have to be a archive containing the PBS dist, right? What if you package up your termux build and name that like the PBS one instead? (very hackish, indeed, but you are running this on your phone too... :P )

SCIE checks a hash of the file, so that won't work. 😞

https://github.com/pantsbuild/scie-pants/blob/d336b8fe86f086942cabc00ab81d062af266f9cf/README.md#L96-L99

Doh. Forgot about that. Lol- Well, it sure is being secure at least. 😅