search

lablup / backend.ai

Backend.AI is a streamlined, container-based computing cluster platform that hosts popular computing/ML frameworks and diverse programming languages, with pluggable heterogeneous accelerator support including CUDA GPU, ROCm GPU, TPU, IPU and other NPUs.
https://www.backend.ai
GNU Lesser General Public License v3.0
491 stars 147 forks source link

Adopt scie to auto-populate static Python builds for seamless client installation #1254

Closed achimnol closed 4 months ago

achimnol commented 1 year ago

Let's provide a seamless automatic installation of static Python builds along with prebuilt client SDK wheels, just like scie-pants.

scie-pants is a small, self-contained bootstrapper executable written in Rust (about 4 MiB), which finds, installs, and caches the desired Python interpeter and the Pants packages based on pants.toml in the current working directory. Once installed, it transparently passes all CLI arguments to the entrypoint of pantsbuild-pants Python package, eliminating the necessity of using the ./pants script.

Let's explore the possibility to adopt the same mechanism (it is based on a kind-of generalized framework also written by one of the Pants developer: scie-a/ptex) for our client SDK distribution. If we could make the same bootstrapper, it will become easier to make a homebrew distribution of our client SDK, for example.

TODO: Check if it works on Windows as well. NOTE: Even if it does not support Windows, we already have a fallback: conda-pack based distribution.

See also: the latest installation instructions of the Pants project

achimnol commented 4 months ago

This is covered in #1680 and #1279. We now have a scie-based self-contained self-bootstrapping client executables in the GitHub releases as of 23.09.6.