search

EricLBuehler / mistral.rs

Blazingly fast LLM inference.
MIT License
3.41k stars 247 forks source link

Installation from PyPi doesn't work #548

Open jaschn12 opened 2 months ago

jaschn12 commented 2 months ago

Describe the bug I tried to install the mistral.rs PyO3 Bindings: mistralrs but it failed. I followed the instructions under https://github.com/EricLBuehler/mistral.rs/blob/master/mistralrs-pyo3/README.md#installation-from-pypi

Here are my terminal outputs:

(base) MY_USERNAME@MY_SERVER_NAME:~$ conda create -n mistralrs python
Retrieving notices: ...working... done
Channels:
 - defaults
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/MY_USERNAME/miniconda3/envs/mistralrs

  added / updated specs:
    - python

The following NEW packages will be INSTALLED:

  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main 
  _openmp_mutex      pkgs/main/linux-64::_openmp_mutex-5.1-1_gnu 
  bzip2              pkgs/main/linux-64::bzip2-1.0.8-h5eee18b_6 
  ca-certificates    pkgs/main/linux-64::ca-certificates-2024.3.11-h06a4308_0 
  expat              pkgs/main/linux-64::expat-2.6.2-h6a678d5_0 
  ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.38-h1181459_1 
  libffi             pkgs/main/linux-64::libffi-3.4.4-h6a678d5_1 
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-11.2.0-h1234567_1 
  libgomp            pkgs/main/linux-64::libgomp-11.2.0-h1234567_1 
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-11.2.0-h1234567_1 
  libuuid            pkgs/main/linux-64::libuuid-1.41.5-h5eee18b_0 
  ncurses            pkgs/main/linux-64::ncurses-6.4-h6a678d5_0 
  openssl            pkgs/main/linux-64::openssl-3.0.14-h5eee18b_0 
  pip                pkgs/main/linux-64::pip-24.0-py312h06a4308_0 
  python             pkgs/main/linux-64::python-3.12.4-h5148396_1 
  readline           pkgs/main/linux-64::readline-8.2-h5eee18b_0 
  setuptools         pkgs/main/linux-64::setuptools-69.5.1-py312h06a4308_0 
  sqlite             pkgs/main/linux-64::sqlite-3.45.3-h5eee18b_0 
  tk                 pkgs/main/linux-64::tk-8.6.14-h39e8969_0 
  tzdata             pkgs/main/noarch::tzdata-2024a-h04d1e81_0 
  wheel              pkgs/main/linux-64::wheel-0.43.0-py312h06a4308_0 
  xz                 pkgs/main/linux-64::xz-5.4.6-h5eee18b_1 
  zlib               pkgs/main/linux-64::zlib-1.2.13-h5eee18b_1 

Proceed ([y]/n)? y

Downloading and Extracting Packages:

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate mistralrs
#
# To deactivate an active environment, use
#
#     $ conda deactivate

(base) MY_USERNAME@MY_SERVER_NAME:~$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
info: downloading installer

Welcome to Rust!

This will download and install the official compiler for the Rust
programming language, and its package manager, Cargo.

Rustup metadata and toolchains will be installed into the Rustup
home directory, located at:

  /home/MY_USERNAME/.rustup

This can be modified with the RUSTUP_HOME environment variable.

The Cargo home directory is located at:

  /home/MY_USERNAME/.cargo

This can be modified with the CARGO_HOME environment variable.

The cargo, rustc, rustup and other commands will be added to
Cargo's bin directory, located at:

  /home/MY_USERNAME/.cargo/bin

This path will then be added to your PATH environment variable by
modifying the profile files located at:

  /home/MY_USERNAME/.profile
  /home/MY_USERNAME/.bashrc

You can uninstall at any time with rustup self uninstall and
these changes will be reverted.

Current installation options:

   default host triple: x86_64-unknown-linux-gnu
     default toolchain: stable (default)
               profile: default
  modify PATH variable: yes

1) Proceed with standard installation (default - just press enter)
2) Customize installation
3) Cancel installation
>

info: profile set to 'default'
info: default host triple is x86_64-unknown-linux-gnu
info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
info: latest update on 2024-06-13, rust version 1.79.0 (129f3b996 2024-06-10)
info: downloading component 'cargo'
  8.2 MiB /   8.2 MiB (100 %)   6.0 MiB/s in  1s ETA:  0s
info: downloading component 'clippy'
info: downloading component 'rust-docs'
 15.3 MiB /  15.3 MiB (100 %)   6.2 MiB/s in  2s ETA:  0s
info: downloading component 'rust-std'
 24.4 MiB /  24.4 MiB (100 %)   6.2 MiB/s in  3s ETA:  0s
info: downloading component 'rustc'
 63.8 MiB /  63.8 MiB (100 %)   6.2 MiB/s in 10s ETA:  0s
info: downloading component 'rustfmt'
info: installing component 'cargo'
info: installing component 'clippy'
info: installing component 'rust-docs'
info: installing component 'rust-std'
 24.4 MiB /  24.4 MiB (100 %)  23.2 MiB/s in  1s ETA:  0s
info: installing component 'rustc'
 63.8 MiB /  63.8 MiB (100 %)  25.5 MiB/s in  2s ETA:  0s
info: installing component 'rustfmt'
info: default toolchain set to 'stable-x86_64-unknown-linux-gnu'

  stable-x86_64-unknown-linux-gnu installed - rustc 1.79.0 (129f3b996 2024-06-10)

Rust is installed now. Great!

To get started you may need to restart your current shell.
This would reload your PATH environment variable to include
Cargo's bin directory ($HOME/.cargo/bin).

To configure your current shell, you need to source
the corresponding env file under $HOME/.cargo.

This is usually done by running one of the following (note the leading DOT):
. "$HOME/.cargo/env"            # For sh/bash/zsh/ash/dash/pdksh
source "$HOME/.cargo/env.fish"  # For fish
(base) MY_USERNAME@MY_SERVER_NAME:~$ source $HOME/.cargo/env
(base) MY_USERNAME@MY_SERVER_NAME:~$ sudo apt install libssl-dev
[sudo] password for MY_USERNAME: 
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  cpu-checker cuda-demo-suite-12-5 ibverbs-providers ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaio1 libboost-iostreams1.74.0
  libboost-thread1.74.0 libcacard0 libdaxctl1 libdecor-0-0 libdecor-0-plugin-1-cairo libfdt1 libgfapi0 libgfrpc0 libgfxdr0 libglusterfs0
  libibverbs1 libiscsi7 libndctl6 libpmem1 libpmemobj1 libqrencode4 librados2 librbd1 librdmacm1 libsdl2-2.0-0 libspice-server1 liburing2
  libusbredirparser1 libvirglrenderer1 libwpe-1.0-1 libwpebackend-fdo-1.0-1 msr-tools nvidia-firmware-535-535.86.05 ovmf pass
  qemu-block-extra qemu-system-common qemu-system-data qemu-system-gui qemu-system-x86 qemu-utils qrencode seabios tree uidmap xclip
Use 'sudo apt autoremove' to remove them.
Suggested packages:
  libssl-doc
The following NEW packages will be installed:
  libssl-dev
0 upgraded, 1 newly installed, 0 to remove and 13 not upgraded.
Need to get 2.375 kB of archives.
After this operation, 12,4 MB of additional disk space will be used.
Get:1 http://de.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.16 [2.375 kB]
Fetched 2.375 kB in 1s (2.387 kB/s)     
Selecting previously unselected package libssl-dev:amd64.
(Reading database ... 234361 files and directories currently installed.)
Preparing to unpack .../libssl-dev_3.0.2-0ubuntu1.16_amd64.deb ...
Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.16) ...
Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.16) ...
(base) MY_USERNAME@MY_SERVER_NAME:~$ sudo apt install pkg-config
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
pkg-config is already the newest version (0.29.2-1ubuntu3).
pkg-config set to manually installed.
The following packages were automatically installed and are no longer required:
  cpu-checker cuda-demo-suite-12-5 ibverbs-providers ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaio1 libboost-iostreams1.74.0
  libboost-thread1.74.0 libcacard0 libdaxctl1 libdecor-0-0 libdecor-0-plugin-1-cairo libfdt1 libgfapi0 libgfrpc0 libgfxdr0 libglusterfs0
  libibverbs1 libiscsi7 libndctl6 libpmem1 libpmemobj1 libqrencode4 librados2 librbd1 librdmacm1 libsdl2-2.0-0 libspice-server1 liburing2
  libusbredirparser1 libvirglrenderer1 libwpe-1.0-1 libwpebackend-fdo-1.0-1 msr-tools nvidia-firmware-535-535.86.05 ovmf pass
  qemu-block-extra qemu-system-common qemu-system-data qemu-system-gui qemu-system-x86 qemu-utils qrencode seabios tree uidmap xclip
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 13 not upgraded.
(base) MY_USERNAME@MY_SERVER_NAME:~$ conda activate mistralrs
(mistralrs) MY_USERNAME@MY_SERVER_NAME:~$ export RUST_BACKTRACE=1
(mistralrs) MY_USERNAME@MY_SERVER_NAME:~$ pip install mistralrs-cuda -v
Using pip 24.0 from /home/MY_USERNAME/miniconda3/envs/mistralrs/lib/python3.12/site-packages/pip (python 3.12)
Collecting mistralrs-cuda
  Using cached mistralrs_cuda-0.1.24.tar.gz (235 kB)
  Running command pip subprocess to install build dependencies
  Collecting maturin==1.4
    Using cached maturin-1.4.0-py3-none-manylinux_2_12_x86_64.manylinux2010_x86_64.musllinux_1_1_x86_64.whl.metadata (18 kB)
  Using cached maturin-1.4.0-py3-none-manylinux_2_12_x86_64.manylinux2010_x86_64.musllinux_1_1_x86_64.whl (10.6 MB)
  Installing collected packages: maturin
  Successfully installed maturin-1.4.0
  Installing build dependencies ... done
  Running command Getting requirements to build wheel
  Getting requirements to build wheel ... done
  Running command Preparing metadata (pyproject.toml)
      Updating git repository `https://github.com/EricLBuehler/candle.git`
  🔗 Found pyo3 bindings
  🐍 Found CPython 3.12 at /home/MY_USERNAME/miniconda3/envs/mistralrs/bin/python
  📡 Using build options profile, features from pyproject.toml
  mistralrs_cuda-0.1.24.dist-info
  Checking for Rust toolchain....
  Running `maturin pep517 write-dist-info --metadata-directory /tmp/pip-modern-metadata-3ad7ce6l --interpreter /home/MY_USERNAME/miniconda3/envs/mistralrs/bin/python`
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: mistralrs-cuda
  Running command Building wheel for mistralrs-cuda (pyproject.toml)
  Running `maturin pep517 build-wheel -i /home/MY_USERNAME/miniconda3/envs/mistralrs/bin/python --compatibility off`
      Updating git repository `https://github.com/EricLBuehler/candle.git`
  🔗 Found pyo3 bindings
  🐍 Found CPython 3.12 at /home/MY_USERNAME/miniconda3/envs/mistralrs/bin/python
  📡 Using build options profile, features from pyproject.toml
      Updating git repository `https://github.com/EricLBuehler/candle.git`
     Compiling proc-macro2 v1.0.86
     Compiling unicode-ident v1.0.12
     Compiling libc v0.2.155
     Compiling autocfg v1.3.0
     Compiling cfg-if v1.0.0
     Compiling serde v1.0.203
     Compiling once_cell v1.19.0
     Compiling crossbeam-utils v0.8.20
     Compiling libm v0.2.8
     Compiling rayon-core v1.12.1
     Compiling log v0.4.22
     Compiling cc v1.0.102
     Compiling scopeguard v1.2.0
     Compiling smallvec v1.13.2
     Compiling memchr v2.7.4
     Compiling ppv-lite86 v0.2.17
     Compiling version_check v0.9.4
     Compiling bitflags v1.3.2
     Compiling zerocopy v0.7.34
     Compiling itoa v1.0.11
     Compiling paste v1.0.15
     Compiling pkg-config v0.3.30
     Compiling pin-project-lite v0.2.14
     Compiling allocator-api2 v0.2.18
     Compiling parking_lot_core v0.9.10
     Compiling target-lexicon v0.12.14
     Compiling thiserror v1.0.61
     Compiling equivalent v1.0.1
     Compiling bytes v1.6.0
     Compiling reborrow v0.5.5
     Compiling regex-syntax v0.8.4
     Compiling futures-sink v0.3.30
     Compiling vcpkg v0.2.15
     Compiling futures-core v0.3.30
     Compiling ident_case v1.0.1
     Compiling fnv v1.0.7
     Compiling simd-adler32 v0.3.7
     Compiling crc32fast v1.4.2
     Compiling raw-cpuid v10.7.0
     Compiling seq-macro v0.3.5
     Compiling ryu v1.0.18
     Compiling either v1.13.0
     Compiling futures-channel v0.3.30
     Compiling adler v1.0.2
     Compiling serde_json v1.0.118
     Compiling futures-task v0.3.30
     Compiling anyhow v1.0.86
     Compiling glob v0.3.1
     Compiling semver v1.0.23
     Compiling num-traits v0.2.19
     Compiling lock_api v0.4.12
     Compiling slab v0.4.9
     Compiling ahash v0.8.11
     Compiling futures-io v0.3.30
     Compiling siphasher v0.3.11
     Compiling pin-utils v0.1.0
     Compiling hashbrown v0.14.5
     Compiling miniz_oxide v0.7.4
     Compiling tracing-core v0.1.32
     Compiling winnow v0.5.40
     Compiling arrayvec v0.7.4
     Compiling openssl v0.10.64
     Compiling phf_shared v0.11.2
     Compiling toml_datetime v0.6.6
     Compiling lazy_static v1.5.0
     Compiling rand_core v0.6.4
     Compiling minimal-lexical v0.2.1
     Compiling foreign-types-shared v0.1.1
     Compiling bitflags v2.6.0
     Compiling rustls-pki-types v1.7.0
     Compiling quote v1.0.36
     Compiling aho-corasick v1.1.3
     Compiling foreign-types v0.3.2
     Compiling rand v0.8.5
     Compiling strsim v0.11.1
     Compiling native-tls v0.2.12
     Compiling spin v0.9.8
     Compiling tinyvec_macros v0.1.1
     Compiling http v1.1.0
     Compiling crossbeam-epoch v0.9.18
     Compiling nom v7.1.3
     Compiling syn v2.0.68
     Compiling pyo3-build-config v0.21.2
     Compiling flate2 v1.0.30
     Compiling rustc_version v0.4.0
     Compiling tinyvec v1.6.1
     Compiling openssl-probe v0.1.5
     Compiling built v0.7.3
     Compiling crossbeam-deque v0.8.5
     Compiling indexmap v2.2.6
     Compiling portable-atomic v1.6.0
     Compiling getrandom v0.2.15
     Compiling signal-hook-registry v1.4.2
     Compiling mio v0.8.11
     Compiling num_cpus v1.16.0
     Compiling socket2 v0.5.7
     Compiling phf_generator v0.11.2
     Compiling parking_lot v0.12.3
     Compiling percent-encoding v2.3.1
     Compiling aligned-vec v0.5.0
     Compiling syn v1.0.109
     Compiling openssl-sys v0.9.102
     Compiling rand_chacha v0.3.1
     Compiling ring v0.17.8
     Compiling form_urlencoded v1.2.1
     Compiling phf_codegen v0.11.2
     Compiling rav1e v0.7.1
     Compiling vob v3.0.3
     Compiling phf v0.11.2
     Compiling httparse v1.9.4
     Compiling untrusted v0.9.0
     Compiling rayon v1.10.0
     Compiling stable_deref_trait v1.2.0
     Compiling unicode-bidi v0.3.15
     Compiling zip v1.1.4
     Compiling signal-hook v0.3.17
     Compiling cudarc v0.11.6
     Compiling num-integer v0.1.46
     Compiling v_frame v0.3.8
     Compiling unicode-normalization v0.1.23
     Compiling http-body v1.0.0
     Compiling onig_sys v69.8.1
     Compiling simd_helpers v0.1.0
     Compiling libloading v0.8.4
     Compiling try-lock v0.2.5
     Compiling base64 v0.22.1
     Compiling utf8parse v0.2.2
     Compiling heck v0.5.0
     Compiling num-bigint v0.4.6
     Compiling noop_proc_macro v0.3.0
     Compiling atomic-waker v1.1.2
     Compiling imgref v1.10.1
     Compiling strsim v0.10.0
     Compiling regex-automata v0.4.7
     Compiling byteorder v1.5.0
     Compiling rustls v0.22.4
     Compiling bitstream-io v2.4.2
     Compiling new_debug_unreachable v1.0.6
     Compiling weezl v0.1.8
     Compiling rustversion v1.0.17
     Compiling unicode-width v0.1.13
     Compiling loop9 v0.1.5
     Compiling anstyle-parse v0.2.4
     Compiling toml_edit v0.21.1
     Compiling console v0.15.8
     Compiling want v0.3.1
     Compiling idna v0.5.0
     Compiling rand_distr v0.4.3
     Compiling memmap2 v0.9.4
     Compiling esaxx-rs v0.1.10
     Compiling flume v0.11.0
     Compiling avif-serialize v0.8.1
     Compiling memoffset v0.9.1
     Compiling zune-inflate v0.2.54
     Compiling fdeflate v0.3.4
     Compiling pyo3-ffi v0.21.2
     Compiling rust_decimal v1.35.0
     Compiling anstyle v1.0.7
     Compiling zeroize v1.8.1
     Compiling colorchoice v1.0.1
     Compiling heck v0.4.1
     Compiling is_terminal_polyfill v1.70.0
     Compiling anstyle-query v1.1.0
     Compiling jpeg-decoder v0.3.1
     Compiling iana-time-zone v0.1.60
     Compiling byteorder-lite v0.1.0
     Compiling zune-core v0.4.12
     Compiling option-ext v0.2.0
     Compiling regex-syntax v0.6.29
     Compiling number_prefix v0.4.0
     Compiling url v2.5.2
     Compiling quick-error v2.0.1
     Compiling lebe v0.5.2
     Compiling subtle v2.6.1
     Compiling color_quant v1.1.0
     Compiling eyre v0.6.12
     Compiling tower-layer v0.3.2
     Compiling tower-service v0.3.2
     Compiling bit_field v0.10.2
     Compiling gif v0.13.1
     Compiling zune-jpeg v0.4.11
     Compiling chrono v0.4.38
     Compiling dirs-sys v0.4.1
     Compiling anstream v0.6.14
     Compiling png v0.17.13
     Compiling pyo3 v0.21.2
     Compiling signal-hook-mio v0.2.3
     Compiling webpki-roots v0.26.3
     Compiling overload v0.1.1
     Compiling clap_lex v0.7.1
     Compiling unicode-segmentation v1.11.0
     Compiling winnow v0.6.13
     Compiling base64 v0.13.1
     Compiling macro_rules_attribute-proc_macro v0.2.0
     Compiling indenter v0.3.3
     Compiling num-rational v0.4.2
     Compiling tiff v0.9.1
     Compiling clap_builder v4.5.8
     Compiling nu-ansi-term v0.46.0
     Compiling crossterm v0.25.0
     Compiling uuid v0.8.2
     Compiling bindgen_cuda v0.1.5
     Compiling proc-macro-crate v3.1.0
     Compiling av1-grain v0.2.3
     Compiling macro_rules_attribute v0.2.0
     Compiling dirs v5.0.1
     Compiling tracing-log v0.2.0
     Compiling sharded-slab v0.1.7
     Compiling nibble_vec v0.1.0
     Compiling unicode-normalization-alignments v0.1.12
     Compiling thread_local v1.1.8
     Compiling unchecked-index v0.2.2
     Compiling candle-kernels v0.6.0 (https://github.com/EricLBuehler/candle.git?rev=b7a3e344#b7a3e344)
     Compiling regex v1.10.5
     Compiling mistralrs-core v0.1.24 (/tmp/pip-install-k3v_9rnc/mistralrs-cuda_d8edf27dd7b84807a46c329fbd384856/mistralrs-core)
     Compiling indoc v2.0.5
     Compiling endian-type v0.1.2
     Compiling unindent v0.2.3
     Compiling defmac v0.1.3
     Compiling unicode_categories v0.1.1
     Compiling galil-seiferas v0.1.5
     Compiling radix_trie v0.2.1
     Compiling tqdm v0.7.0
     Compiling mistralrs-pyo3 v0.1.24 (/tmp/pip-install-k3v_9rnc/mistralrs-cuda_d8edf27dd7b84807a46c329fbd384856/mistralrs-pyo3)
     Compiling parse-zoneinfo v0.3.1
     Compiling regex-automata v0.1.10
     Compiling synstructure v0.13.1
     Compiling darling_core v0.20.9
     Compiling pyo3-macros-backend v0.21.2
     Compiling str_inflector v0.12.0
     Compiling Inflector v0.11.4
     Compiling chrono-tz-build v0.2.1
     Compiling darling_core v0.11.0
     Compiling rustls-pemfile v2.1.2
     Compiling rand_isaac v0.3.0
     Compiling encoding_rs v0.8.34
     Compiling mime v0.3.17
     Compiling rustc-hash v1.1.0
     Compiling akin v0.4.0
     Compiling matchers v0.1.0
     Compiling sync_wrapper v1.0.1
     Compiling ipnet v2.9.0
     Compiling range-checked v0.1.0 (https://github.com/EricLBuehler/range-checked.git#655349cc)
     Compiling chrono-tz v0.8.6
     Compiling serde_derive v1.0.203
     Compiling bytemuck_derive v1.7.0
     Compiling thiserror-impl v1.0.61
     Compiling tokio-macros v2.3.0
     Compiling futures-macro v0.3.30
     Compiling openssl-macros v0.1.1
     Compiling tracing-attributes v0.1.27
     Compiling profiling-procmacros v1.0.15
     Compiling zerofrom-derive v0.1.4
     Compiling num_enum_derive v0.7.2
     Compiling darling_macro v0.20.9
     Compiling yoke-derive v0.7.4
     Compiling displaydoc v0.2.5
     Compiling pin-project-internal v1.1.5
     Compiling arg_enum_proc_macro v0.3.4
     Compiling num-derive v0.4.2
     Compiling profiling v1.0.15
     Compiling darling_macro v0.11.0
     Compiling monostate-impl v0.1.13
     Compiling tokio v1.38.0
     Compiling futures-util v0.3.30
     Compiling darling v0.20.9
     Compiling derive_builder_core v0.20.0
     Compiling try_match_inner v0.5.1
     Compiling image-webp v0.1.2
     Compiling strum_macros v0.26.4
     Compiling clap_derive v4.5.8
     Compiling derive_more v0.99.18
     Compiling async-trait v0.1.80
     Compiling bytemuck v1.16.1
     Compiling darling v0.11.0
     Compiling pyo3-macros v0.21.2
     Compiling derive-new v0.6.0
     Compiling variantly v0.4.0
     Compiling half v2.4.1
     Compiling num-complex v0.4.6
     Compiling dyn-stack v0.10.0
     Compiling tracing v0.1.40
     Compiling num_enum v0.7.2
     Compiling zerofrom v0.1.4
     Compiling pin-project v1.1.5
     Compiling rgb v0.8.40
     Compiling exr v1.72.0
     Compiling qoi v0.4.1
     Compiling yoke v0.7.4
     Compiling derive_builder_macro v0.20.0
     Compiling try_match v0.4.1
     Compiling buildstructor v0.5.4
     Compiling tracing-subscriber v0.3.18
     Compiling pulp v0.18.21
     Compiling derive_builder v0.20.0
     Compiling clap v4.5.8
     Compiling strum v0.26.3
     Compiling futures-executor v0.3.30
     Compiling http-body-util v0.1.2
     Compiling onig v6.4.0
     Compiling rustls-webpki v0.102.4
     Compiling futures v0.3.30
     Compiling tokio-util v0.7.11
     Compiling tower v0.4.13
     Compiling tokio-native-tls v0.3.1
     Compiling serde_spanned v0.6.6
     Compiling packedvec v1.2.4
     Compiling spm_precompiled v0.1.4
     Compiling monostate v0.1.13
     Compiling minijinja v2.0.2
     Compiling serde_urlencoded v0.7.1
     Compiling sparsevec v0.2.0
     Compiling itertools v0.12.1
     Compiling itertools v0.11.0
     Compiling itertools v0.13.0
     Compiling h2 v0.4.5
     Compiling cfgrammar v0.13.7
     Compiling toml_edit v0.22.14
     Compiling safetensors v0.4.3
     Compiling ureq v2.9.7
     Compiling lrtable v0.13.7
     Compiling minijinja-contrib v2.0.2
     Compiling toml v0.8.14
     Compiling gemm-common v0.17.1
     Compiling maybe-rayon v0.1.1
     Compiling indicatif v0.17.8
     Compiling rayon-cond v0.3.0
     Compiling tokio-rayon v2.1.0
     Compiling gemm-f32 v0.17.1
     Compiling gemm-c64 v0.17.1
     Compiling gemm-f64 v0.17.1
     Compiling gemm-c32 v0.17.1
     Compiling tokenizers v0.19.1
     Compiling hf-hub v0.3.2
  error: failed to run custom build command for `mistralrs-core v0.1.24 (/tmp/pip-install-k3v_9rnc/mistralrs-cuda_d8edf27dd7b84807a46c329fbd384856/mistralrs-core)`
  note: To improve backtraces for build dependencies, set the CARGO_PROFILE_RELEASE_BUILD_OVERRIDE_DEBUG=true environment variable to enable debug information generation.

  Caused by:
    process didn't exit successfully: `/tmp/pip-install-k3v_9rnc/mistralrs-cuda_d8edf27dd7b84807a46c329fbd384856/target/release/build/mistralrs-core-d56f995efd94fdb6/build-script-build` (exit status: 101)
    --- stdout
    cargo:rerun-if-changed=build.rs
    cargo:rerun-if-changed=src/cuda/nonzero_bitwise.cu
    cargo:info=["/usr", "/usr/local/cuda", "/opt/cuda", "/usr/lib/cuda", "C:/Program Files/NVIDIA GPU Computing Toolkit", "C:/CUDA"]
    cargo:rerun-if-env-changed=CUDA_COMPUTE_CAP
    cargo:rustc-env=CUDA_COMPUTE_CAP=86

    --- stderr
    #$ _NVVM_BRANCH_=nvvm
    #$ _SPACE_=
    #$ _CUDART_=cudart
    #$ _HERE_=/usr/lib/nvidia-cuda-toolkit/bin
    #$ _THERE_=/usr/lib/nvidia-cuda-toolkit/bin
    #$ _TARGET_SIZE_=
    #$ _TARGET_DIR_=
    #$ _TARGET_SIZE_=64
    #$ NVVMIR_LIBRARY_DIR=/usr/lib/nvidia-cuda-toolkit/libdevice
    #$ PATH=/usr/lib/nvidia-cuda-toolkit/bin:/tmp/pip-build-env-gky7_fw4/overlay/bin:/tmp/pip-build-env-gky7_fw4/normal/bin:/home/MY_USERNAME/.cargo/bin:/home/MY_USERNAME/miniconda3/envs/mistralrs/bin:/home/MY_USERNAME/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
    #$ LIBRARIES=  -L/usr/lib/x86_64-linux-gnu/stubs -L/usr/lib/x86_64-linux-gnu
    #$ gcc -std=c++17 -D__CUDA_ARCH__=860 -D__CUDA_ARCH_LIST__=860 -E -x c++  -DCUDA_DOUBLE_MATH_FUNCTIONS -D__CUDACC__ -D__NVCC__ -D__CUDACC_EXTENDED_LAMBDA__ -D__CUDACC_RELAXED_CONSTEXPR__  -O3  -U "__CUDA_NO_HALF_OPERATORS__" -U "__CUDA_NO_HALF_CONVERSIONS__" -U "__CUDA_NO_HALF2_OPERATORS__" -U "__CUDA_NO_BFLOAT16_CONVERSIONS__" -D__CUDACC_VER_MAJOR__=11 -D__CUDACC_VER_MINOR__=5 -D__CUDACC_VER_BUILD__=119 -D__CUDA_API_VER_MAJOR__=11 -D__CUDA_API_VER_MINOR__=5 -DCUDA_API_PER_THREAD_DEFAULT_STREAM=1 -D__NVCC_DIAG_PRAGMA_SUPPORT__=1 -include "cuda_runtime.h" -m64 "src/cuda/nonzero_bitwise.cu" -o "/tmp/tmpxft_00009a2e_00000000-7_nonzero_bitwise.cpp1.ii"
    #$ cicc --c++17 --gnu_version=110400 --display_error_number --orig_src_file_name "src/cuda/nonzero_bitwise.cu" --orig_src_path_name "/tmp/pip-install-k3v_9rnc/mistralrs-cuda_d8edf27dd7b84807a46c329fbd384856/mistralrs-core/src/cuda/nonzero_bitwise.cu" --allow_managed --extended-lambda --relaxed_constexpr  -arch compute_86 -m64 --no-version-ident -ftz=1 -prec_div=0 -prec_sqrt=0 -fmad=1 -fast-math --gen_div_approx_ftz --include_file_name "tmpxft_00009a2e_00000000-3_nonzero_bitwise.fatbin.c" -tused --gen_module_id_file --module_id_file_name "/tmp/tmpxft_00009a2e_00000000-4_nonzero_bitwise.module_id" --gen_c_file_name "/tmp/tmpxft_00009a2e_00000000-6_nonzero_bitwise.cudafe1.c" --stub_file_name "/tmp/tmpxft_00009a2e_00000000-6_nonzero_bitwise.cudafe1.stub.c" --gen_device_file_name "/tmp/tmpxft_00009a2e_00000000-6_nonzero_bitwise.cudafe1.gpu"  "/tmp/tmpxft_00009a2e_00000000-7_nonzero_bitwise.cpp1.ii" -o "/tmp/tmpxft_00009a2e_00000000-6_nonzero_bitwise.ptx"
    #$ ptxas -arch=sm_86 -m64 "/tmp/tmpxft_00009a2e_00000000-6_nonzero_bitwise.ptx"  -o "/tmp/tmpxft_00009a2e_00000000-8_nonzero_bitwise.sm_86.cubin"
    #$ fatbinary -64 --cicc-cmdline="-ftz=1 -prec_div=0 -prec_sqrt=0 -fmad=1 " "--image3=kind=elf,sm=86,file=/tmp/tmpxft_00009a2e_00000000-8_nonzero_bitwise.sm_86.cubin" "--image3=kind=ptx,sm=86,file=/tmp/tmpxft_00009a2e_00000000-6_nonzero_bitwise.ptx" --embedded-fatbin="/tmp/tmpxft_00009a2e_00000000-3_nonzero_bitwise.fatbin.c"
    #$ rm /tmp/tmpxft_00009a2e_00000000-3_nonzero_bitwise.fatbin
    #$ gcc -std=c++17 -D__CUDA_ARCH_LIST__=860 -E -x c++ -D__CUDACC__ -D__NVCC__ -D__CUDACC_EXTENDED_LAMBDA__ -D__CUDACC_RELAXED_CONSTEXPR__  -O3  -U "__CUDA_NO_HALF_OPERATORS__" -U "__CUDA_NO_HALF_CONVERSIONS__" -U "__CUDA_NO_HALF2_OPERATORS__" -U "__CUDA_NO_BFLOAT16_CONVERSIONS__" -D__CUDACC_VER_MAJOR__=11 -D__CUDACC_VER_MINOR__=5 -D__CUDACC_VER_BUILD__=119 -D__CUDA_API_VER_MAJOR__=11 -D__CUDA_API_VER_MINOR__=5 -DCUDA_API_PER_THREAD_DEFAULT_STREAM=1 -D__NVCC_DIAG_PRAGMA_SUPPORT__=1 -include "cuda_runtime.h" -m64 "src/cuda/nonzero_bitwise.cu" -o "/tmp/tmpxft_00009a2e_00000000-5_nonzero_bitwise.cpp4.ii"
    #$ cudafe++ --c++17 --gnu_version=110400 --display_error_number --orig_src_file_name "src/cuda/nonzero_bitwise.cu" --orig_src_path_name "/tmp/pip-install-k3v_9rnc/mistralrs-cuda_d8edf27dd7b84807a46c329fbd384856/mistralrs-core/src/cuda/nonzero_bitwise.cu" --allow_managed --extended-lambda --relaxed_constexpr --m64 --parse_templates --gen_c_file_name "/tmp/tmpxft_00009a2e_00000000-6_nonzero_bitwise.cudafe1.cpp" --stub_file_name "tmpxft_00009a2e_00000000-6_nonzero_bitwise.cudafe1.stub.c" --module_id_file_name "/tmp/tmpxft_00009a2e_00000000-4_nonzero_bitwise.module_id" "/tmp/tmpxft_00009a2e_00000000-5_nonzero_bitwise.cpp4.ii"
    #$ gcc -std=c++17 -D__CUDA_ARCH__=860 -D__CUDA_ARCH_LIST__=860 -c -x c++  -DCUDA_DOUBLE_MATH_FUNCTIONS -O3 -m64 "/tmp/tmpxft_00009a2e_00000000-6_nonzero_bitwise.cudafe1.cpp" -o "/tmp/pip-install-k3v_9rnc/mistralrs-cuda_d8edf27dd7b84807a46c329fbd384856/target/release/build/mistralrs-core-b94df057dc002129/out/nonzero_bitwise-b50867152df76f01.o"
    /usr/include/c++/11/bits/std_function.h:435:145: error: parameter packs not expanded with ‘...’:
      435 |         function(_Functor&& __f)
          |                                                                                                                                                 ^
    /usr/include/c++/11/bits/std_function.h:435:145: note:         ‘_ArgTypes’
    /usr/include/c++/11/bits/std_function.h:530:146: error: parameter packs not expanded with ‘...’:
      530 |         operator=(_Functor&& __f)
          |                                                                                                                                                  ^
    /usr/include/c++/11/bits/std_function.h:530:146: note:         ‘_ArgTypes’
    # --error 0x1 --
    thread 'main' panicked at /home/MY_USERNAME/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen_cuda-0.1.5/src/lib.rs:262:21:
    nvcc error while executing compiling: "nvcc" "--gpu-architecture=sm_86" "-c" "-o" "/tmp/pip-install-k3v_9rnc/mistralrs-cuda_d8edf27dd7b84807a46c329fbd384856/target/release/build/mistralrs-core-b94df057dc002129/out/nonzero_bitwise-b50867152df76f01.o" "--default-stream" "per-thread" "-std=c++17" "-O3" "-U__CUDA_NO_HALF_OPERATORS__" "-U__CUDA_NO_HALF_CONVERSIONS__" "-U__CUDA_NO_HALF2_OPERATORS__" "-U__CUDA_NO_BFLOAT16_CONVERSIONS__" "--expt-relaxed-constexpr" "--expt-extended-lambda" "--use_fast_math" "--verbose" "src/cuda/nonzero_bitwise.cu"

    # stdout

    # stderr

    stack backtrace:
       0: rust_begin_unwind
       1: core::panicking::panic_fmt
       2: bindgen_cuda::Builder::build_lib::{{closure}}
       3: core::ops::function::impls::<impl core::ops::function::FnMut<A> for &F>::call_mut
       4: core::iter::adapters::map::map_try_fold::{{closure}}
       5: core::iter::traits::iterator::Iterator::try_fold
       6: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
       7: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
       8: <core::iter::adapters::take_while::TakeWhile<I,P> as core::iter::traits::iterator::Iterator>::try_fold
       9: <core::iter::adapters::take_while::TakeWhile<I,P> as core::iter::traits::iterator::Iterator>::fold
      10: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::fold
      11: core::iter::traits::iterator::Iterator::for_each
      12: <rayon::iter::noop::NoopConsumer as rayon::iter::plumbing::Folder<T>>::consume_iter
      13: <rayon::iter::while_some::WhileSomeFolder<C> as rayon::iter::plumbing::Folder<core::option::Option<T>>>::consume_iter
      14: <rayon::iter::map::MapFolder<C,F> as rayon::iter::plumbing::Folder<T>>::consume_iter
      15: <rayon::iter::map::MapFolder<C,F> as rayon::iter::plumbing::Folder<T>>::consume_iter
      16: rayon::iter::plumbing::Producer::fold_with
      17: rayon::iter::plumbing::bridge_producer_consumer::helper
      18: rayon::iter::plumbing::bridge_producer_consumer
      19: <rayon::iter::plumbing::bridge::Callback<C> as rayon::iter::plumbing::ProducerCallback<I>>::callback
      20: <rayon::slice::Iter<T> as rayon::iter::IndexedParallelIterator>::with_producer
      21: rayon::iter::plumbing::bridge
      22: <rayon::slice::Iter<T> as rayon::iter::ParallelIterator>::drive_unindexed
      23: <rayon::iter::map::Map<I,F> as rayon::iter::ParallelIterator>::drive_unindexed
      24: <rayon::iter::map::Map<I,F> as rayon::iter::ParallelIterator>::drive_unindexed
      25: <rayon::iter::while_some::WhileSome<I> as rayon::iter::ParallelIterator>::drive_unindexed
      26: rayon::iter::from_par_iter::<impl rayon::iter::FromParallelIterator<()> for ()>::from_par_iter
      27: rayon::iter::ParallelIterator::collect
      28: rayon::result::<impl rayon::iter::FromParallelIterator<core::result::Result<T,E>> for core::result::Result<C,E>>::from_par_iter
      29: rayon::iter::ParallelIterator::collect
      30: bindgen_cuda::Builder::build_lib
      31: build_script_build::main
      32: core::ops::function::FnOnce::call_once
    note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
  warning: build failed, waiting for other jobs to finish...
  💥 maturin failed
    Caused by: Failed to build a native library through cargo
    Caused by: Cargo build finished with "exit status: 101": `env -u CARGO PYO3_ENVIRONMENT_SIGNATURE="cpython-3.12-64bit" PYO3_PYTHON="/home/MY_USERNAME/miniconda3/envs/mistralrs/bin/python" PYTHON_SYS_EXECUTABLE="/home/MY_USERNAME/miniconda3/envs/mistralrs/bin/python" "cargo" "rustc" "--profile" "release" "--features" "pyo3/extension-module" "--message-format" "json-render-diagnostics" "--manifest-path" "/tmp/pip-install-k3v_9rnc/mistralrs-cuda_d8edf27dd7b84807a46c329fbd384856/mistralrs-pyo3/Cargo.toml" "--lib"`
  Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/home/MY_USERNAME/miniconda3/envs/mistralrs/bin/python', '--compatibility', 'off'] returned non-zero exit status 1
  error: subprocess-exited-with-error

  × Building wheel for mistralrs-cuda (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /home/MY_USERNAME/miniconda3/envs/mistralrs/bin/python /home/MY_USERNAME/miniconda3/envs/mistralrs/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /tmp/tmp8she9qat
  cwd: /tmp/pip-install-k3v_9rnc/mistralrs-cuda_d8edf27dd7b84807a46c329fbd384856
  Building wheel for mistralrs-cuda (pyproject.toml) ... error
  ERROR: Failed building wheel for mistralrs-cuda
Failed to build mistralrs-cuda
ERROR: Could not build wheels for mistralrs-cuda, which is required to install pyproject.toml-based projects

Latest commit Which commit you ran with.

EricLBuehler commented 2 months ago

Hi @jaschn12!

Can you please let me know what the output of 

nvcc --version
gcc --version

Are?

jaschn12 commented 2 months ago 
(base) MY_USERNAME@MY_SERVER_NAME:~$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Thu_Nov_18_09:45:30_PST_2021
Cuda compilation tools, release 11.5, V11.5.119
Build cuda_11.5.r11.5/compiler.30672275_0

(base) MY_USERNAME@MY_SERVER_NAME:~$ gcc --version
gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
EricLBuehler commented 1 month ago

@jaschn12 no idea unfortunately. @guoqingbao have you seen anything like this by chance?

guoqingbao commented 1 month ago

One of the CUDA source files (src/cuda/nonzero_bitwise.cu) is unable to be compiled because it is not compatible with the system C++ library under current NVCC compiler. Some people said it was a bug in a specific CUDA version. I suggest @jaschn12 use a different CUDA package.

https://stackoverflow.com/questions/74350584/nvcc-compilation-error-using-thrust-in-cuda-11-5

Root cause:

$ gcc -std=c++17 -D__CUDA_ARCH=860 -DCUDA_ARCH_LIST__=860 -c -x c++ -DCUDA_DOUBLE_MATH_FUNCTIONS -O3 -m64 "/tmp/tmpxft_00009a2e_00000000-6_nonzero_bitwise.cudafe1.cpp" -o "/tmp/pip-install-k3v_9rnc/mistralrs-cuda_d8edf27dd7b84807a46c329fbd384856/target/release/build/mistralrs-core-b94df057dc002129/out/nonzero_bitwise-b50867152df76f01.o"

/usr/include/c++/11/bits/std_function.h:435:145: error: parameter packs not expanded with ‘...’:
  435 |         function(_Functor&& __f)
      |                                                                                                                                                 ^
/usr/include/c++/11/bits/std_function.h:435:145: note:         ‘_ArgTypes’
/usr/include/c++/11/bits/std_function.h:530:146: error: parameter packs not expanded with ‘...’:
  530 |         operator=(_Functor&& __f)
      |                                                                                                                                                  ^
/usr/include/c++/11/bits/std_function.h:530:146: note:         ‘_ArgTypes’
# --error 0x1 --
thread 'main' panicked at /home/MY_USERNAME/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen_cuda-0.1.5/src/lib.rs:262:21:
nvcc error while executing compiling: "nvcc" "--gpu-architecture=sm_86" "-c" "-o" "/tmp/pip-install-k3v_9rnc/mistralrs-cuda_d8edf27dd7b84807a46c329fbd384856/target/release/build/mistralrs-core-b94df057dc002129/out/nonzero_bitwise-b50867152df76f01.o" "--default-stream" "per-thread" "-std=c++17" "-O3" "-U__CUDA_NO_HALF_OPERATORS__" "-U__CUDA_NO_HALF_CONVERSIONS__" "-U__CUDA_NO_HALF2_OPERATORS__" "-U__CUDA_NO_BFLOAT16_CONVERSIONS__" "--expt-relaxed-constexpr" "--expt-extended-lambda" "--use_fast_math" "--verbose" "src/cuda/nonzero_bitwise.cu"

Perhaps he's using Windows Subsystem for Linux (WSL) for compiling Mistral.rs, do we support WSL at the moment?

EricLBuehler commented 1 month ago

Thanks for getting back @guoqingbao. @jaschn12 perhaps you could upgrade your CUDA package - upgrading to 11.8 worked in the linked thread.

Perhaps he's using Windows Subsystem for Linux (WSL) for compiling Mistral.rs, do we support WSL at the moment?

Yes, we support WSL.

EricLBuehler commented 1 month ago

@jaschn12 I think I found a fix. Given that you are on GCC 11, this is actually a compiler bug! Please install GCC 10 and run with NVCC_CCBIN=/usr/lib/gcc/x86_64-linux-gnu/10.