search

khoj-ai / khoj

Your AI second brain. Self-hostable. Get answers from the web or your docs. Build custom agents, schedule automations, do deep research. Turn any online or local LLM into your personal, autonomous AI (e.g gpt, claude, gemini, llama, qwen, mistral).
https://khoj.dev
GNU Affero General Public License v3.0
14.27k stars 706 forks source link

Failed building wheel for safetensors #477

Closed HamedMP closed 1 year ago

HamedMP commented 1 year ago

Hi,

Super interesting project.

I created a new conda env with python 11 conda create -n khoj python=3.11 and ran the pip installation, but got the following error.

Any idea how to resolve it?

I'm on Mac M1, with the latest MacOS 13.5.1. 

      Some errors have detailed explanations: E0405, E0412, E0425, E0433, E0463, E0531.
      For more information about an error, try `rustc --explain E0405`.
      error: could not compile `smallvec` due to 145 previous errors

      Caused by:
        process didn't exit successfully: `rustc --crate-name smallvec --edition=2018 /Users/hamed/.cargo/registry/src/github.com-1ecc6299db9ec823/smallvec-1.11.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C metadata=b41f1767cebac6a1 -C extra-filename=-b41f1767cebac6a1 --out-dir /private/var/folders/zl/m2lw7vkd495djcf9x2499md80000gn/T/pip-install-zmbcfh5z/safetensors_22bdd82a17b14939a5597ba399186ebe/target/x86_64-apple-darwin/release/deps --target x86_64-apple-darwin -L dependency=/private/var/folders/zl/m2lw7vkd495djcf9x2499md80000gn/T/pip-install-zmbcfh5z/safetensors_22bdd82a17b14939a5597ba399186ebe/target/x86_64-apple-darwin/release/deps -L dependency=/private/var/folders/zl/m2lw7vkd495djcf9x2499md80000gn/T/pip-install-zmbcfh5z/safetensors_22bdd82a17b14939a5597ba399186ebe/target/release/deps --cap-lints allow` (exit status: 1)
      warning: build failed, waiting for other jobs to finish...
      error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --target x86_64-apple-darwin --release -v --features pyo3/extension-module --crate-type cdylib -- -C 'link-args=-undefined dynamic_lookup -Wl,-install_name,@rpath/_safetensors_rust.cpython-311-darwin.so'` failed with code 101
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for safetensors
Failed to build safetensors
ERROR: Could not build wheels for safetensors, which is required to install pyproject.toml-based projects
sabaimran commented 1 year ago

Hey @HamedMP ! Sorry about the setup issues. i've seen this before, and the root cause is usually a missing rust installation.

these instructions would be helpful for installing rust -- https://docs.khoj.dev/#/setup?id=install-fails-while-building-tokenizer-dependency. Could you try that out and let me know if it works?

brew install rustup
rustup-init
source ~/.cargo/env
HamedMP commented 1 year ago

Thanks @sabaimran, I've tried rustup-init but I get the error that PATH already exist. 

error: could not create link from '/Users/hamed/.cargo/bin/rustup' to '/Users/hamed/.cargo/bin/rustfmt': File exists (os error 17)

Running rustc --version gives:

❯ rustc --version
rustc 1.66.0 (69f9c33d7 2022-12-12)
HamedMP commented 1 year ago

I just looked up safetensors specifically and found this thread

Installing rust in my conda environment resolved the issue. So it was mainly conda-related. 

❯ conda install -c conda-forge rust

P.s. After running khoj, I didn't see any info that I need to go to http://127.0.0.1:42110 to see the interface. Adding it to the doc would be amazing <3

Found the URL from the Obsidian plugin settings 😄

image