Open xianbaoqian opened 3 weeks ago
Hey @xianbaoqian, could you share a bit about what is the exact problem you're seeing? I see two problems above, and I'm unsure of which one is affecting you, I see this one first:
(base) root@C.11121520:$ conda conda create -n text-generation-inference python=3.11
usage: conda [-h] [-v] [--no-plugins] [-V] COMMAND ...
conda: error: argument COMMAND: invalid choice: 'conda' (choose from 'activate', 'deactivate', 'clean', 'compare'
, 'config', 'create', 'env', 'export', 'info', 'init', 'install', 'list', 'notices', 'package', 'remove', 'uninst
all', 'rename', 'run', 'search', 'update', 'upgrade', 'content-trust', 'doctor', 'repoquery', 'build', 'develop',
'metapackage', 'render', 'index', 'debug', 'inspect', 'convert', 'skeleton')
Here it seems like you're calling conda
twice.
I'm happy to help out, but it would be helpful if the formatting of the issue was easier to parse :hugs: thanks!
Hi @LysandreJik the problem is that after conda activate I still can't use text-generation-launcher as the binary file can't be found:
(base) root@C.11121520:$ conda activate text-generation-inference (text-generation-inference) root@C.11121520:$ text-generation-launcher --model-id mistralai/Mistral-7B-Instruct- v0.2 bash: text-generation-launcher: command not found
I updated the style (surrending it with a code block) to make it easier to read. The original format is indeed a bit misleading!
Hey @xianbaoqian, thanks for clearing the style. There is definitely an issue in the code you run:
$ conda create -n text-generation-inference python=3.11
$ conda activate text-generation-inference
$ text-generation-launcher --model-id mistralai/Mistral-7B-Instruct-v0.2
bash: text-generation-launcher: command not found
Here you're only creating a new conda environment and activating it. You named it text-generation-inference
, but that doesn't mean that it's installed.
If you took the example from this part of the README, it seems that you forgot this line:
+ BUILD_EXTENSIONS=True make install # Install repository and HF/transformer fork with CUDA kernels
text-generation-launcher --model-id mistralai/Mistral-7B-Instruct-v0.2
Beware: it's a long installation. Using TGI with the docker image is recommended as much faster.
I had the same issue. Resolved by adding "$HOME/text-generation-inference/target/release-opt/" to PATH. Change $HOME to wherever your text-generation-inference directory is.
On another note, cargo (or Rust) version 1.78.0 will not work as the Rust source files are written using inline const feature which is only available in cargo >= 1.79.0 .
System Info
uname -a Linux a3eb1d6a21b4 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
cargo --version cargo 1.78.0 (54d8815d0 2024-03-26)
nvidia-smi Wed Jun 12 07:26:30 2024
+-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 550.67 Driver Version: 550.67 CUDA Version: 12.4 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce RTX 4090 On | 00000000:81:00.0 Off | Off | | 30% 28C P8 17W / 450W | 1MiB / 24564MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+
Information
Tasks
Reproduction
Expected behavior
expects TGI to be running correctly without issues since the above is copied from installation section in the readme.md