jdemouth-nvidia
commented
11 months ago How did you build TensorRT-LLM?
Open namang301 opened 11 months ago
jdemouth-nvidia
commented
11 months ago How did you build TensorRT-LLM?
namang301
commented
11 months ago Thanks for reply.
I had installed TensorRT-LLM using :
$ pip3 install tensorrt_llm --extra-index-url https://pypi.nvidia.com/ --extra-index-url https://download.pytorch.org/whl/cu122
because, when I build using $ python3 ./scripts/build_wheel.py --trt_root /usr/local/tensorrt,
I got this message
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com/ Requirement already satisfied: accelerate==0.20.3 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 1)) (0.20.3) Requirement already satisfied: build in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 2)) (1.0.3) Requirement already satisfied: colored in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 3)) (2.2.4) Requirement already satisfied: cuda-python==12.2.0 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 4)) (12.2.0) Requirement already satisfied: diffusers==0.15.0 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 5)) (0.15.0) Requirement already satisfied: lark in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 6)) (1.1.8) Requirement already satisfied: mpi4py in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 7)) (3.1.5) Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 8)) (1.26.1) Requirement already satisfied: onnx>=1.12.0 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 9)) (1.15.0) Requirement already satisfied: polygraphy in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 10)) (0.49.0) Requirement already satisfied: sentencepiece>=0.1.99 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 11)) (0.1.99) Requirement already satisfied: tensorrt>=8.6.0 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 12)) (9.1.0.post12.dev4) Requirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 13)) (2.1.0) Requirement already satisfied: transformers==4.33.1 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 14)) (4.33.1) Requirement already satisfied: wheel in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 15)) (0.41.2) Requirement already satisfied: optimum in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 16)) (1.16.1) Requirement already satisfied: evaluate in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 17)) (0.4.1) Requirement already satisfied: datasets in /usr/local/lib/python3.10/dist-packages (from -r requirements-dev.txt (line 2)) (2.16.0) Requirement already satisfied: einops in /usr/local/lib/python3.10/dist-packages (from -r requirements-dev.txt (line 3)) (0.7.0) Requirement already satisfied: graphviz in /usr/local/lib/python3.10/dist-packages (from -r requirements-dev.txt (line 5)) (0.20.1) Requirement already satisfied: mypy in /usr/local/lib/python3.10/dist-packages (from -r requirements-dev.txt (line 6)) (1.8.0) Requirement already satisfied: parameterized in /usr/local/lib/python3.10/dist-packages (from -r requirements-dev.txt (line 7)) (0.9.0) Requirement already satisfied: pre-commit in /usr/local/lib/python3.10/dist-packages (from -r requirements-dev.txt (line 8)) (3.6.0) Requirement already satisfied: pybind11-stubgen in /usr/local/lib/python3.10/dist-packages (from -r requirements-dev.txt (line 9)) (2.4.2) Requirement already satisfied: pynvml>=11.5.0 in /usr/local/lib/python3.10/dist-packages (from -r requirements-dev.txt (line 10)) (11.5.0) Requirement already satisfied: pytest-cov in /usr/local/lib/python3.10/dist-packages (from -r requirements-dev.txt (line 11)) (4.1.0) Requirement already satisfied: pytest-forked in /usr/local/lib/python3.10/dist-packages (from -r requirements-dev.txt (line 12)) (1.6.0) Requirement already satisfied: pytest-xdist in /usr/local/lib/python3.10/dist-packages (from -r requirements-dev.txt (line 13)) (3.5.0) Requirement already satisfied: rouge_score in /usr/local/lib/python3.10/dist-packages (from -r requirements-dev.txt (line 14)) (0.1.2) Requirement already satisfied: typing-extensions==4.8.0 in /usr/local/lib/python3.10/dist-packages (from -r requirements-dev.txt (line 16)) (4.8.0) Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from accelerate==0.20.3->-r requirements.txt (line 1)) (23.2) Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate==0.20.3->-r requirements.txt (line 1)) (5.9.7) Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from accelerate==0.20.3->-r requirements.txt (line 1)) (6.0.1) Requirement already satisfied: cython in /usr/local/lib/python3.10/dist-packages (from cuda-python==12.2.0->-r requirements.txt (line 4)) (3.0.7) Requirement already satisfied: Pillow in /usr/local/lib/python3.10/dist-packages (from diffusers==0.15.0->-r requirements.txt (line 5)) (10.1.0) Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from diffusers==0.15.0->-r requirements.txt (line 5)) (3.12.4) Requirement already satisfied: huggingface-hub>=0.13.2 in /usr/local/lib/python3.10/dist-packages (from diffusers==0.15.0->-r requirements.txt (line 5)) (0.20.1) Requirement already satisfied: importlib-metadata in /usr/lib/python3/dist-packages (from diffusers==0.15.0->-r requirements.txt (line 5)) (4.6.4) Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from diffusers==0.15.0->-r requirements.txt (line 5)) (2023.10.3) Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from diffusers==0.15.0->-r requirements.txt (line 5)) (2.31.0) Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.10/dist-packages (from transformers==4.33.1->-r requirements.txt (line 14)) (0.13.3) Requirement already satisfied: safetensors>=0.3.1 in /usr/local/lib/python3.10/dist-packages (from transformers==4.33.1->-r requirements.txt (line 14)) (0.4.0) Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers==4.33.1->-r requirements.txt (line 14)) (4.66.1) Requirement already satisfied: pyproject_hooks in /usr/local/lib/python3.10/dist-packages (from build->-r requirements.txt (line 2)) (1.0.0) Requirement already satisfied: tomli>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from build->-r requirements.txt (line 2)) (2.0.1) Requirement already satisfied: protobuf>=3.20.2 in /usr/local/lib/python3.10/dist-packages (from onnx>=1.12.0->-r requirements.txt (line 9)) (4.25.1) Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch->-r requirements.txt (line 13)) (1.12) Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch->-r requirements.txt (line 13)) (3.2) Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch->-r requirements.txt (line 13)) (3.1.2) Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch->-r requirements.txt (line 13)) (2023.9.2) Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch->-r requirements.txt (line 13)) (12.1.105) Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch->-r requirements.txt (line 13)) (12.1.105) Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch->-r requirements.txt (line 13)) (12.1.105) Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /usr/local/lib/python3.10/dist-packages (from torch->-r requirements.txt (line 13)) (8.9.2.26) Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /usr/local/lib/python3.10/dist-packages (from torch->-r requirements.txt (line 13)) (12.1.3.1) Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /usr/local/lib/python3.10/dist-packages (from torch->-r requirements.txt (line 13)) (11.0.2.54) Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /usr/local/lib/python3.10/dist-packages (from torch->-r requirements.txt (line 13)) (10.3.2.106) Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /usr/local/lib/python3.10/dist-packages (from torch->-r requirements.txt (line 13)) (11.4.5.107) Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /usr/local/lib/python3.10/dist-packages (from torch->-r requirements.txt (line 13)) (12.1.0.106) Requirement already satisfied: nvidia-nccl-cu12==2.18.1 in /usr/local/lib/python3.10/dist-packages (from torch->-r requirements.txt (line 13)) (2.18.1) Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch->-r requirements.txt (line 13)) (12.1.105) Requirement already satisfied: triton==2.1.0 in /usr/local/lib/python3.10/dist-packages (from torch->-r requirements.txt (line 13)) (2.1.0) Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.10/dist-packages (from nvidia-cusolver-cu12==11.4.5.107->torch->-r requirements.txt (line 13)) (12.3.101) Requirement already satisfied: coloredlogs in /usr/local/lib/python3.10/dist-packages (from optimum->-r requirements.txt (line 16)) (15.0.1) Requirement already satisfied: dill in /usr/local/lib/python3.10/dist-packages (from evaluate->-r requirements.txt (line 17)) (0.3.7) Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from evaluate->-r requirements.txt (line 17)) (2.1.4) Requirement already satisfied: xxhash in /usr/local/lib/python3.10/dist-packages (from evaluate->-r requirements.txt (line 17)) (3.4.1) Requirement already satisfied: multiprocess in /usr/local/lib/python3.10/dist-packages (from evaluate->-r requirements.txt (line 17)) (0.70.15) Requirement already satisfied: responses<0.19 in /usr/local/lib/python3.10/dist-packages (from evaluate->-r requirements.txt (line 17)) (0.18.0) Requirement already satisfied: pyarrow>=8.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets->-r requirements-dev.txt (line 2)) (14.0.2) Requirement already satisfied: pyarrow-hotfix in /usr/local/lib/python3.10/dist-packages (from datasets->-r requirements-dev.txt (line 2)) (0.6) Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets->-r requirements-dev.txt (line 2)) (3.9.1) Requirement already satisfied: mypy-extensions>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from mypy->-r requirements-dev.txt (line 6)) (1.0.0) Requirement already satisfied: cfgv>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from pre-commit->-r requirements-dev.txt (line 8)) (3.4.0) Requirement already satisfied: identify>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from pre-commit->-r requirements-dev.txt (line 8)) (2.5.33) Requirement already satisfied: nodeenv>=0.11.1 in /usr/local/lib/python3.10/dist-packages (from pre-commit->-r requirements-dev.txt (line 8)) (1.8.0) Requirement already satisfied: virtualenv>=20.10.0 in /usr/local/lib/python3.10/dist-packages (from pre-commit->-r requirements-dev.txt (line 8)) (20.25.0) Requirement already satisfied: pytest>=4.6 in /usr/local/lib/python3.10/dist-packages (from pytest-cov->-r requirements-dev.txt (line 11)) (7.4.3) Requirement already satisfied: coverage>=5.2.1 in /usr/local/lib/python3.10/dist-packages (from coverage[toml]>=5.2.1->pytest-cov->-r requirements-dev.txt (line 11)) (7.4.0) Requirement already satisfied: py in /usr/local/lib/python3.10/dist-packages (from pytest-forked->-r requirements-dev.txt (line 12)) (1.11.0) Requirement already satisfied: execnet>=1.1 in /usr/local/lib/python3.10/dist-packages (from pytest-xdist->-r requirements-dev.txt (line 13)) (2.0.2) Requirement already satisfied: absl-py in /usr/local/lib/python3.10/dist-packages (from rouge_score->-r requirements-dev.txt (line 14)) (2.0.0) Requirement already satisfied: nltk in /usr/local/lib/python3.10/dist-packages (from rouge_score->-r requirements-dev.txt (line 14)) (3.8.1) Requirement already satisfied: six>=1.14.0 in /usr/lib/python3/dist-packages (from rouge_score->-r requirements-dev.txt (line 14)) (1.16.0) Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements-dev.txt (line 2)) (23.1.0) Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements-dev.txt (line 2)) (6.0.4) Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements-dev.txt (line 2)) (1.9.4) Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements-dev.txt (line 2)) (1.4.1) Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements-dev.txt (line 2)) (1.3.1) Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->-r requirements-dev.txt (line 2)) (4.0.3) Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from nodeenv>=0.11.1->pre-commit->-r requirements-dev.txt (line 8)) (68.2.2) Requirement already satisfied: iniconfig in /usr/local/lib/python3.10/dist-packages (from pytest>=4.6->pytest-cov->-r requirements-dev.txt (line 11)) (2.0.0) Requirement already satisfied: pluggy<2.0,>=0.12 in /usr/local/lib/python3.10/dist-packages (from pytest>=4.6->pytest-cov->-r requirements-dev.txt (line 11)) (1.3.0) Requirement already satisfied: exceptiongroup>=1.0.0rc8 in /usr/local/lib/python3.10/dist-packages (from pytest>=4.6->pytest-cov->-r requirements-dev.txt (line 11)) (1.2.0) Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->diffusers==0.15.0->-r requirements.txt (line 5)) (3.3.0) Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->diffusers==0.15.0->-r requirements.txt (line 5)) (3.4) Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->diffusers==0.15.0->-r requirements.txt (line 5)) (2.0.7) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->diffusers==0.15.0->-r requirements.txt (line 5)) (2023.7.22) Requirement already satisfied: distlib<1,>=0.3.7 in /usr/local/lib/python3.10/dist-packages (from virtualenv>=20.10.0->pre-commit->-r requirements-dev.txt (line 8)) (0.3.8) Requirement already satisfied: platformdirs<5,>=3.9.1 in /usr/local/lib/python3.10/dist-packages (from virtualenv>=20.10.0->pre-commit->-r requirements-dev.txt (line 8)) (4.1.0) Requirement already satisfied: humanfriendly>=9.1 in /usr/local/lib/python3.10/dist-packages (from coloredlogs->optimum->-r requirements.txt (line 16)) (10.0) Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch->-r requirements.txt (line 13)) (2.1.3) Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from nltk->rouge_score->-r requirements-dev.txt (line 14)) (8.1.7) Requirement already satisfied: joblib in /usr/local/lib/python3.10/dist-packages (from nltk->rouge_score->-r requirements-dev.txt (line 14)) (1.3.2) Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas->evaluate->-r requirements.txt (line 17)) (2.8.2) Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->evaluate->-r requirements.txt (line 17)) (2023.3.post1) Requirement already satisfied: tzdata>=2022.1 in /usr/local/lib/python3.10/dist-packages (from pandas->evaluate->-r requirements.txt (line 17)) (2023.3) Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch->-r requirements.txt (line 13)) (1.3.0) WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv [notice] A new release of pip is available: 23.3 -> 23.3.2 [notice] To update, run: python3 -m pip install --upgrade pip -- The CXX compiler identification is GNU 11.4.0 -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /usr/bin/c++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- NVTX is disabled -- Importing batch manager -- Building PyTorch -- Building Google tests -- Building benchmarks -- Looking for a CUDA compiler -- Looking for a CUDA compiler - NOTFOUND CMake Error at CMakeLists.txt:118 (message): No CUDA compiler found
-- Configuring incomplete, errors occurred! See also "{mydir}/TensorRT-LLM/cpp/build/CMakeFiles/CMakeOutput.log". See also "{mydir}/TensorRT-LLM/cpp/build/CMakeFiles/CMakeError.log". Traceback (most recent call last): File "{mydir}/TensorRT-LLM/./scripts/build_wheel.py", line 306, in
main(**vars(args)) File "{mydir}/TensorRT-LLM/./scripts/build_wheel.py", line 160, in main build_run( File "/usr/lib/python3.10/subprocess.py", line 526, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command 'cmake -DCMAKE_BUILD_TYPE="Release" -DBUILD_PYT="ON" -DBUILD_PYBIND="OFF" -DTRT_LIB_DIR=/usr/local/tensorrt/targets/x86_64-linux-gnu/lib -DTRT_INCLUDE_DIR=/usr/local/tensorrt/include -S "{mydir}/TensorRT-LLM/cpp"' returned non-zero exit status 1
I need to build TensorRT-LLM again? then how I can build clear this error?
Shixiaowei02
commented
11 months ago -- Looking for a CUDA compiler
-- Looking for a CUDA compiler - NOTFOUND
CMake Error at CMakeLists.txt:118 (message):
No CUDA compiler foundAccording the error message, please verify that CUDA is installed correctly. Thank you!
namang301
commented
11 months ago Thanks for reply. after I did check my CUDA, when build TensorRT-LLM again, I got this error.
{skip logs...} [100%] Building CUDA object tensorrt_llm/kernels/CMakeFiles/kernels_src.dir/weightOnlyBatchedGemv/weightOnlyBatchedGemvBs2Int8b.cu.o [100%] Building CUDA object tensorrt_llm/kernels/CMakeFiles/kernels_src.dir/weightOnlyBatchedGemv/weightOnlyBatchedGemvBs4Int4b.cu.o [100%] Building CUDA object tensorrt_llm/kernels/CMakeFiles/kernels_src.dir/weightOnlyBatchedGemv/weightOnlyBatchedGemvBs4Int8b.cu.o [100%] Building CUDA object tensorrt_llm/kernels/CMakeFiles/kernels_src.dir/weightOnlyBatchedGemv/weightOnlyBatchedGemvBs3Int8b.cu.o [100%] Built target layers_src [100%] Built target common_src [100%] Built target runtime_src [100%] Built target kernels_src [100%] Linking CXX static library libtensorrt_llm_static.a [100%] Built target tensorrt_llm_static [100%] Linking CXX shared library libtensorrt_llm.so /usr/bin/ld: ../../tensorrt_llm/batch_manager/x86_64-linux-gnu/libtensorrt_llm_batch_manager_static.pre_cxx11.a(kvCacheManager.cpp.o): in function
tensorrt_llm::batch_manager::kv_cache_manager::KVCacheManager::getMaxNumTokens(tensorrt_llm::batch_manager::kv_cache_manager::KvCacheConfig const&, nvinfer1::DataType, tensorrt_llm::runtime::GptModelConfig const&, tensorrt_llm::runtime::WorldConfig const&, tensorrt_llm::runtime::BufferManager const&)': kvCacheManager.cpp:(.text+0x213e): undefined reference toompi_mpi_comm_world' collect2: error: ld returned 1 exit status gmake[3]: [tensorrt_llm/CMakeFiles/tensorrt_llm.dir/build.make:1223: tensorrt_llm/libtensorrt_llm.so] Error 1 gmake[2]: [CMakeFiles/Makefile2:727: tensorrt_llm/CMakeFiles/tensorrt_llm.dir/all] Error 2 gmake[1]: [CMakeFiles/Makefile2:734: tensorrt_llm/CMakeFiles/tensorrt_llm.dir/rule] Error 2 gmake: [Makefile:179: tensorrt_llm] Error 2 Traceback (most recent call last): File "{mydir}/TensorRT-LLM/./scripts/build_wheel.py", line 306, inmain(**vars(args)) File "{mydir}/TensorRT-LLM/./scripts/build_wheel.py", line 164, in main build_run( File "/usr/lib/python3.10/subprocess.py", line 526, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command 'cmake --build . --config Release --parallel 255 --target tensorrt_llm tensorrt_llm_static nvinfer_plugin_tensorrt_llm th_common ' returned non-zero exit status 2.
I already checked #617, but I couldn't solve that.
I think it already setting about mpi (/usr/local/mpi ( -> /opt/hcpx/ompi)) in "nvcr.io-nvidia-tritionserver-23.10-trtllm-python-py3".
but when I build tensorRT-LLM, I had to use $ apt install mpich, because of installation of 'mpi4py'.
and I had add below:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/mpi/lib
export OPAL_PREFIX=/opt/hpcx/ompimaybe I guess I will need to change linking about openmpi, but I don't know how its can.. how I could solve that?
namang301
commented
11 months ago @Shixiaowei02 Is there any way I can solve this problem?
Shixiaowei02
commented
11 months ago Please try running these commands. Thank you!
# Pull and launch the Docker container
docker run --rm -it --entrypoint bash nvidia/cuda:12.2.2-devel-ubuntu22.04
# Install dependencies, TensorRT-LLM requires Python 3.10
apt-get update && apt-get -y install python3.10 python3-pip openmpi-bin libopenmpi-dev
# Install the latest version of TensorRT-LLM
pip3 install tensorrt_llm -U --extra-index-url https://pypi.nvidia.com
# Check installation
python3 -c "import tensorrt_llm; print(tensorrt_llm.__version__)"
Nam-ang
commented
10 months ago @Shixiaowei02 Thanks, I build my docker image again.
and I now meet new error, when I run below
$ python3 ../run.py --max_output_len=50 \ --tokenizer_dir meta-llama/llama-2-7b-hf \ --engine_dir=./tmp/llama/out/
error message is:
... { skip error message } ...
hello-11
Hi, I had tried to test llama2 based on TensorRT-LLM.
my environments (based on "nvcr.io-nvidia-tritionserver-23.10-trtllm-python-py3"):
I tried to use llama2 based on examples/llama/README
I had success build, when I run :
$ python build.py --model_dir meta-llama/llama-2-7b-hf\ --dtype float16 \ --remove_input_padding \ --use_gpt_attention_plugin float16 \ --enable_context_fmha \ --use_gemm_plugin float16 \ --output_dir ./tmp/llama/out/when i tried to run engine after build :
$ python3 ../run.py --max_output_len=50 \ --tokenizer_dir meta-llama/llama-2-7b-hf \ --engine_dir=./tmp/llama/out/I faced to this error message
When I tested GPT-NeoX Examples (using Polyglot Models), I got same issue. I can use build.py that, but I can't use run.py that and got same error message.
I guess it is because of mpi.
Can any body help me solve this problem?