Pip Installation

[rmccorm4]

Hi folks,

I recently discovered a tensorrt_llm pip package was published as described here: https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/installation.md#install-from-the-wheel-package

I was hoping given this, it would be possible to pip install optimum-nvidia within another container, but ran into some issues described below.

Package sanity check:

root@ced35d0-lcedt:/mnt/triton/jira/optimum/optimum-nvidia/examples/text-generation# python3
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorrt_llm
>>> tensorrt_llm.__version__
'0.7.1'
>>> import optimum
>>> import transformers
>>>

Try to run example:

root@ced35d0-lcedt:/mnt/triton/jira/optimum/optimum-nvidia/examples/text-generation# python3 llama.py
Traceback (most recent call last):
  File "/mnt/triton/jira/optimum/optimum-nvidia/examples/text-generation/llama.py", line 22, in <module>
    from optimum.nvidia import setup_logging
ModuleNotFoundError: No module named 'optimum.nvidia'

root@ced35d0-lcedt:/mnt/triton/jira/optimum/optimum-nvidia/examples/text-generation# export PYTHONPATH=/mnt/triton/jira/optimum/optimum-nvidia/src:$PYTHONPATH

root@ced35d0-lcedt:/mnt/triton/jira/optimum/optimum-nvidia/examples/text-generation# python3 llama.py
Traceback (most recent call last):
  File "/mnt/triton/jira/optimum/optimum-nvidia/examples/text-generation/llama.py", line 22, in <module>
    from optimum.nvidia import setup_logging
  File "/mnt/triton/jira/optimum/optimum-nvidia/src/optimum/nvidia/__init__.py", line 28, in <module>
    from .runtime import TensorRTForCausalLM, TensorRTPreTrainedModel
  File "/mnt/triton/jira/optimum/optimum-nvidia/src/optimum/nvidia/runtime.py", line 7, in <module>
    import tensorrt_llm.bindings as ctrrt
ModuleNotFoundError: No module named 'tensorrt_llm.bindings'

(I also tried pip installing the root of this repo containing the pyproject.toml, but that didn't help either)

---

Is there a set of commands I can follow to successfully pip install optimum-nvidia using pre-built or pre-installed TRT LLM wheels? Any help would be greatly appreciated!

[JohnGiorgi]

Also very interested in a pip installable optimum-nvidia as its difficult to run dockers on our cluster

[viktor-ferenczi]

"An Optimum-NVIDIA package that can be installed with pip will be made available soon."

[Auth0rM0rgan]

@viktor-ferenczi any update regarding pip installation?

[viktor-ferenczi]

I don't work on it, just quoted the information as an answer here, because I also wanted to use it from pip and found this ticket.