[BUG: ModuleNotFoundError: No module named 'triton']

[MaxAkbar]

Python -VV

Python 3.11.9 | packaged by Anaconda, Inc. | (main, Apr 19 2024, 16:40:41) [MSC v.1916 64 bit (AMD64)]

Pip Freeze

annotated-types==0.7.0
asttokens==2.4.1
attrs==23.2.0
Brotli @ file:///C:/b/abs_3d36mno480/croot/brotli-split_1714483178642/work
certifi @ file:///C:/b/abs_7bbxxprxir/croot/certifi_1720453609121/work/certifi
charset-normalizer @ file:///tmp/build/80754af9/charset-normalizer_1630003229654/work
colorama==0.4.6
comm==0.2.0
debugpy==1.8.0
decorator==5.1.1
docstring_parser==0.16
executing==2.0.1
filelock @ file:///C:/b/abs_f2gie28u58/croot/filelock_1700591233643/work
fire==0.6.0
fsspec==2024.6.1
gmpy2 @ file:///C:/ci_311/gmpy2_1677743390134/work
huggingface-hub==0.24.0
idna @ file:///C:/b/abs_aad84bnnw5/croot/idna_1714398896795/work
intel-openmp==2021.4.0
ipython==8.18.1
jedi==0.19.1
Jinja2 @ file:///C:/b/abs_92fccttino/croot/jinja2_1716993447201/work
jsonschema==4.21.1
jsonschema-specifications==2023.12.1
jupyter_client==8.6.0
jupyter_core==5.5.0
MarkupSafe @ file:///C:/b/abs_ecfdqh67b_/croot/markupsafe_1704206030535/work
matplotlib-inline==0.1.6
mistral_common==1.3.3
mistral_inference==1.3.1
mkl==2021.4.0
mkl-fft==1.3.1
mkl-random @ file:///C:/ci_311/mkl_random_1676481991689/work
mkl-service==2.4.0
mpmath @ file:///C:/b/abs_7833jrbiox/croot/mpmath_1690848321154/work
networkx @ file:///C:/b/abs_36fmumtynt/croot/networkx_1720002497414/work
numpy @ file:///C:/b/abs_54abayvc9j/croot/numpy_and_numpy_base_1682520598361/work
packaging==24.1
parso==0.8.3
pillow @ file:///C:/b/abs_32o8er3uqp/croot/pillow_1721059447598/work
platformdirs==4.0.0
prompt-toolkit==3.0.41
pure-eval==0.2.2
pydantic==2.6.1
pydantic_core==2.16.2
PySocks @ file:///C:/ci_311/pysocks_1676425991111/work
PyYAML @ file:///C:/b/abs_782o3mbw7z/croot/pyyaml_1698096085010/work
pyzmq==25.1.1
referencing==0.35.1
regex==2024.5.15
requests @ file:///C:/b/abs_9c5n24p1y7/croot/requests_1716902867639/work
rpds-py==0.19.0
safetensors==0.4.3
sentencepiece==0.2.0
simple_parsing==0.1.5
six @ file:///tmp/build/80754af9/six_1644875935023/work
stack-data==0.6.3
sympy @ file:///C:/b/abs_82njkonm7f/croot/sympy_1701397685028/work
tbb==2021.13.0
termcolor==2.4.0
tiktoken==0.7.0
tokenizers==0.19.1
torch==2.3.1
torchaudio==2.3.1
torchvision==0.18.1
tornado==6.4
tqdm==4.66.4
traitlets==5.14.0
transformers==4.42.4
typing_extensions @ file:///C:/b/abs_0as9mdbkfl/croot/typing_extensions_1715268906610/work
urllib3 @ file:///C:/b/abs_a7hvzm4y95/croot/urllib3_1718912661242/work
win-inet-pton @ file:///C:/ci_311/win_inet_pton_1676425458225/work
xformers==0.0.27

Reproduction Steps

1. Download the model from Mistral-7B-Instruct-v0.3 on windows 11 system with python 3.11.9 with nvidia RTX 3090.
2. Create Conda environment
3. Run the following code in a python file with updated path to the model path:

   
   from mistral_inference.model import Transformer
   from mistral_inference.generate import generate

from mistral_common.tokens.tokenizers.mistral import MistralTokenizer from mistral_common.protocol.instruct.messages import UserMessage from mistral_common.protocol.instruct.request import ChatCompletionRequest

tokenizer = MistralTokenizer.from_file(f"{mistral_models_path}/tokenizer.model.v3") model = Transformer.from_folder(mistral_models_path)

completion_request = ChatCompletionRequest(messages=[UserMessage(content="Explain Machine Learning to me in a nutshell.")])

tokens = tokenizer.encode_chat_completion(completion_request).tokens

outtokens, = generate([tokens], model, max_tokens=64, temperature=0.0, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id) result = tokenizer.instruct_tokenizer.tokenizer.decode(out_tokens[0])

print(result)

Actual Result: 
A matching Triton is not available, some optimizations will not be enabled
Traceback (most recent call last):
  File "C:\Users\\-----\anaconda3\envs\mistral\Lib\site-packages\xformers\__init__.py", line 57, in _is_triton_available
    import triton  # noqa
    ^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'triton'
Traceback (most recent call last):
  File "C:\Users\\-----\source\repos\Mistral-7B-Instruct-v0.3\simplechat.py", line 1, in <module>
    from mistral_inference.model import Transformer
ImportError: cannot import name 'Transformer' from 'mistral_inference.model' (C:\Users\\-----\anaconda3\envs\mistral\Lib\site-packages\mistral_inference\model.py)

### Expected Behavior

Response from the model.

### Additional Context

_No response_

### Suggested Solutions

_No response_

[bhargavyagnik]

Trying to debug the ModuleNotFoundError for triton, while then, just wanted to suggest the import statement in line 1 is wrong. It should be : from mistral_inference.transformer import Transformer

Edit: Regarding ModuleNotfoundError, Windows does not have support for triton and thus you are getting this error, but xformers runs without it too just triton based optimizations will not take place.

[MaxAkbar]

@bhargavyagnik Thank you good sir that was it. Now I do get the error, but it still generates the response. I will go ahead and close this ticket.

python simplechat.py
A matching Triton is not available, some optimizations will not be enabled
Traceback (most recent call last):
  File "C:\Users\\****\anaconda3\envs\mistral\Lib\site-packages\xformers\__init__.py", line 57, in _is_triton_available
    import triton  # noqa
    ^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'triton'
Machine Learning (ML) is a subset of artificial intelligence (AI) that provides systems the ability to automatically learn and improve from experience without being explicitly programmed.

In simpler terms, Machine Learning is a way of teaching computers to learn by themselves, allowing them to make decisions or predictions based on data. This is

EDIT: By the way that script was from the huggingface for Mistralai model sheet.

[bhargavyagnik]

Yeah, I think the model sheet has yet to be updated. PR for the change has not been reviewed.