michaelfeil
commented
3 months ago Thanks for opening the issue. Did you really try to get nomic running?
I would not be concerned about the stacktrace of
infinity | NotImplementedError: The model type mistral is notIts just a info warning, that says that the optimum package already uses a better attention implementation for mistral, and no better one is available.
nomic
python3 -m venv venv
source ./venv/bin/activate
pip install infinity_emb[all]
pip install einops # einops is a package required just by the custom code of nomic.
infinity_emb --model-name-or-path nomic-ai/nomic-embed-text-v1.5(.venv) (base) michael@michael-laptop:~/infinity/libs/infinity_emb$ infinity_emb --model-name-or-path nomic-ai/nomic-embed-text-v1.5
INFO: Started server process [426215]
INFO: Waiting for application startup.
INFO 2024-03-30 09:31:45,673 infinity_emb INFO: model=`nomic-ai/nomic-embed-text-v1.5` selected, using engine=`torch` and device=`None` select_model.py:54
INFO 2024-03-30 09:31:46,118 sentence_transformers.SentenceTransformer INFO: Load pretrained SentenceTransformer: SentenceTransformer.py:107
nomic-ai/nomic-embed-text-v1.5
model.safetensors: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████| 547M/547M [00:25<00:00, 21.9MB/s]
WARNING 2024-03-30 09:32:14,036 modeling_hf_nomic_bert.py:357
transformers_modules.nomic-ai.nomic-embed-text-v1-unsupervised.3916676c856f1e25a4cc7a4e0ac740ea6ca9723a.modeling_hf_nomic_bert
WARNING: <All keys matched successfully>
tokenizer_config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 1.19k/1.19k [00:00<00:00, 8.19MB/s]
vocab.txt: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 232k/232k [00:00<00:00, 1.87MB/s]
tokenizer.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 711k/711k [00:00<00:00, 2.94MB/s]
special_tokens_map.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 695/695 [00:00<00:00, 5.14MB/s]
1_Pooling/config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 286/286 [00:00<00:00, 1.99MB/s]
INFO 2024-03-30 09:32:16,061 sentence_transformers.SentenceTransformer INFO: Use pytorch device_name: cuda SentenceTransformer.py:213
INFO 2024-03-30 09:32:16,502 infinity_emb INFO: Adding optimizations via Huggingface optimum. acceleration.py:17
ERROR 2024-03-30 09:32:16,503 infinity_emb ERROR: BetterTransformer is not available for model. The model type nomic_bert is not yet supported acceleration.py:21
to be used with BetterTransformer. Feel free to open an issue at https://github.com/huggingface/optimum/issues if you would like this
model type to be supported. Currently supported models are: dict_keys(['albert', 'bark', 'bart', 'bert', 'bert-generation', 'blenderbot',
'bloom', 'camembert', 'blip-2', 'clip', 'codegen', 'data2vec-text', 'deit', 'distilbert', 'electra', 'ernie', 'fsmt', 'gpt2', 'gptj',
'gpt_neo', 'gpt_neox', 'hubert', 'layoutlm', 'm2m_100', 'marian', 'markuplm', 'mbart', 'opt', 'pegasus', 'rembert', 'prophetnet',
'roberta', 'roc_bert', 'roformer', 'splinter', 'tapas', 't5', 'vilt', 'vit', 'vit_mae', 'vit_msn', 'wav2vec2', 'xlm-roberta', 'yolos'])..
Continue without bettertransformer modeling code.
Traceback (most recent call last):
File "/home/michael/infinity/libs/infinity_emb/infinity_emb/transformer/acceleration.py", line 19, in to_bettertransformer
model = BetterTransformer.transform(model)
File "/usr/lib/python3.10/contextlib.py", line 79, in inner
return func(*args, **kwds)
File "/home/michael/infinity/libs/infinity_emb/.venv/lib/python3.10/site-packages/optimum/bettertransformer/transformation.py", line
234, in transform
raise NotImplementedError(
NotImplementedError: The model type nomic_bert is not yet supported to be used with BetterTransformer. Feel free to open an issue at
https://github.com/huggingface/optimum/issues if you would like this model type to be supported. Currently supported models are:
dict_keys(['albert', 'bark', 'bart', 'bert', 'bert-generation', 'blenderbot', 'bloom', 'camembert', 'blip-2', 'clip', 'codegen',
'data2vec-text', 'deit', 'distilbert', 'electra', 'ernie', 'fsmt', 'gpt2', 'gptj', 'gpt_neo', 'gpt_neox', 'hubert', 'layoutlm',
'm2m_100', 'marian', 'markuplm', 'mbart', 'opt', 'pegasus', 'rembert', 'prophetnet', 'roberta', 'roc_bert', 'roformer', 'splinter',
'tapas', 't5', 'vilt', 'vit', 'vit_mae', 'vit_msn', 'wav2vec2', 'xlm-roberta', 'yolos']).
INFO 2024-03-30 09:32:16,510 infinity_emb INFO: Switching to half() precision (cuda: fp16). sentence_transformer.py:73
INFO 2024-03-30 09:32:17,047 infinity_emb INFO: Getting timings for batch_size=32 and avg tokens per sentence=1 select_model.py:77
5.65 ms tokenization
13.25 ms inference
0.26 ms post-processing
19.16 ms total
embeddings/sec: 1670.14
INFO 2024-03-30 09:32:18,570 infinity_emb INFO: Getting timings for batch_size=32 and avg tokens per sentence=512 select_model.py:83
14.14 ms tokenization
13.47 ms inference
726.95 ms post-processing
754.57 ms total
embeddings/sec: 42.41
INFO 2024-03-30 09:32:18,572 infinity_emb INFO: model warmed up, between 42.41-1670.14 embeddings/sec at batch_size=32 select_model.py:84
INFO 2024-03-30 09:32:18,574 infinity_emb INFO: creating batching engine batch_handler.py:392
INFO 2024-03-30 09:32:18,575 infinity_emb INFO: ready to batch requests. batch_handler.py:249
INFO 2024-03-30 09:32:18,577 infinity_emb INFO: infinity_server.py:64
♾️ Infinity - Embedding Inference Server
MIT License; Copyright (c) 2023 Michael Feil
Version 0.0.31
Open the Docs via Swagger UI:
http://0.0.0.0:7997/docs
Access model via 'GET':
curl http://0.0.0.0:7997/models
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:7997 (Press CTRL+C to quit)Mistral
@prasannakrish97 Can you try running the above commands and post it here?
prasannakrish97
Model description
You have mentioned that sfr-embedding model is supported along with all other huggingface embedding models (ref.nomic). However, both are not working : infinity | ERROR 2024-03-21 14:35:59,554 infinity_emb ERROR: acceleration.py:21 infinity | BetterTransformer is not available for model. The infinity | model type mistral is not yet supported to be used infinity | with BetterTransformer. Feel free to open an issue infinity | at https://github.com/huggingface/optimum/issues if infinity | you would like this model type to be supported. infinity | Currently supported models are: dict_keys(['albert', infinity | 'bark', 'bart', 'bert', 'bert-generation', infinity | 'blenderbot', 'bloom', 'camembert', 'blip-2', infinity | 'clip', 'codegen', 'data2vec-text', 'deit', infinity | 'distilbert', 'electra', 'ernie', 'fsmt', 'gpt2', infinity | 'gptj', 'gpt_neo', 'gpt_neox', 'hubert', 'layoutlm', infinity | 'm2m_100', 'marian', 'markuplm', 'mbart', 'opt', infinity | 'pegasus', 'rembert', 'prophetnet', 'roberta', infinity | 'roc_bert', 'roformer', 'splinter', 'tapas', 't5', infinity | 'vilt', 'vit', 'vit_mae', 'vit_msn', 'wav2vec2', infinity | 'xlm-roberta', 'yolos']).. Continue without infinity | bettertransformer modeling code. infinity | Traceback (most recent call last): infinity | File infinity | "/app/infinity_emb/transformer/acceleration.py", infinity | line 19, in to_bettertransformer infinity | model = BetterTransformer.transform(model) infinity | File "/usr/lib/python3.10/contextlib.py", line 79, infinity | in inner infinity | return func(*args, **kwds) infinity | File infinity | "/app/.venv/lib/python3.10/site-packages/optimum/bet infinity | tertransformer/transformation.py", line 234, in infinity | transform infinity | raise NotImplementedError( infinity | NotImplementedError: The model type mistral is not infinity | yet supported to be used with BetterTransformer.
Open source status
Provide useful links for the implementation
No response