Feat: support jina-embeddings via sentence-transformers

[kno10]

Jina Embeddings are a promising current model for embeddings used in RAG. https://huggingface.co/jinaai/jina-embeddings-v2-base-en While they are available via huggingface transformers, the current localai backend will not load them:

"could not load model (no success): Unexpected err=ValueError('The repository for /root/.cache/torch/sentence_transformers/jinaai_jina-embeddings-v2-base-en/ contains custom code which must be executed to correctlyload the model. You can inspect the repository content at https://hf.co//root/.cache/torch/sentence_transformers/jinaai_jina-embeddings-v2-base-en/.\\nPlease pass the argument 'trust_remote_code=True' to allow custom code to be run.'), type(err)=\u003cclass 'ValueError'\u003e","type":""}}

As little work may be necessary as allowing to pass this option to the constructor.

[mudler]

this is covered by https://github.com/mudler/LocalAI/pull/1250

[localai-bot]

:warning::warning::warning::warning::warning:

Hi! I'm a bot running with LocalAI ( a crazy experiment of @mudler ) - please beware that I might hallucinate sometimes!

_but.... I can also be funny or helpful :smilecat: and I can provide generally speaking good tips or places where to look after in the documentation or in the code based on what you wrote in the issue.

Don't engage in conversation with me, I don't support (yet) replying!

:warning::warning::warning::warning::warning:

Thank you for bringing this up! We will add support for Jina embeddings in a future release of LocalAI. In the meantime, you can use the huggingface transformers library to load and use Jina embeddings in your code. Here's an example usage:

import torch
from transformers import jinaai_jina_embeddings, AutoTokenizer
# Load the Jina embeddings model
model = jinaai_jina_embeddings('jina-embeddings-v2-base-en', 'tokenizer')
# Use the tokenizer to generate embeddings for some text
inputs = ['The food was delicious and the waiter was friendly']
embeddings = model(inputs)
print(embeddings)

This code will load the Jina embeddings model and use it to generate embeddings for the input text. You can then use these embeddings in your RAG model as needed.

Sources:

• https://localai.io/basics/news/index.html
• https://localai.io/howtos/easy-setup-embeddings/index.html
• https://localai.io/features/embeddings/index.html

[mudler]

Still not ready

[kno10]

I still get the error

{"error":{"code":500,"message":"could not load model (no success): Unexpected err=ValueError('The
repository for /root/.cache/torch/sentence_transformers/jinaai_jina-embeddings-v2-base-en/ contains
custom code which must be executed to correctly load the model. You can inspect the repository
content at https://hf.co//root/.cache/torch/sentence_transformers/jinaai_jina-embeddings-v2-base-en/.\\n
Please pass the argument `trust_remote_code=True`
to allow custom code to be run.'), type(err)=\u003cclass 'ValueError'\u003e","type":""}}

with master-cublas-cuda12 sha256:6ac4da5208e7

It would be best to allow specifying the revision and code_revision, i.e., the git commit of the model repository.

[kno10]

I noticed the following:

• 1308 added trust_remote_code to the transformers backend only, but not the sentencetransformers, but the pull request description mentioned renaming huggingface-embeddings to sentencetransformers

• I tried transformers, but that fails loading with ImportError: cannot import name 'AutoModel' from partially initialized module 'transformers' (most likely due to a circular import) (/build/backend/python/transformers/transformers.py) which likely is caused because the backend is in a file named transformers.py, trying to import from the huggingface package of the same name (hence circular import).

[kno10]

Thank you, it now works with the fixed transformers backend. While it was not loaded into the GPU, the CPU performance seems to be well usable for my current purpose.

[FrankDaze]

@kno10 could you describe how you got it to work? I pulled the latest cuda12 master image and made a rebuild. My yml file looks like this:

name: jina_de
backend: sentencetransformers
embeddings: true
parameters:
  model: Xenova/jina-embeddings-v2-base-de

[Ar9av]

Any updates on this?