dosubot[bot]
commented
1 month ago To set up a locally served embedding model using LiteLLMEmbeddingModel, you can configure it similarly to how you set up the main LLM model. Although the from_name method isn't recognized, you can directly specify the configuration using the Settings class.
Here's an example of how you might configure it:
from paperqa import Settings, ask
local_embedding_config = dict(
model_list=[
dict(
model_name="my_embedding_model",
litellm_params=dict(
model="mxbai-embed-large-v1-f16",
api_base="http://localhost:8081/v1",
api_key="sk-no-key-required",
# Add any other necessary parameters here
),
)
]
)
answer = ask(
"What manufacturing challenges are unique to bispecific antibodies?",
settings=Settings(
embedding="my_embedding_model",
embedding_config=local_embedding_config,
),
)This setup specifies the IP address and port for your local embedding model. Make sure to replace "my_embedding_model" and other parameters with your actual model details. This approach should help you connect to your locally served embedding model [1][2].
To continue talking to Dosu, mention @dosu.
Snikch63200
medatasci
Hello,
Is there a way to create an embedding model object (with
LiteLLMEmbeddingModel()i guess) from a locally served embedding model.For more precision I run in parallel :
I ever asked how setting up a local embedding model (https://github.com/Future-House/paper-qa/issues/557) but given solution does not seem to have any entry for IP adress and port for embedding model as specified for mail LLM model.
This solution doesn't work :
LiteLLMEmbeddingModel.from_name("my-llm-model")because from_name method is not recognized...Is it a clear and simple way to set up a locally served embedding model ?