LocalAI for Embeddings

[netandreus]

How can I use LocalAI for Embeddings? There is no option for this at Embedding Preference window.

[tlandenberger]

Hi @netandreus I've created a draft pull request with a working solution for my use case #361. Would love to have some feedback if that also solves your needs.

[netandreus]

Hello! @tlandenberger ! Thank you for your efforts! I'm just tried this branch and got an error when build. Maybe you forget to commit image ../../../media/llmprovider/localai.png?

Here is an output:

=> ERROR [anything build-stage 2/2] RUN cd ./frontend/ && yarn build && yarn cache clean                                                                                              1.4s
------
 > [anything build-stage 2/2] RUN cd ./frontend/ && yarn build && yarn cache clean:
0.277 yarn run v1.22.19
0.290 $ vite build
0.595 vite v4.3.9 building for production...
0.614 transforming...
1.351 ✓ 72 modules transformed.
1.351 ✓ built in 756ms
1.351 Could not resolve "../../../media/llmprovider/localai.png" from "src/pages/GeneralSettings/EmbeddingPreference/index.jsx"
1.351 file: /app/frontend/src/pages/GeneralSettings/EmbeddingPreference/index.jsx
1.352 error during build:
1.352 RollupError: Could not resolve "../../../media/llmprovider/localai.png" from "src/pages/GeneralSettings/EmbeddingPreference/index.jsx"
1.352     at error (file:///app/frontend/node_modules/rollup/dist/es/shared/node-entry.js:2213:30)
1.352     at ModuleLoader.handleInvalidResolvedId (file:///app/frontend/node_modules/rollup/dist/es/shared/node-entry.js:24215:24)
1.352     at file:///app/frontend/node_modules/rollup/dist/es/shared/node-entry.js:24177:26
1.372 error Command failed with exit code 1.
1.372 info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
------
failed to solve: process "/bin/sh -c cd ./frontend/ && yarn build && yarn cache clean" did not complete successfully: exit code: 1

[tlandenberger]

Thanks, I've added the missing image ✅

[netandreus]

Just tried to build with this fix! Now it built successfull. But there are some new errors, during chatting through Anything. Seems like we need to define LLM model in the form near LocalAI address. Because now LocalAI try to search unexisted model:

Loading model in memory from file: /Users/andrey/sandbox/llm/current/models/model-placeholder

Anything screenshot

LocalAI logs

5:55PM DBG [whisper] Attempting to load
5:55PM DBG Loading model whisper from model-placeholder
5:55PM DBG Loading model in memory from file: /Users/andrey/sandbox/llm/current/models/model-placeholder
5:55PM DBG Loading GRPC Model whisper: {backendString:whisper model:model-placeholder threads:4 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0x14000313a00 externalBackends:map[] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false}
5:55PM DBG Loading GRPC Process: /tmp/localai/backend_data/backend-assets/grpc/whisper
5:55PM DBG GRPC Service for model-placeholder will be running at: '127.0.0.1:51095'
5:55PM DBG GRPC Service state dir: /var/folders/f9/1b1jz83s4ysfn9zfncbsb8y40000gn/T/go-processmanager1786773264
5:55PM DBG GRPC Service Started
rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:51095: connect: connection refused"
5:55PM DBG GRPC(model-placeholder-127.0.0.1:51095): stderr 2023/11/13 17:55:51 gRPC Server listening at 127.0.0.1:51095
5:55PM DBG GRPC Service Ready
5:55PM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:<nil>} sizeCache:0 unknownFields:[] Model:model-placeholder ContextSize:512 Seed:0 NBatch:512 F16Memory:false MLock:false MMap:false VocabOnly:false LowVRAM:false Embeddings:false NUMA:false NGPULayers:0 MainGPU: TensorSplit: Threads:4 LibrarySearchPath:/tmp/localai/backend_data/backend-assets/gpt4all RopeFreqBase:0 RopeFreqScale:0 RMSNormEps:0 NGQA:0 ModelFile:/Users/andrey/sandbox/llm/current/models/model-placeholder Device: UseTriton:false ModelBaseName: UseFastTokenizer:false PipelineType: SchedulerType: CUDA:false CFGScale:0 IMG2IMG:false CLIPModel: CLIPSubfolder: CLIPSkip:0 Tokenizer: LoraBase: LoraAdapter: NoMulMatQ:false DraftModel: AudioPath: Quantization:}
5:55PM DBG [whisper] Fails: could not load model: rpc error: code = Unknown desc = stat /Users/andrey/sandbox/llm/current/models/model-placeholder: no such file or directory
5:55PM DBG [stablediffusion] Attempting to load
5:55PM DBG Loading model stablediffusion from model-placeholder
5:55PM DBG Loading model in memory from file: /Users/andrey/sandbox/llm/current/models/model-placeholder
5:55PM DBG Loading GRPC Model stablediffusion: {backendString:stablediffusion model:model-placeholder threads:4 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0x14000313a00 externalBackends:map[] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false}
5:55PM DBG [stablediffusion] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/stablediffusion. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
5:55PM DBG [piper] Attempting to load
5:55PM DBG Loading model piper from model-placeholder
5:55PM DBG Loading model in memory from file: /Users/andrey/sandbox/llm/current/models/model-placeholder
5:55PM DBG Loading GRPC Model piper: {backendString:piper model:model-placeholder threads:4 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0x14000313a00 externalBackends:map[] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false}
5:55PM DBG [piper] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/piper. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
[127.0.0.1]:50930 500 - POST /v1/chat/completions

[tlandenberger]

Currently, I set the model via the .env file (included it in the .env.example file inside the server directory): EMBEDDING_MODEL_PREF='bert-embeddings' If you start with docker you need to add it to the .env file inside the docker directory.

I like the idea of selecting the model in the UI. We could implement an endpoint to fetch the installed models inside LocalAI with openai.listModels().

[netandreus]

I fill it with my model, and there is this error. Maybe somewhere in code model name is hardcoded to model-placeholder?

.env

SERVER_PORT=3001
CACHE_VECTORS="true"
VECTOR_DB="lancedb"
STORAGE_DIR="/app/server/storage"
UID='501'
GID='20'
EMBEDDING_BASE_PATH="http://host.docker.internal:8080/"
EMBEDDING_MODEL_PREF="text-embedding"

[tlandenberger]

I am not sure if this error is caused by the new embedding engine. The model-placeholder is something inside the LMStudio AiProvider so I assume your problem comes from a wrongly setup LMStudio connection. Does it work when using LMStudio as LLMProvider with another embedding engine?

[timothycarambat]

Resolved by https://github.com/Mintplex-Labs/anything-llm/pull/361