search

mudler / LocalAI

:robot: The free, Open Source OpenAI alternative. Self-hosted, community-driven and local-first. Drop-in replacement for OpenAI running on consumer-grade hardware. No GPU required. Runs gguf, transformers, diffusers and many more models architectures. It allows to generate Text, Audio, Video, Images. Also with voice cloning capabilities.
https://localai.io
MIT License
21.75k stars 1.66k forks source link

Basic Bert embedding (from example) not working #1598

Open apstrom opened 5 months ago

apstrom commented 5 months ago

LocalAI version: Latest.

Environment, CPU architecture, OS, and Version: 6.5.0-14-generic #14~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC x86_64 x86_64 x86_64 GNU/Linux

Describe the bug LocalAI does not run the bert embedding (either text-ada or bert-embeddings). LocalAI documents do not detail how to install Huggingface embeddings, which apparently require some sort of conversion.

To Reproduce LOCALAI=http://localhost:8080 curl $LOCALAI/models/apply -H "Content-Type: application/json" -d '{ "id": "model-gallery@bert-embeddings" }'

Run a task.

Expected behavior The embedding model runs and returns a result.

Logs localai-api-1 | 10:01PM INF Loading model 'bert-MiniLM-L6-v2q4_0.bin' with backend bert-embeddings localai-api-1 | 10:01PM DBG Model already loaded in memory: bert-MiniLM-L6-v2q4_0.bin localai-api-1 | 10:01PM WRN GRPC Model not responding: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:39703: connect: connection refused" localai-api-1 | 10:01PM WRN Deleting the process in order to recreate it localai-api-1 | 10:01PM DBG GRPC Process is not responding: bert-MiniLM-L6-v2q4_0.bin localai-api-1 | 10:01PM DBG Loading model in memory from file: /models/bert-MiniLM-L6-v2q4_0.bin localai-api-1 | 10:01PM DBG Loading Model bert-MiniLM-L6-v2q4_0.bin with gRPC (file: /models/bert-MiniLM-L6-v2q4_0.bin) (backend: bert-embeddings): {backendString:bert-embeddings model:bert-MiniLM-L6-v2q4_0.bin threads:10 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0xc00026b0e0 externalBackends:map[autogptq:/build/backend/python/autogptq/run.sh bark:/build/backend/python/bark/run.sh coqui:/build/backend/python/coqui/run.sh diffusers:/build/backend/python/diffusers/run.sh exllama:/build/backend/python/exllama/run.sh exllama2:/build/backend/python/exllama2/run.sh huggingface-embeddings:/build/backend/python/sentencetransformers/run.sh petals:/build/backend/python/petals/run.sh sentencetransformers:/build/backend/python/sentencetransformers/run.sh transformers:/build/backend/python/transformers/run.sh transformers-musicgen:/build/backend/python/transformers-musicgen/run.sh vall-e-x:/build/backend/python/vall-e-x/run.sh vllm:/build/backend/python/vllm/run.sh] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false parallelRequests:false} localai-api-1 | 10:01PM DBG Loading GRPC Process: /tmp/localai/backend_data/backend-assets/grpc/bert-embeddings localai-api-1 | 10:01PM DBG GRPC Service for bert-MiniLM-L6-v2q4_0.bin will be running at: '127.0.0.1:45999' localai-api-1 | 10:01PM DBG GRPC Service state dir: /tmp/go-processmanager2902677880 localai-api-1 | 10:01PM DBG GRPC Service Started localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr 2024/01/16 22:01:44 gRPC Server listening at 127.0.0.1:45999 localai-api-1 | 10:01PM DBG GRPC Service Ready localai-api-1 | 10:01PM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:} sizeCache:0 unknownFields:[] Model:bert-MiniLM-L6-v2q4_0.bin ContextSize:0 Seed:0 NBatch:512 F16Memory:false MLock:false MMap:false VocabOnly:false LowVRAM:false Embeddings:true NUMA:false NGPULayers:0 MainGPU: TensorSplit: Threads:10 LibrarySearchPath: RopeFreqBase:0 RopeFreqScale:0 RMSNormEps:0 NGQA:0 ModelFile:/models/bert-MiniLM-L6-v2q4_0.bin Device: UseTriton:false ModelBaseName: UseFastTokenizer:false PipelineType: SchedulerType: CUDA:false CFGScale:0 IMG2IMG:false CLIPModel: CLIPSubfolder: CLIPSkip:0 ControlNet: Tokenizer: LoraBase: LoraAdapter: LoraScale:0 NoMulMatQ:false DraftModel: AudioPath: Quantization: MMProj: RopeScaling: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0} localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stdout bert_load_from_file: loading model from '/models/bert-MiniLM-L6-v2q4_0.bin' - please wait ... localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stdout bert_load_from_file: n_vocab = 30522 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stdout bert_load_from_file: n_max_tokens = 512 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stdout bert_load_from_file: n_embd = 384 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stdout bert_load_from_file: n_intermediate = 1536 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stdout bert_load_from_file: n_head = 12 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stdout bert_load_from_file: n_layer = 6 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stdout bert_load_from_file: f16 = 2 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stdout bert_load_from_file: ggml ctx size = 13.57 MB localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr SIGSEGV: segmentation violation localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr PC=0x8714ca m=7 sigcode=1 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr signal arrived during cgo execution localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr goroutine 56 [syscall]: localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr runtime.cgocall(0x83d390, 0xc0002957e8) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/cgocall.go:157 +0x4b fp=0xc0002957c0 sp=0xc000295788 pc=0x40c90b localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr github.com/go-skynet/go-bert%2ecpp._Cfunc_bert_embeddings(0x7ff76c000ca0, 0x7ff76c000e00, 0xc000400000) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr _cgo_gotypes.go:128 +0x4b fp=0xc0002957e8 sp=0xc0002957c0 pc=0x4f5c6b localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr github.com/go-skynet/go-bert%2ecpp.(Bert).Embeddings.func1(0x7ff75c000b60?, 0xa?, 0x7ff76c000ca0?) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /build/sources/go-bert/gobert.go:38 +0x6a fp=0xc000295830 sp=0xc0002957e8 pc=0x4f63aa localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr github.com/go-skynet/go-bert%2ecpp.(Bert).Embeddings(0x104e?, {0xc000303300, 0x1043}, {0xc000295918, 0x1, 0x8e82e0?}) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /build/sources/go-bert/gobert.go:38 +0x105 fp=0xc000295898 sp=0xc000295830 pc=0x4f61e5 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr main.(Embeddings).Embeddings(0xe6aaa0?, 0x41253a?) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /build/backend/go/llm/bert/bert.go:33 +0xb3 fp=0xc000295930 sp=0xc000295898 pc=0x83c9f3 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr github.com/go-skynet/LocalAI/pkg/grpc.(server).Embedding(0xc000034ec0, {0xc000308000?, 0x4f86a6?}, 0x0?) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /build/pkg/grpc/server.go:37 +0xc6 fp=0xc0002959b0 sp=0xc000295930 pc=0x83ab26 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr github.com/go-skynet/LocalAI/pkg/grpc/proto._Backend_Embedding_Handler({0x94f120?, 0xc000034ec0}, {0xa3bdd0, 0xc000282b40}, 0xc0002c2300, 0x0) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /build/pkg/grpc/proto/backend_grpc.pb.go:303 +0x169 fp=0xc000295a08 sp=0xc0002959b0 pc=0x82e849 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr google.golang.org/grpc.(Server).processUnaryRPC(0xc0001d81e0, {0xa3bdd0, 0xc0001b2810}, {0xa3f358, 0xc0000f1520}, 0xc000246120, 0xc0001daea0, 0xd4f568, 0x0) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:1343 +0xe03 fp=0xc000295df0 sp=0xc000295a08 pc=0x817383 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr google.golang.org/grpc.(Server).handleStream(0xc0001d81e0, {0xa3f358, 0xc0000f1520}, 0xc000246120) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:1737 +0xc4c fp=0xc000295f78 sp=0xc000295df0 pc=0x81c2ec localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr google.golang.org/grpc.(Server).serveStreams.func1.1() localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:986 +0x86 fp=0xc000295fe0 sp=0xc000295f78 pc=0x815286 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr runtime.goexit() localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000295fe8 sp=0xc000295fe0 pc=0x46fce1 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr created by google.golang.org/grpc.(Server).serveStreams.func1 in goroutine 9 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:997 +0x145 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr goroutine 1 [IO wait]: localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr runtime.gopark(0x482ad0?, 0xc0001c9b28?, 0x78?, 0x9b?, 0x4dc3bd?) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0001c9b08 sp=0xc0001c9ae8 pc=0x4410ae localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr runtime.netpollblock(0x46dd52?, 0x40c0a6?, 0x0?) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/netpoll.go:564 +0xf7 fp=0xc0001c9b40 sp=0xc0001c9b08 pc=0x439b57 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr internal/poll.runtime_pollWait(0x7ff7798326e0, 0x72) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/netpoll.go:343 +0x85 fp=0xc0001c9b60 sp=0xc0001c9b40 pc=0x46ac05 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr internal/poll.(pollDesc).wait(0xc000036800?, 0x4?, 0x0) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0001c9b88 sp=0xc0001c9b60 pc=0x4d5027 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr internal/poll.(pollDesc).waitRead(...) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr internal/poll.(FD).Accept(0xc000036800) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/internal/poll/fd_unix.go:611 +0x2ac fp=0xc0001c9c30 sp=0xc0001c9b88 pc=0x4da50c localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr net.(netFD).accept(0xc000036800) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/net/fd_unix.go:172 +0x29 fp=0xc0001c9ce8 sp=0xc0001c9c30 pc=0x5a0ec9 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr net.(TCPListener).accept(0xc00007e4c0) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/net/tcpsock_posix.go:152 +0x1e fp=0xc0001c9d10 sp=0xc0001c9ce8 pc=0x5b805e localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr net.(TCPListener).Accept(0xc00007e4c0) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/net/tcpsock.go:315 +0x30 fp=0xc0001c9d40 sp=0xc0001c9d10 pc=0x5b7210 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr google.golang.org/grpc.(Server).Serve(0xc0001d81e0, {0xa3b3b8?, 0xc00007e4c0}) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:852 +0x462 fp=0xc0001c9e80 sp=0xc0001c9d40 pc=0x813ee2 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr github.com/go-skynet/LocalAI/pkg/grpc.StartServer({0x7fff778f3a27?, 0xc000024160?}, {0xa3fa80?, 0xc000034df0}) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /build/pkg/grpc/server.go:178 +0x17d fp=0xc0001c9f10 sp=0xc0001c9e80 pc=0x83c79d localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr main.main() localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /build/backend/go/llm/bert/main.go:18 +0x85 fp=0xc0001c9f40 sp=0xc0001c9f10 pc=0x83cbc5 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr runtime.main() localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/proc.go:267 +0x2bb fp=0xc0001c9fe0 sp=0xc0001c9f40 pc=0x440c5b localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr runtime.goexit() localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0001c9fe8 sp=0xc0001c9fe0 pc=0x46fce1 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr goroutine 2 [force gc (idle)]: localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000066fa8 sp=0xc000066f88 pc=0x4410ae localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr runtime.goparkunlock(...) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/proc.go:404 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr runtime.forcegchelper() localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/proc.go:322 +0xb3 fp=0xc000066fe0 sp=0xc000066fa8 pc=0x440f33 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr runtime.goexit() localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000066fe8 sp=0xc000066fe0 pc=0x46fce1 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr created by runtime.init.6 in goroutine 1 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/proc.go:310 +0x1a localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr goroutine 3 [GC sweep wait]: localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000067778 sp=0xc000067758 pc=0x4410ae localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr runtime.goparkunlock(...) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/proc.go:404 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr runtime.bgsweep(0x0?) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/mgcsweep.go:280 +0x94 fp=0xc0000677c8 sp=0xc000067778 pc=0x42cfd4 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr runtime.gcenable.func1() localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/mgc.go:200 +0x25 fp=0xc0000677e0 sp=0xc0000677c8 pc=0x422185 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr runtime.goexit() localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000677e8 sp=0xc0000677e0 pc=0x46fce1 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr created by runtime.gcenable in goroutine 1 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/mgc.go:200 +0x66 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr goroutine 4 [GC scavenge wait]: localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr runtime.gopark(0xc000090000?, 0xa340f8?, 0x1?, 0x0?, 0xc000007380?) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000067f70 sp=0xc000067f50 pc=0x4410ae localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr runtime.goparkunlock(...) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/proc.go:404 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr runtime.(scavengerState).park(0xd9a5a0) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000067fa0 sp=0xc000067f70 pc=0x42a8a9 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr runtime.bgscavenge(0x0?) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc000067fc8 sp=0xc000067fa0 pc=0x42ae3c localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr runtime.gcenable.func2() localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/mgc.go:201 +0x25 fp=0xc000067fe0 sp=0xc000067fc8 pc=0x422125 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr runtime.goexit() localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000067fe8 sp=0xc000067fe0 pc=0x46fce1 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr created by runtime.gcenable in goroutine 1 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/mgc.go:201 +0xa5 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr goroutine 5 [finalizer wait]: localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr runtime.gopark(0x198?, 0x97b6a0?, 0x1?, 0x22?, 0x0?) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000066620 sp=0xc000066600 pc=0x4410ae localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr runtime.runfinq() localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc0000667e0 sp=0xc000066620 pc=0x4211a7 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr runtime.goexit() localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000667e8 sp=0xc0000667e0 pc=0x46fce1 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr created by runtime.createfing in goroutine 1 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/mfinal.go:163 +0x3d localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr goroutine 7 [select]: localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr runtime.gopark(0xc000251f00?, 0x2?, 0x1e?, 0x0?, 0xc000251ed4?) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000251d80 sp=0xc000251d60 pc=0x4410ae localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr runtime.selectgo(0xc000251f00, 0xc000251ed0, 0x7ae656?, 0x0, 0xc0003a4000?, 0x1) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000251ea0 sp=0xc000251d80 pc=0x450b25 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr google.golang.org/grpc/internal/transport.(controlBuffer).get(0xc00038c820, 0x1) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /go/pkg/mod/google.golang.org/grpc@v1.59.0/internal/transport/controlbuf.go:418 +0x113 fp=0xc000251f30 sp=0xc000251ea0 pc=0x78d4b3 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr google.golang.org/grpc/internal/transport.(loopyWriter).run(0xc000200150) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /go/pkg/mod/google.golang.org/grpc@v1.59.0/internal/transport/controlbuf.go:552 +0x86 fp=0xc000251f90 sp=0xc000251f30 pc=0x78dbc6 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr google.golang.org/grpc/internal/transport.NewServerTransport.func2() localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /go/pkg/mod/google.golang.org/grpc@v1.59.0/internal/transport/http2_server.go:336 +0xd5 fp=0xc000251fe0 sp=0xc000251f90 pc=0x7a4415 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr runtime.goexit() localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000251fe8 sp=0xc000251fe0 pc=0x46fce1 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 55 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /go/pkg/mod/google.golang.org/grpc@v1.59.0/internal/transport/http2_server.go:333 +0x1acc localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr goroutine 8 [select]: localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr runtime.gopark(0xc0002b8770?, 0x4?, 0x5?, 0x0?, 0xc0002b86c0?) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0002b8528 sp=0xc0002b8508 pc=0x4410ae localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr runtime.selectgo(0xc0002b8770, 0xc0002b86b8, 0x0?, 0x0, 0x0?, 0x1) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc0002b8648 sp=0xc0002b8528 pc=0x450b25 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr google.golang.org/grpc/internal/transport.(http2Server).keepalive(0xc0000f1520) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /go/pkg/mod/google.golang.org/grpc@v1.59.0/internal/transport/http2_server.go:1152 +0x225 fp=0xc0002b87c8 sp=0xc0002b8648 pc=0x7ab6c5 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr google.golang.org/grpc/internal/transport.NewServerTransport.func4() localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /go/pkg/mod/google.golang.org/grpc@v1.59.0/internal/transport/http2_server.go:339 +0x25 fp=0xc0002b87e0 sp=0xc0002b87c8 pc=0x7a4305 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr runtime.goexit() localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0002b87e8 sp=0xc0002b87e0 pc=0x46fce1 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 55 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /go/pkg/mod/google.golang.org/grpc@v1.59.0/internal/transport/http2_server.go:339 +0x1b0e localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr goroutine 9 [IO wait]: localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr runtime.gopark(0x100000000?, 0xb?, 0x0?, 0x0?, 0x6?) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000218aa0 sp=0xc000218a80 pc=0x4410ae localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr runtime.netpollblock(0x480d58?, 0x40c0a6?, 0x0?) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/netpoll.go:564 +0xf7 fp=0xc000218ad8 sp=0xc000218aa0 pc=0x439b57 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr internal/poll.runtime_pollWait(0x7ff7798325e8, 0x72) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/netpoll.go:343 +0x85 fp=0xc000218af8 sp=0xc000218ad8 pc=0x46ac05 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr internal/poll.(pollDesc).wait(0xc0001b8100?, 0xc00039c000?, 0x0) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc000218b20 sp=0xc000218af8 pc=0x4d5027 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr internal/poll.(pollDesc).waitRead(...) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr internal/poll.(FD).Read(0xc0001b8100, {0xc00039c000, 0x8000, 0x8000}) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a fp=0xc000218bb8 sp=0xc000218b20 pc=0x4d631a localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr net.(netFD).Read(0xc0001b8100, {0xc00039c000?, 0x1060100000000?, 0x8?}) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/net/fd_posix.go:55 +0x25 fp=0xc000218c00 sp=0xc000218bb8 pc=0x59eea5 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr net.(conn).Read(0xc000210018, {0xc00039c000?, 0x0?, 0xc000218cd0?}) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/net/net.go:179 +0x45 fp=0xc000218c48 sp=0xc000218c00 pc=0x5af785 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr net.(TCPConn).Read(0x0?, {0xc00039c000?, 0xc000218ca0?, 0x45efcd?}) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr :1 +0x25 fp=0xc000218c78 sp=0xc000218c48 pc=0x5c1f25 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr bufio.(Reader).Read(0xc000095020, {0xc000388200, 0x9, 0xc161dd12829bc7a6?}) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/bufio/bufio.go:244 +0x197 fp=0xc000218cb0 sp=0xc000218c78 pc=0x5e53b7 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr io.ReadAtLeast({0xa38aa0, 0xc000095020}, {0xc000388200, 0x9, 0x9}, 0x9) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/io/io.go:335 +0x90 fp=0xc000218cf8 sp=0xc000218cb0 pc=0x4badb0 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr io.ReadFull(...) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/io/io.go:354 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr golang.org/x/net/http2.readFrameHeader({0xc000388200, 0x9, 0xc00020e090?}, {0xa38aa0?, 0xc000095020?}) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /go/pkg/mod/golang.org/x/net@v0.17.0/http2/frame.go:237 +0x65 fp=0xc000218d48 sp=0xc000218cf8 pc=0x779f25 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr golang.org/x/net/http2.(Framer).ReadFrame(0xc0003881c0) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /go/pkg/mod/golang.org/x/net@v0.17.0/http2/frame.go:498 +0x85 fp=0xc000218df0 sp=0xc000218d48 pc=0x77a665 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr google.golang.org/grpc/internal/transport.(http2Server).HandleStreams(0xc0000f1520, 0x1?) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /go/pkg/mod/google.golang.org/grpc@v1.59.0/internal/transport/http2_server.go:636 +0x145 fp=0xc000218f00 sp=0xc000218df0 pc=0x7a7565 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr google.golang.org/grpc.(Server).serveStreams(0xc0001d81e0, {0xa3f358?, 0xc0000f1520}) localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:979 +0x1c2 fp=0xc000218f80 sp=0xc000218f00 pc=0x815022 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr google.golang.org/grpc.(Server).handleRawConn.func1() localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:920 +0x45 fp=0xc000218fe0 sp=0xc000218f80 pc=0x814885 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr runtime.goexit() localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000218fe8 sp=0xc000218fe0 pc=0x46fce1 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 55 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr /go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:919 +0x185 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr rax 0x0 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr rbx 0x7ff75bbef970 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr rcx 0x0 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr rdx 0x7ff77982b700 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr rdi 0x7ff779812610 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr rsi 0x0 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr rbp 0x7ff779812bf0 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr rsp 0x7ff779812ba0 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr r8 0x0 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr r9 0x7ff779812a3f localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr r10 0xf8a0340 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr r11 0x0 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr r12 0x7ff75b9af4c0 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr r13 0x7ff75b6af010 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr r14 0xdc92e8 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr r15 0x0 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr rip 0x8714ca localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr rflags 0x10246 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr cs 0x33 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr fs 0x0 localai-api-1 | 10:01PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:45999): stderr gs 0x0 localai-api-1 | [192.168.134.64]:39418 500 - POST /v1/embeddings

Additional context This issue has arisen before. Essentially, without being able to run embeddings, my use case for LocalAI is nullified. LocalAI is great; I really want to be able to use it freely--the embeddings have to work, and better documentation is required to allow neophytes (like myself) to properly implement things like embeddings.

To be clear: models work fine. The embeddings are not working. The only error in the above logs is:

localai-api-1 | 10:01PM INF Loading model 'bert-MiniLM-L6-v2q4_0.bin' with backend bert-embeddings localai-api-1 | 10:01PM DBG Model already loaded in memory: bert-MiniLM-L6-v2q4_0.bin localai-api-1 | 10:01PM WRN GRPC Model not responding: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:39703: connect: connection refused" localai-api-1 | 10:01PM WRN Deleting the process in order to recreate it localai-api-1 | 10:01PM DBG GRPC Process is not responding: bert-MiniLM-L6-v2q4_0.bin

lunamidori5 commented 5 months ago

@apstrom yes the site is being worked on by Mudler and is unable to be updated at this time, for now our how tos are being updated on my site. Thank you <3 Please check out our updated how tos - https://io.midori-ai.xyz/howtos/ Installing a embedding model - https://io.midori-ai.xyz/howtos/easy-setup-embeddings/ Installing a LLM, SD, Embedding, TTS, and other starting models - https://io.midori-ai.xyz/howtos/easy-model-installer/

apstrom commented 5 months ago

@lunamidori5 I am happy to hear about the updated documentation. That's exciting news.

I've got models installed; the issue that I am having is with the embedding model, and any custom embedding models that I would like to use. Only one of these issues is related to LocalAI -- the other issue is that many Huggingface pytorch embedding models do not seem to work natively with LocalAI. This issue is a conversion problem that I am trying to solve. I would like to run law-specific embedding models for my applications.

The issue that I detailed above with logs is that the bert-embedding model, which is installed and which does run on LocalAI, crashes. The error looks like a gRPC error. A similar error was reported in #1198.

lunamidori5 commented 5 months ago

@apstrom did you reinstall using the model installer or the updated documentation on how to install it? The previous documentation was completely wrong. And I didn't catch it in time I do apologize for that that is my fault

apstrom commented 5 months ago

@lunamidori5 The reinstalled model responds to basic (short) texts; it does not respond to long texts. I get the above error again. Based on the behaviour, I am guessing that I need to set a context size for the model to handle longer texts.

I am looking at your new documentation. The presentation is clearer. I will provide you with some feedback on the other thread.

lunamidori5 commented 5 months ago

@apstrom feedback is always welcome to. And sometimes I'm not sure where the issues lay. Until they are reported. I did notice that somebody mentioned API keys. I think I'm going to make a how to just like explaining each variable and stuff. I'm not sure. Is that something you're interested in?

mudler commented 5 months ago

@apstrom did you tried with one of the examples in the quickstart? this works for me quite well:

docker run -ti -p 8080:8080 localai/localai:v2.5.1-ffmpeg all-minilm-l6-v2

You can also set that with an environment variable, for example:

MODELS=all-minilm-l6-v2
tnpb-encora commented 2 months ago

@apstrom feedback is always welcome to. And sometimes I'm not sure where the issues lay. Until they are reported. I did notice that somebody mentioned API keys. I think I'm going to make a how to just like explaining each variable and stuff. I'm not sure. Is that something you're interested in?

I think explaining each variable is something worth doing! Also, I'm having the same issue mentioned by @apstrom, the models do not seem to work well with longer queries and when i call the embeddings api (localhost:p/embeddings) it always returns a rpc error...

womd commented 2 months ago

had the same errors when i ran embeddings with flowise for a pdf-answering-bot - setting "Chunk-Size" to 500 solved it for me. embeddingsx

tnpb-encora commented 2 months ago

@apstrom did you tried with one of the examples in the quickstart? this works for me quite well:

docker run -ti -p 8080:8080 localai/localai:v2.5.1-ffmpeg all-minilm-l6-v2

You can also set that with an environment variable, for example:

MODELS=all-minilm-l6-v2

Mudler, this models also works for me but when I try the text-ada-002 that comes with the aio-latest-cpu or when I try another embeddings models they do not work. Do you have any idea of how i investigate this better?

I think that I may be making wrong configs for the models.