search

mudler / LocalAI

:robot: The free, Open Source alternative to OpenAI, Claude and others. Self-hosted and local-first. Drop-in replacement for OpenAI, running on consumer-grade hardware. No GPU required. Runs gguf, transformers, diffusers and many more models architectures. Features: Generate Text, Audio, Video, Images, Voice Cloning, Distributed, P2P inference
https://localai.io
MIT License
25.88k stars 1.94k forks source link

Segfault on long embeddings #1463

Closed lenaxia closed 11 months ago

lenaxia commented 11 months ago

LocalAI version: v2.0.0 10:23PM INF LocalAI version: v2.0.0 (238fec244ae6c9a66bc7fafd76c7e14671110a6f)

Environment, CPU architecture, OS, and Version:

Linux localai-64b4946689-mcxkk 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 GNU/Linux
CPU info:
model name      : Intel(R) Core(TM) i7-6700T CPU @ 2.80GHz
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d arch_capabilities
CPU:    AVX    found OK
CPU:    AVX2   found OK
CPU: no AVX512 found

Describe the bug Any embedding request with long text generates a segfault

To Reproduce

curl $LOCALAI/models/apply -H "Content-Type: application/json" -d '{
     "id": "model-gallery@bert-embeddings"
   }'  
curl $LOCALAI/v1/embeddings -X POST -H "Content-Type: application/json" -d '{
  "input": "ANY RANDOMLY LONG TEXT",                                                                                                                                          "model": "bert-embeddings"                                                                                                                                                    }' | jq "."

Expected behavior Return an embedding vector

Logs

10:27PM DBG Request received:
10:27PM DBG Parameter Config: &{PredictionOptions:{Model:bert-MiniLM-L6-v2q4_0.bin Language: N:0 TopP:0 TopK:0 Temperature:0 Maxtokens:0 Echo:false Batch:0 F16:false IgnoreEOS:false RepeatPenalty:0 Keep:0 MirostatETA:0 MirostatTAU:0 Mirostat:0 FrequencyPenalty:0 TFZ:0 TypicalP:0 Seed:0 NegativePrompt: RopeFreqBase:0 RopeFreqScale:0 NegativePromptScale:0 UseFastTokenizer:false ClipSkip:0 Tokenizer:} Name:bert-embeddings F16:false Threads:4 Debug:true Roles:map[] Embeddings:true Backend:bert-embeddings TemplateConfig:{Chat: ChatMessage: Completion: Edit: Functions:} PromptStrings:[] InputStrings:[H o w   t o   w i n   i n   a   J / 8 0
R e f e r e n c e :
h t t p s : / / w e b . a r c h i v e . o r g / w e b / 2 0 1 4 0 6 2 6 1 5 0 1 1 4 / h t t p s : / / s h o r e s a i l s . c o m / o n e d e s i g n / j 8 0 g u i
d e . h t m
B a l e a r i a ,   M a l l o r c a   S o t h e b y ' s   W o m e n ' s   C u p   o n   d a y   2   o f   t h e   4 0 t h   C o p a   d e l   R e y   M A P F R E
S h o r e   S a i l s   T u n i n g   G u i d e   w a s   w r i t t e n   t o   h e l p   y o u   g e t   t h e   b e s t   p e r f o r m a n c e   f r o m   y o u r   J
8 0 .   S h o r e   S a i l s   h a s   b e e n   s a i l i n g   J   8 0 ' s   s i n c e   t h e   b o a t   w a s   f i r s t   i n t r o d u c e d .   O u r
c o m m i t m e n t   t o   t h e   c l a s s ,   a n d   t o   i t s   s a i l o r s ,   s e t s   u s   a p a r t   f r o m   a n y   o t h e r   c o m p a n y   i n   t h e
...
LOTS MORE TEXT HERE

] InputToken:[] functionCallString: functionCallNameString: FunctionsConfig:{DisableNoAction:false NoActionFunctionName: NoActionDescriptionName:} FeatureFlag:map[] LLMConfig:{SystemPrompt: TensorSplit: MainGPU: RMSNormEps:0 NGQA:0 PromptCachePath: PromptCacheAll:false PromptCacheRO:false MirostatETA:0 MirostatTAU:0 Mirostat:0 NGPULayers:0 MMap:false MMlock:false LowVRAM:false Grammar: StopWords:[] Cutstrings:[] TrimSpace:[] ContextSize:0 NUMA:false LoraAdapter: LoraBase: LoraScale:0 NoMulMatQ:false DraftModel: NDraft:0 Quantization: MMProj: RopeScaling: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0} AutoGPTQ:{ModelBaseName: Device: Triton:false UseFastTokenizer:false} Diffusers:{PipelineType: SchedulerType: CUDA:false EnableParameters: CFGScale:0 IMG2IMG:false ClipSkip:0 ClipModel: ClipSubFolder:} Step:0 GRPC:{Attempts:0 AttemptsSleepTime:0} VallE:{AudioPath:}}
10:27PM DBG Loading model bert-embeddings from bert-MiniLM-L6-v2q4_0.bin
10:27PM DBG Loading model in memory from file: /models/bert-MiniLM-L6-v2q4_0.bin
10:27PM DBG Loading Model bert-MiniLM-L6-v2q4_0.bin with gRPC (file: /models/bert-MiniLM-L6-v2q4_0.bin) (backend: bert-embeddings): {backendString:bert-embeddings model:bert-MiniLM-L6-v2q4_0.bin threads:4 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0xc000240960 externalBackends:map[autogptq:/build/backend/python/autogptq/run.sh bark:/build/backend/python/bark/run.sh diffusers:/build/backend/python/diffusers/run.sh exllama:/build/backend/python/exllama/run.sh huggingface-embeddings:/build/backend/python/sentencetransformers/run.sh petals:/build/backend/python/petals/run.sh sentencetransformers:/build/backend/python/sentencetransformers/run.sh transformers:/build/backend/python/transformers/run.sh vall-e-x:/build/backend/python/vall-e-x/run.sh vllm:/build/backend/python/vllm/run.sh] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false parallelRequests:false}
10:27PM DBG Loading GRPC Process: /tmp/localai/backend_data/backend-assets/grpc/bert-embeddings
10:27PM DBG GRPC Service for bert-MiniLM-L6-v2q4_0.bin will be running at: '127.0.0.1:35005'
10:27PM DBG GRPC Service state dir: /tmp/go-processmanager1788141561
10:27PM DBG GRPC Service Started
rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:35005: connect: connection refused"
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr 2023/12/19 22:27:37 gRPC Server listening at 127.0.0.1:35005
10:27PM DBG GRPC Service Ready
10:27PM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:<nil>} sizeCache:0 unknownFields:[] Model:bert-MiniLM-L6-v2q4_0.bin ContextSize:0 Seed:0 NBatch:512 F16Memory:false MLock:false MMap:false VocabOnly:false LowVRAM:false Embeddings:true NUMA:false NGPULayers:0 MainGPU: TensorSplit: Threads:4 LibrarySearchPath: RopeFreqBase:0 RopeFreqScale:0 RMSNormEps:0 NGQA:0 ModelFile:/models/bert-MiniLM-L6-v2q4_0.bin Device: UseTriton:false ModelBaseName: UseFastTokenizer:false PipelineType: SchedulerType: CUDA:false CFGScale:0 IMG2IMG:false CLIPModel: CLIPSubfolder: CLIPSkip:0 Tokenizer: LoraBase: LoraAdapter: LoraScale:0 NoMulMatQ:false DraftModel: AudioPath: Quantization: MMProj: RopeScaling: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0}
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stdout bert_load_from_file: loading model from '/models/bert-MiniLM-L6-v2q4_0.bin' - please wait ...
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stdout bert_load_from_file: n_vocab = 30522
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stdout bert_load_from_file: n_max_tokens   = 512
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stdout bert_load_from_file: n_embd  = 384
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stdout bert_load_from_file: n_intermediate  = 1536
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stdout bert_load_from_file: n_head  = 12
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stdout bert_load_from_file: n_layer = 6
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stdout bert_load_from_file: f16     = 2
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stdout bert_load_from_file: ggml ctx size =  13.57 MB
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr SIGSEGV: segmentation violation
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr PC=0x8429ea m=4 sigcode=1
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr signal arrived during cgo execution
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr goroutine 38 [syscall]:
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr runtime.cgocall(0x80e8b0, 0xc0003377e8)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/cgocall.go:157 +0x4b fp=0xc0003377c0 sp=0xc000337788 pc=0x40c90b
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr github.com/go-skynet/go-bert%2ecpp._Cfunc_bert_embeddings(0x7f05980017d0, 0x7f0594000b60, 0xc000400000)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     _cgo_gotypes.go:127 +0x4b fp=0xc0003377e8 sp=0xc0003377c0 pc=0x4f5a6b
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr github.com/go-skynet/go-bert%2ecpp.(*Bert).Embeddings.func1(0x7f0598000ca0?, 0x4?, 0x7f05980017d0?)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /build/sources/go-bert/gobert.go:38 +0x6a fp=0xc000337830 sp=0xc0003377e8 pc=0x4f61aa
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr github.com/go-skynet/go-bert%2ecpp.(*Bert).Embeddings(0xb2b?, {0xc00038c400, 0xb20}, {0xc000337918, 0x1, 0x8b2420?})
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /build/sources/go-bert/gobert.go:38 +0x105 fp=0xc000337898 sp=0xc000337830 pc=0x4f5fe5
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr main.(*Embeddings).Embeddings(0xe07a00?, 0x41253a?)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /build/backend/go/llm/bert/bert.go:33 +0xb3 fp=0xc000337930 sp=0xc000337898 pc=0x80df13
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr github.com/go-skynet/LocalAI/pkg/grpc.(*server).Embedding(0xc000038d80, {0xc000396000?, 0x4f84a6?}, 0x0?)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /build/pkg/grpc/server.go:37 +0xc6 fp=0xc0003379b0 sp=0xc000337930 pc=0x80c046
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr github.com/go-skynet/LocalAI/pkg/grpc/proto._Backend_Embedding_Handler({0x9149e0?, 0xc000038d80}, {0x9fa3f0, 0xc000392030}, 0xc000394000, 0x0)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /build/pkg/grpc/proto/backend_grpc.pb.go:303 +0x169 fp=0xc000337a08 sp=0xc0003379b0 pc=0x7ffd69
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr google.golang.org/grpc.(*Server).processUnaryRPC(0xc0001c41e0, {0x9fa3f0, 0xc000222540}, {0x9fd918, 0xc0001029c0}, 0xc000252000, 0xc0001cec90, 0xcee1c8, 0x0)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:1343 +0xe03 fp=0xc000337df0 sp=0xc000337a08 pc=0x7e88c3
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr google.golang.org/grpc.(*Server).handleStream(0xc0001c41e0, {0x9fd918, 0xc0001029c0}, 0xc000252000)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:1737 +0xc4c fp=0xc000337f78 sp=0xc000337df0 pc=0x7ed82c
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr google.golang.org/grpc.(*Server).serveStreams.func1.1()
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:986 +0x86 fp=0xc000337fe0 sp=0xc000337f78 pc=0x7e67c6
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr runtime.goexit()
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000337fe8 sp=0xc000337fe0 pc=0x46fc21
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 24
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:997 +0x145
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr goroutine 1 [IO wait]:
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr runtime.gopark(0x482a10?, 0xc0001b7b28?, 0x78?, 0x7b?, 0x4dc21d?)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0001b7b08 sp=0xc0001b7ae8 pc=0x44106e
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr runtime.netpollblock(0x46dc92?, 0x40c0a6?, 0x0?)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/netpoll.go:564 +0xf7 fp=0xc0001b7b40 sp=0xc0001b7b08 pc=0x439af7
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr internal/poll.runtime_pollWait(0x7f05a83f5eb0, 0x72)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/netpoll.go:343 +0x85 fp=0xc0001b7b60 sp=0xc0001b7b40 pc=0x46ab45
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr internal/poll.(*pollDesc).wait(0xc000184680?, 0x4?, 0x0)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0001b7b88 sp=0xc0001b7b60 pc=0x4d4e87
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr internal/poll.(*pollDesc).waitRead(...)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr internal/poll.(*FD).Accept(0xc000184680)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/internal/poll/fd_unix.go:611 +0x2ac fp=0xc0001b7c30 sp=0xc0001b7b88 pc=0x4da36c
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr net.(*netFD).accept(0xc000184680)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/net/fd_unix.go:172 +0x29 fp=0xc0001b7ce8 sp=0xc0001b7c30 pc=0x636c09
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr net.(*TCPListener).accept(0xc00007c4c0)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/net/tcpsock_posix.go:152 +0x1e fp=0xc0001b7d10 sp=0xc0001b7ce8 pc=0x64dbbe
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr net.(*TCPListener).Accept(0xc00007c4c0)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/net/tcpsock.go:315 +0x30 fp=0xc0001b7d40 sp=0xc0001b7d10 pc=0x64cd70
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr google.golang.org/grpc.(*Server).Serve(0xc0001c41e0, {0x9f99d0?, 0xc00007c4c0})
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:852 +0x462 fp=0xc0001b7e80 sp=0xc0001b7d40 pc=0x7e5422
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr github.com/go-skynet/LocalAI/pkg/grpc.StartServer({0x7ffda5812d45?, 0xc000024160?}, {0x9fe040?, 0xc000038cb0})
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /build/pkg/grpc/server.go:178 +0x17d fp=0xc0001b7f10 sp=0xc0001b7e80 pc=0x80dcbd
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr main.main()
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /build/backend/go/llm/bert/main.go:18 +0x85 fp=0xc0001b7f40 sp=0xc0001b7f10 pc=0x80e0e5
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr runtime.main()
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/proc.go:267 +0x2bb fp=0xc0001b7fe0 sp=0xc0001b7f40 pc=0x440c1b
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr runtime.goexit()
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0001b7fe8 sp=0xc0001b7fe0 pc=0x46fc21
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr goroutine 2 [force gc (idle)]:
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00005cfa8 sp=0xc00005cf88 pc=0x44106e
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr runtime.goparkunlock(...)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/proc.go:404
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr runtime.forcegchelper()
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/proc.go:322 +0xb3 fp=0xc00005cfe0 sp=0xc00005cfa8 pc=0x440ef3
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr runtime.goexit()
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00005cfe8 sp=0xc00005cfe0 pc=0x46fc21
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr created by runtime.init.6 in goroutine 1
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/proc.go:310 +0x1a
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr goroutine 18 [GC sweep wait]:
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000058778 sp=0xc000058758 pc=0x44106e
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr runtime.goparkunlock(...)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/proc.go:404
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr runtime.bgsweep(0x0?)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/mgcsweep.go:280 +0x94 fp=0xc0000587c8 sp=0xc000058778 pc=0x42cf74
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr runtime.gcenable.func1()
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/mgc.go:200 +0x25 fp=0xc0000587e0 sp=0xc0000587c8 pc=0x422145
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr runtime.goexit()
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000587e8 sp=0xc0000587e0 pc=0x46fc21
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr created by runtime.gcenable in goroutine 1
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/mgc.go:200 +0x66
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr goroutine 19 [GC scavenge wait]:
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr runtime.gopark(0xc00007e000?, 0x9f2c18?, 0x1?, 0x0?, 0xc0001024e0?)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000058f70 sp=0xc000058f50 pc=0x44106e
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr runtime.goparkunlock(...)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/proc.go:404
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr runtime.(*scavengerState).park(0xd37500)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000058fa0 sp=0xc000058f70 pc=0x42a849
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr runtime.bgscavenge(0x0?)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc000058fc8 sp=0xc000058fa0 pc=0x42addc
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr runtime.gcenable.func2()
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/mgc.go:201 +0x25 fp=0xc000058fe0 sp=0xc000058fc8 pc=0x4220e5
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr runtime.goexit()
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000058fe8 sp=0xc000058fe0 pc=0x46fc21
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr created by runtime.gcenable in goroutine 1
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/mgc.go:201 +0xa5
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr goroutine 3 [finalizer wait]:
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr runtime.gopark(0x93e860?, 0x100442201?, 0x0?, 0x0?, 0x449225?)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00005c628 sp=0xc00005c608 pc=0x44106e
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr runtime.runfinq()
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc00005c7e0 sp=0xc00005c628 pc=0x4211c7
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr runtime.goexit()
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00005c7e8 sp=0xc00005c7e0 pc=0x46fc21
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr created by runtime.createfing in goroutine 1
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/mfinal.go:163 +0x3d
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr goroutine 23 [select]:
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr runtime.gopark(0xc00032af70?, 0x4?, 0x0?, 0x40?, 0xc00032aec0?)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00032ad28 sp=0xc00032ad08 pc=0x44106e
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr runtime.selectgo(0xc00032af70, 0xc00032aeb8, 0x0?, 0x0, 0x0?, 0x1)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc00032ae48 sp=0xc00032ad28 pc=0x450ac5
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0001029c0)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /go/pkg/mod/google.golang.org/grpc@v1.59.0/internal/transport/http2_server.go:1152 +0x225 fp=0xc00032afc8 sp=0xc00032ae48 pc=0x77cbc5
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr google.golang.org/grpc/internal/transport.NewServerTransport.func4()
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /go/pkg/mod/google.golang.org/grpc@v1.59.0/internal/transport/http2_server.go:339 +0x25 fp=0xc00032afe0 sp=0xc00032afc8 pc=0x775805
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr runtime.goexit()
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00032afe8 sp=0xc00032afe0 pc=0x46fc21
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 21
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /go/pkg/mod/google.golang.org/grpc@v1.59.0/internal/transport/http2_server.go:339 +0x1b0e
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr goroutine 22 [select]:
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr runtime.gopark(0xc00024ff00?, 0x2?, 0x0?, 0x0?, 0xc00024fecc?)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00024fd78 sp=0xc00024fd58 pc=0x44106e
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr runtime.selectgo(0xc00024ff00, 0xc00024fec8, 0xc00024fee8?, 0x0, 0x8dcb40?, 0x1)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc00024fe98 sp=0xc00024fd78 pc=0x450ac5
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0001145f0, 0x1)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /go/pkg/mod/google.golang.org/grpc@v1.59.0/internal/transport/controlbuf.go:418 +0x113 fp=0xc00024ff30 sp=0xc00024fe98 pc=0x75e993
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000216070)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /go/pkg/mod/google.golang.org/grpc@v1.59.0/internal/transport/controlbuf.go:552 +0x86 fp=0xc00024ff90 sp=0xc00024ff30 pc=0x75f0c6
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr google.golang.org/grpc/internal/transport.NewServerTransport.func2()
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /go/pkg/mod/google.golang.org/grpc@v1.59.0/internal/transport/http2_server.go:336 +0xd5 fp=0xc00024ffe0 sp=0xc00024ff90 pc=0x775915
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr runtime.goexit()
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00024ffe8 sp=0xc00024ffe0 pc=0x46fc21
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 21
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /go/pkg/mod/google.golang.org/grpc@v1.59.0/internal/transport/http2_server.go:333 +0x1acc
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr goroutine 24 [IO wait]:
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr runtime.gopark(0xd4f660?, 0xb?, 0x0?, 0x0?, 0x6?)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00006daa0 sp=0xc00006da80 pc=0x44106e
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr runtime.netpollblock(0x480c98?, 0x40c0a6?, 0x0?)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/netpoll.go:564 +0xf7 fp=0xc00006dad8 sp=0xc00006daa0 pc=0x439af7
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr internal/poll.runtime_pollWait(0x7f05a83f5db8, 0x72)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/netpoll.go:343 +0x85 fp=0xc00006daf8 sp=0xc00006dad8 pc=0x46ab45
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr internal/poll.(*pollDesc).wait(0xc000118380?, 0xc000284000?, 0x0)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc00006db20 sp=0xc00006daf8 pc=0x4d4e87
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr internal/poll.(*pollDesc).waitRead(...)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr internal/poll.(*FD).Read(0xc000118380, {0xc000284000, 0x8000, 0x8000})
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a fp=0xc00006dbb8 sp=0xc00006db20 pc=0x4d617a
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr net.(*netFD).Read(0xc000118380, {0xc000284000?, 0x1060100000000?, 0x8?})
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/net/fd_posix.go:55 +0x25 fp=0xc00006dc00 sp=0xc00006dbb8 pc=0x634be5
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr net.(*conn).Read(0xc000144050, {0xc000284000?, 0xc00006dc90?, 0x3?})
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/net/net.go:179 +0x45 fp=0xc00006dc48 sp=0xc00006dc00 pc=0x6452e5
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr net.(*TCPConn).Read(0x0?, {0xc000284000?, 0xc00006dca0?, 0x45ef4d?})
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     <autogenerated>:1 +0x25 fp=0xc00006dc78 sp=0xc00006dc48 pc=0x657a85
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr bufio.(*Reader).Read(0xc000112660, {0xc000294040, 0x9, 0xc158a416f31ffe54?})
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/bufio/bufio.go:244 +0x197 fp=0xc00006dcb0 sp=0xc00006dc78 pc=0x5b01b7
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr io.ReadAtLeast({0x9f7460, 0xc000112660}, {0xc000294040, 0x9, 0x9}, 0x9)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/io/io.go:335 +0x90 fp=0xc00006dcf8 sp=0xc00006dcb0 pc=0x4bad30
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr io.ReadFull(...)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/io/io.go:354
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr golang.org/x/net/http2.readFrameHeader({0xc000294040, 0x9, 0xc00021a078?}, {0x9f7460?, 0xc000112660?})
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /go/pkg/mod/golang.org/x/net@v0.17.0/http2/frame.go:237 +0x65 fp=0xc00006dd48 sp=0xc00006dcf8 pc=0x74b405
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr golang.org/x/net/http2.(*Framer).ReadFrame(0xc000294000)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /go/pkg/mod/golang.org/x/net@v0.17.0/http2/frame.go:498 +0x85 fp=0xc00006ddf0 sp=0xc00006dd48 pc=0x74bb45
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0001029c0, 0x1?)
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /go/pkg/mod/google.golang.org/grpc@v1.59.0/internal/transport/http2_server.go:636 +0x145 fp=0xc00006df00 sp=0xc00006ddf0 pc=0x778a65
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr google.golang.org/grpc.(*Server).serveStreams(0xc0001c41e0, {0x9fd918?, 0xc0001029c0})
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:979 +0x1c2 fp=0xc00006df80 sp=0xc00006df00 pc=0x7e6562
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr google.golang.org/grpc.(*Server).handleRawConn.func1()
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:920 +0x45 fp=0xc00006dfe0 sp=0xc00006df80 pc=0x7e5dc5
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr runtime.goexit()
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0x46fc21
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 21
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr     /go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:919 +0x185
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr rax    0x0
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr rbx    0x7f0587bef970
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr rcx    0x0
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr rdx    0x7f05a3fff700
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr rdi    0x7f05a3fe6610
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr rsi    0x0
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr rbp    0x7f05a3fe6bf0
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr rsp    0x7f05a3fe6ba0
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr r8     0x0
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr r9     0x7f05a3fe6a3f
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr r10    0xf8a0340
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr r11    0x0
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr r12    0x7f05879af4c0
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr r13    0x7f05876af010
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr r14    0xd66248
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr r15    0x0
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr rip    0x8429ea
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr rflags 0x10246
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr cs     0x33
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr fs     0x0
10:27PM DBG GRPC(bert-MiniLM-L6-v2q4_0.bin-127.0.0.1:35005): stderr gs     0x0
[10.42.0.138]:46246 500 - POST /v1/embeddings

Additional context

lenaxia commented 11 months ago

shortened context window fixed it

luoweb commented 8 months ago

shortened context window fixed it

how to shortened context window? in model config?

InputToken:[] functionCallString: functionCallNameString: FunctionsConfig:{DisableNoAction:false NoActionFunctionName: NoActionDescriptionName: ParallelCalls:false} FeatureFlag:map[] LLMConfig:{SystemPrompt: TensorSplit: MainGPU: RMSNormEps:0 NGQA:0 PromptCachePath: PromptCacheAll:false PromptCacheRO:false MirostatETA:0x140001aa848 MirostatTAU:0x140001aa840 Mirostat:0x140001aa838 NGPULayers:0x140001aa850 MMap:0x140001aa858 MMlock:0x140001aa859 LowVRAM:0x140001aa859 Grammar: StopWords:[] Cutstrings:[] TrimSpace:[] TrimSuffix:[] ContextSize:0x140001aa808 NUMA:false LoraAdapter: LoraBase: LoraScale:0 NoMulMatQ:false DraftModel: NDraft:0 Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 MMProj: RopeScaling: ModelType: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0} AutoGPTQ:{ModelBaseName: Device: Triton:false UseFastTokenizer:false} Diffusers:{CUDA:false PipelineType: SchedulerType: EnableParameters: CFGScale:0 IMG2IMG:false ClipSkip:0 ClipModel: ClipSubFolder: ControlNet:} Step:0 GRPC:{Attempts:0 AttemptsSleepTime:0} VallE:{AudioPath:} CUDA:false DownloadFiles:[] Description: Usage:}
11:22PM INF Loading model 'bert-MiniLM-L6-v2q4_1.bin' with backend bert-embeddings
11:22PM DBG Model already loaded in memory: bert-MiniLM-L6-v2q4_1.bin
11:22PM WRN GRPC Model not responding: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:59492: connect: connection refused"
11:22PM WRN Deleting the process in order to recreate it
11:22PM DBG GRPC Process is not responding: bert-MiniLM-L6-v2q4_1.bin
11:22PM DBG Loading model in memory from file: /Users/block/code/data/models/bert-MiniLM-L6-v2q4_1.bin
11:22PM DBG Loading Model bert-MiniLM-L6-v2q4_1.bin with gRPC (file: /Users/block/code/data/models/bert-MiniLM-L6-v2q4_1.bin) (backend: bert-embeddings): {backendString:bert-embeddings model:bert-MiniLM-L6-v2q4_1.bin threads:10 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0x140001f3000 externalBackends:map[sentencetransformers:/Users/block/code/LocalAI/backend/python/sentencetransformers/run.sh] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false parallelRequests:false}
11:22PM DBG Loading GRPC Process: /tmp/localai/backend_data/backend-assets/grpc/bert-embeddings
11:22PM DBG GRPC Service for bert-MiniLM-L6-v2q4_1.bin will be running at: '127.0.0.1:59527'
11:22PM DBG GRPC Service state dir: /var/folders/t0/y4k0vcfx5_bd9qx7pl7lbj9h0000gn/T/go-processmanager380860985
11:22PM DBG GRPC Service Started
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stderr 2024/03/20 23:22:55 gRPC Server listening at 127.0.0.1:59527
11:22PM DBG GRPC Service Ready
11:22PM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:<nil>} sizeCache:0 unknownFields:[] Model:bert-MiniLM-L6-v2q4_1.bin ContextSize:1024 Seed:2084307436 NBatch:512 F16Memory:false MLock:false MMap:true VocabOnly:false LowVRAM:false Embeddings:true NUMA:false NGPULayers:99999999 MainGPU: TensorSplit: Threads:10 LibrarySearchPath: RopeFreqBase:0 RopeFreqScale:0 RMSNormEps:0 NGQA:0 ModelFile:/Users/block/code/data/models/bert-MiniLM-L6-v2q4_1.bin Device: UseTriton:false ModelBaseName: UseFastTokenizer:false PipelineType: SchedulerType: CUDA:false CFGScale:0 IMG2IMG:false CLIPModel: CLIPSubfolder: CLIPSkip:0 ControlNet: Tokenizer: LoraBase: LoraAdapter: LoraScale:0 NoMulMatQ:false DraftModel: AudioPath: Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 MMProj: RopeScaling: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0 Type:}
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stdout bert_load_from_file: loading model from '/Users/block/code/data/models/bert-MiniLM-L6-v2q4_1.bin' - please wait ...
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stdout bert_load_from_file: n_vocab = 30522
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stdout bert_load_from_file: n_max_tokens   = 512
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stdout bert_load_from_file: n_embd  = 384
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stdout bert_load_from_file: n_intermediate  = 1536
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stdout bert_load_from_file: n_head  = 12
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stdout bert_load_from_file: n_layer = 6
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stdout bert_load_from_file: f16     = 3
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stdout bert_load_from_file: ggml ctx size =  16.26 MB
11:22PM INF Loading model 'bert-MiniLM-L6-v2q4_1.bin' with backend bert-embeddings
11:22PM DBG Model already loaded in memory: bert-MiniLM-L6-v2q4_1.bin

.................

11:22PM INF Loading model 'bert-MiniLM-L6-v2q4_1.bin' with backend bert-embeddings
11:22PM DBG Model already loaded in memory: bert-MiniLM-L6-v2q4_1.bin
11:22PM INF Loading model 'bert-MiniLM-L6-v2q4_1.bin' with backend bert-embeddings
11:22PM DBG Model already loaded in memory: bert-MiniLM-L6-v2q4_1.bin
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stderr SIGSEGV: segmentation violation
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stderr PC=0x102fa2600 m=4 sigcode=2
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stderr signal arrived during cgo execution
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stderr 
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stderr goroutine 194 [syscall]:
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stderr runtime.cgocall(0x102f8aa9c, 0x140001377a8)
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stderr     /usr/local/go/src/runtime/cgocall.go:157 +0x44 fp=0x14000137770 sp=0x14000137730 pc=0x102bf7804
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stderr github.com/go-skynet/go-bert%2ecpp._Cfunc_bert_embeddings(0x1557044e0, 0x155610a00, 0x14000404000)
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stderr     _cgo_gotypes.go:138 +0x34 fp=0x140001377a0 sp=0x14000137770 pc=0x102cdb624
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stderr github.com/go-skynet/go-bert%2ecpp.(*Bert).Embeddings.func1(0x1557258f0?, 0xa?, 0x1557044e0?)
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stderr     /Users/block/code/LocalAI/sources/go-bert/gobert.go:38 +0x74 fp=0x140001377f0 sp=0x140001377a0 pc=0x102cdbca4
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stderr github.com/go-skynet/go-bert%2ecpp.(*Bert).Embeddings(0x140001c6d80?, {0x1400019c800, 0x3e7}, {0x140001378f0, 0x1, 0x140001f8900?})
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stderr     /Users/block/code/LocalAI/sources/go-bert/gobert.go:38 +0xe8 fp=0x14000137860 sp=0x140001377f0 pc=0x102cdbac8
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stderr main.(*Embeddings).Embeddings(0x14000137958?, 0x102bfe084?)
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stderr     /Users/block/code/LocalAI/backend/go/llm/bert/bert.go:33 +0x90 fp=0x14000137900 sp=0x14000137860 pc=0x102f8a070
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stderr github.com/go-skynet/LocalAI/pkg/grpc.(*server).Embedding(0x14000192c90, {0x1400020e180?, 0x14000218380?}, 0x0?)
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stderr     /Users/block/code/LocalAI/pkg/grpc/server.go:37 +0xb8 fp=0x14000137990 sp=0x14000137900 pc=0x102f88488
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stderr github.com/go-skynet/LocalAI/pkg/grpc/proto._Backend_Embedding_Handler({0x10313d6c0?, 0x14000192c90}, {0x103170358, 0x140002bf980}, 0x14000218380, 0x0)
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stderr     /Users/block/code/LocalAI/pkg/grpc/proto/backend_grpc.pb.go:303 +0x164 fp=0x140001379f0 sp=0x14000137990 pc=0x102f82bc4
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stderr google.golang.org/grpc.(*Server).processUnaryRPC(0x140002481e0, {0x103170358, 0x140002bf8c0}, {0x103173878, 0x1400008a820}, 0x140001f8240, 0x1400025e9f0, 0x10342ea68, 0x0)
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stderr     /Users/block/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:1343 +0xb8c fp=0x14000137de0 sp=0x140001379f0 pc=0x102f6d33c
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stderr google.golang.org/grpc.(*Server).handleStream(0x140002481e0, {0x103173878, 0x1400008a820}, 0x140001f8240)
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stderr     /Users/block/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:1737 +0x988 fp=0x14000137f60 sp=0x14000137de0 pc=0x102f714e8
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stderr google.golang.org/grpc.(*Server).serveStreams.func1.1()

.............

11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stdout bert_load_from_file: ............ done
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stdout bert_load_from_file: model size =    16.24 MB / num tensors = 101
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stdout bert_load_from_file: mem_per_token 452 KB, mem_per_input 248 MB
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stdout loaded
11:22PM DBG GRPC(bert-MiniLM-L6-v2q4_1.bin-127.0.0.1:59527): stdout ggml_new_tensor_impl: not enough space in the context's memory pool (needed 271388624, available 260919296)
[127.0.0.1]:59525 500 - POST /v1/embeddings