search

mudler / LocalAI

:robot: The free, Open Source OpenAI alternative. Self-hosted, community-driven and local-first. Drop-in replacement for OpenAI running on consumer-grade hardware. No GPU required. Runs gguf, transformers, diffusers and many more models architectures. It allows to generate Text, Audio, Video, Images. Also with voice cloning capabilities.
https://localai.io
MIT License
21.75k stars 1.66k forks source link

SIGSEGV: GGML_ASSERT: /build/go-llama-stable/llama.cpp/ggml-cuda.cu:5276: src0->type == GGML_TYPE_F16 #976

Open racerxdl opened 10 months ago

racerxdl commented 10 months ago

LocalAI version:

v1.25.0-cublas-cuda12-ffmpeg

Environment, CPU architecture, OS, and Version:

# uname -a
Linux localai-ix-chart-f8bbbb7c7-x6xx9 6.1.42-production+truenas #2 SMP PREEMPT_DYNAMIC Mon Aug 14 23:21:26 UTC 2023 x86_64 GNU/Linux
# nvidia-smi
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.54.03              Driver Version: 535.54.03    CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla P40                      Off | 00000000:23:00.0 Off |                  Off |
| N/A   24C    P8              10W / 250W |      0MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  Tesla P40                      Off | 00000000:24:00.0 Off |                  Off |
| N/A   23C    P8               9W / 250W |      0MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

# cat /proc/cpuinfo |grep "model name" | nl
     1  model name      : Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz
     2  model name      : Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz
     3  model name      : Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz
     4  model name      : Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz
     5  model name      : Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz
     6  model name      : Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz
     7  model name      : Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz
     8  model name      : Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz
     9  model name      : Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz
    10  model name      : Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz
    11  model name      : Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz
    12  model name      : Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz
    13  model name      : Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz
    14  model name      : Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz
    15  model name      : Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz
    16  model name      : Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz
    17  model name      : Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz
    18  model name      : Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz
    19  model name      : Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz
    20  model name      : Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz
# cat /proc/meminfo  | grep Mem
MemTotal:       32701568 kB
MemFree:        18305148 kB
MemAvailable:   18767368 kB

Describe the bug Assert crash for GGML

To Reproduce Use wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin model.

Yaml Config File:

backend: llama-stable
context_size: 2048
name: thebloke__wizardlm-33b-v1.0-uncensored-ggml__wizardlm-33b-v1.0-uncensored.ggmlv3.q4_k_s.bin
parameters:
  model: wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin
  temperature: 0.2
  top_k: 80
  top_p: 0.7
  f16: false
template:
  chat: wizardlm-chat
  completion: wizardlm-completion
gpu_layers: 63
f16: false

Call:

curl $LOCALAI/v1/chat/completions -H "Content-Type: application/json" -d '{
     "model": "thebloke__wizardlm-33b-v1.0-uncensored-ggml__wizardlm-33b-v1.0-uncensored.ggmlv3.q4_k_s.bin",
     "messages": [{"role": "user", "content": "Give me a HTTP REST server made in rust that uses sqlite."}],
     "temperature": 0.9
   }' | jq

Response from CURL:

{
  "error": {
    "code": 500,
    "message": "rpc error: code = Unavailable desc = error reading from server: EOF",
    "type": ""
  }
}

Expected behavior Work

Logs

2023-08-29 21:09:52.546551-07:004:09AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr create_gpt_params: loading model /models/wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin
2023-08-29 21:09:52.615503-07:004:09AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr ggml_init_cublas: found 2 CUDA devices:
2023-08-29 21:09:52.615640-07:004:09AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr   Device 0: Tesla P40, compute capability 6.1
2023-08-29 21:09:52.615664-07:004:09AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr   Device 1: Tesla P40, compute capability 6.1
2023-08-29 21:09:53.032791-07:004:09AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr llama.cpp: loading model from /models/wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin
2023-08-29 21:09:53.135327-07:004:09AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr llama_model_load_internal: format     = ggjt v3 (latest)
2023-08-29 21:09:53.135398-07:004:09AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr llama_model_load_internal: n_vocab    = 32000
2023-08-29 21:09:53.135462-07:004:09AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr llama_model_load_internal: n_ctx      = 2048
2023-08-29 21:09:53.135482-07:004:09AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr llama_model_load_internal: n_embd     = 6656
2023-08-29 21:09:53.135516-07:004:09AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr llama_model_load_internal: n_mult     = 256
2023-08-29 21:09:53.135555-07:004:09AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr llama_model_load_internal: n_head     = 52
2023-08-29 21:09:53.135573-07:004:09AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr llama_model_load_internal: n_head_kv  = 52
2023-08-29 21:09:53.135626-07:004:09AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr llama_model_load_internal: n_layer    = 60
2023-08-29 21:09:53.135663-07:004:09AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr llama_model_load_internal: n_rot      = 128
2023-08-29 21:09:53.135692-07:004:09AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr llama_model_load_internal: n_gqa      = 1
2023-08-29 21:09:53.135744-07:004:09AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr llama_model_load_internal: rnorm_eps  = 5.0e-06
2023-08-29 21:09:53.135774-07:004:09AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr llama_model_load_internal: n_ff       = 17920
2023-08-29 21:09:53.135799-07:004:09AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr llama_model_load_internal: freq_base  = 10000.0
2023-08-29 21:09:53.135862-07:004:09AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr llama_model_load_internal: freq_scale = 1
2023-08-29 21:09:53.135905-07:004:09AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr llama_model_load_internal: ftype      = 14 (mostly Q4_K - Small)
2023-08-29 21:09:53.135935-07:004:09AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr llama_model_load_internal: model size = 30B
2023-08-29 21:09:53.135946-07:004:09AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr llama_model_load_internal: ggml ctx size = 17505.06 MB
2023-08-29 21:09:59.391824-07:004:09AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr llama_model_load_internal: using CUDA for GPU acceleration
2023-08-29 21:09:59.391895-07:004:09AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr ggml_cuda_set_main_device: using device 0 (Tesla P40) as main device
2023-08-29 21:09:59.392656-07:004:09AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr llama_model_load_internal: mem required  =  773.42 MB (+ 6240.00 MB per state)
2023-08-29 21:09:59.392733-07:004:09AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr llama_model_load_internal: allocating batch_size x (768 kB + n_ctx x 208 B) = 592 MB VRAM for the scratch buffer
2023-08-29 21:09:59.392754-07:004:09AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr llama_model_load_internal: offloading 60 repeating layers to GPU
2023-08-29 21:09:59.392801-07:004:09AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr llama_model_load_internal: offloading non-repeating layers to GPU
2023-08-29 21:09:59.392849-07:004:09AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr llama_model_load_internal: offloading v cache to GPU
2023-08-29 21:09:59.392871-07:004:09AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr llama_model_load_internal: offloading k cache to GPU
2023-08-29 21:09:59.392920-07:004:09AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr llama_model_load_internal: offloaded 63/63 layers to GPU
2023-08-29 21:09:59.392939-07:004:09AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr llama_model_load_internal: total VRAM used: 21103 MB
2023-08-29 21:10:18.116739-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr llama_new_context_with_model: kv self size  = 6240.00 MB
2023-08-29 21:10:18.809091-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr GGML_ASSERT: /build/go-llama-stable/llama.cpp/ggml-cuda.cu:5276: src0->type == GGML_TYPE_F16
2023-08-29 21:10:18.811030-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr SIGABRT: abort
2023-08-29 21:10:18.811054-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr PC=0x7ff872616ce1 m=0 sigcode=18446744073709551610
2023-08-29 21:10:18.811068-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr signal arrived during cgo execution
2023-08-29 21:10:18.811088-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr 
2023-08-29 21:10:18.811657-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr goroutine 27 [syscall]:
2023-08-29 21:10:18.811684-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr runtime.cgocall(0x819bf0, 0xc00017d618)
2023-08-29 21:10:18.811709-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr   /usr/local/go/src/runtime/cgocall.go:157 +0x4b fp=0xc00017d5f0 sp=0xc00017d5b8 pc=0x41442b
2023-08-29 21:10:18.811907-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr github.com/go-skynet/go-llama%2ecpp._Cfunc_llama_predict(0x327ce20, 0x7ff7e8001110, 0xc000380000, 0x1)
2023-08-29 21:10:18.812892-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr   _cgo_gotypes.go:235 +0x4b fp=0xc00017d618 sp=0xc00017d5f0 pc=0x81124b
2023-08-29 21:10:18.813159-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr github.com/go-skynet/go-llama%2ecpp.(*LLama).Predict.func2(0xc00019e3c0?, 0xc00017d810?, {0xc000380000, 0x0?, 0x1125d00?}, 0xc00013d700?)
2023-08-29 21:10:18.813281-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr   /build/go-llama-stable/llama.go:233 +0x98 fp=0xc00017d668 sp=0xc00017d618 pc=0x813f78
2023-08-29 21:10:18.813880-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr github.com/go-skynet/go-llama%2ecpp.(*LLama).Predict(0xc000012450, {0xc00019e3c0, 0x48}, {0xc0002a03c0, 0x13, 0x5e?})
2023-08-29 21:10:18.814529-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr   /build/go-llama-stable/llama.go:233 +0x28f fp=0xc00017d930 sp=0xc00017d668 pc=0x813c2f
2023-08-29 21:10:18.814631-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr github.com/go-skynet/LocalAI/pkg/backend/llm/llama-stable.(*LLM).Predict(0xc000118c60, 0xc00015d080)
2023-08-29 21:10:18.815014-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr   /build/pkg/backend/llm/llama-stable/llama.go:170 +0x52 fp=0xc00017d970 sp=0xc00017d930 pc=0x816eb2
2023-08-29 21:10:18.816693-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr github.com/go-skynet/LocalAI/pkg/grpc.(*server).Predict(0xc000118d30, {0xc00015d080?, 0x509426?}, 0x0?)
2023-08-29 21:10:18.816764-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr   /build/pkg/grpc/server.go:62 +0xd4 fp=0xc00017da08 sp=0xc00017d970 pc=0x817ff4
2023-08-29 21:10:18.816826-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr github.com/go-skynet/LocalAI/pkg/grpc/proto._Backend_Predict_Handler({0x979ec0?, 0xc000118d30}, {0xa60070, 0xc0001e55c0}, 0xc0001d9650, 0x0)
2023-08-29 21:10:18.816950-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr   /build/pkg/grpc/proto/backend_grpc.pb.go:246 +0x169 fp=0xc00017da60 sp=0xc00017da08 pc=0x806989
2023-08-29 21:10:18.817219-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr google.golang.org/grpc.(*Server).processUnaryRPC(0xc0001d61e0, {0xa631b8, 0xc0001fe1a0}, 0xc0001837a0, 0xc0001e4c00, 0x10dc5d8, 0x0)
2023-08-29 21:10:18.818920-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr   /go/pkg/mod/google.golang.org/grpc@v1.57.0/server.go:1360 +0xe15 fp=0xc00017de40 sp=0xc00017da60 pc=0x7f0175
2023-08-29 21:10:18.818941-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr google.golang.org/grpc.(*Server).handleStream(0xc0001d61e0, {0xa631b8, 0xc0001fe1a0}, 0xc0001837a0, 0x0)
2023-08-29 21:10:18.818973-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr   /go/pkg/mod/google.golang.org/grpc@v1.57.0/server.go:1737 +0x9e7 fp=0xc00017df68 sp=0xc00017de40 pc=0x7f4ea7
2023-08-29 21:10:18.818999-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr google.golang.org/grpc.(*Server).serveStreams.func1.1()
2023-08-29 21:10:18.819032-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr   /go/pkg/mod/google.golang.org/grpc@v1.57.0/server.go:982 +0x8d fp=0xc00017dfe0 sp=0xc00017df68 pc=0x7edeed
2023-08-29 21:10:18.819049-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr runtime.goexit()
2023-08-29 21:10:18.819071-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr   /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00017dfe8 sp=0xc00017dfe0 pc=0x477741
2023-08-29 21:10:18.819381-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 26
2023-08-29 21:10:18.819406-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr   /go/pkg/mod/google.golang.org/grpc@v1.57.0/server.go:980 +0x165
2023-08-29 21:10:18.819455-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr 
2023-08-29 21:10:18.819508-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr goroutine 1 [IO wait]:
2023-08-29 21:10:18.819621-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr runtime.gopark(0x4c3870?, 0xc0001c9b28?, 0xa0?, 0x9b?, 0xc0001c9b78?)
2023-08-29 21:10:18.819743-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr   /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00027bb08 sp=0xc00027bae8 pc=0x448bce
2023-08-29 21:10:18.819898-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr runtime.netpollblock(0x4757b2?, 0x413bc6?, 0x0?)
2023-08-29 21:10:18.820673-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr   /usr/local/go/src/runtime/netpoll.go:564 +0xf7 fp=0xc00027bb40 sp=0xc00027bb08 pc=0x441657
2023-08-29 21:10:18.820728-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr internal/poll.runtime_pollWait(0x7ff80660be58, 0x72)
2023-08-29 21:10:18.820845-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr   /usr/local/go/src/runtime/netpoll.go:343 +0x85 fp=0xc00027bb60 sp=0xc00027bb40 pc=0x472665
2023-08-29 21:10:18.821779-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr internal/poll.(*pollDesc).wait(0xc000198600?, 0x4?, 0x0)
2023-08-29 21:10:18.821887-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr   /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc00027bb88 sp=0xc00027bb60 pc=0x4dc8c7
2023-08-29 21:10:18.822045-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr internal/poll.(*pollDesc).waitRead(...)
2023-08-29 21:10:18.822105-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr   /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
2023-08-29 21:10:18.822309-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr internal/poll.(*FD).Accept(0xc000198600)
2023-08-29 21:10:18.822803-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr   /usr/local/go/src/internal/poll/fd_unix.go:611 +0x2ac fp=0xc00027bc30 sp=0xc00027bb88 pc=0x4e1dac
2023-08-29 21:10:18.823388-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr net.(*netFD).accept(0xc000198600)
2023-08-29 21:10:18.823731-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr   /usr/local/go/src/net/fd_unix.go:172 +0x29 fp=0xc00027bce8 sp=0xc00027bc30 pc=0x640169
2023-08-29 21:10:18.823843-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr net.(*TCPListener).accept(0xc00015a440)
2023-08-29 21:10:18.824530-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr   /usr/local/go/src/net/tcpsock_posix.go:152 +0x1e fp=0xc00027bd10 sp=0xc00027bce8 pc=0x6570fe
2023-08-29 21:10:18.824730-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr net.(*TCPListener).Accept(0xc00015a440)
2023-08-29 21:10:18.825339-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr   /usr/local/go/src/net/tcpsock.go:315 +0x30 fp=0xc00027bd40 sp=0xc00027bd10 pc=0x6562b0
2023-08-29 21:10:18.825911-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr google.golang.org/grpc.(*Server).Serve(0xc0001d61e0, {0xa5f628?, 0xc00015a440})
2023-08-29 21:10:18.826372-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr   /go/pkg/mod/google.golang.org/grpc@v1.57.0/server.go:844 +0x462 fp=0xc00027be80 sp=0xc00027bd40 pc=0x7ecba2
2023-08-29 21:10:18.827566-07:004:10AM DBG GRPC(wizardlm-33b-v1.0-uncensored.ggmlv3.q4_K_S.bin-127.0.0.1:45367): stderr github.com/go-skynet/LocalAI/pkg/grpc.StartServer({0x7ffcb86b29a6?, 0xc00011c130?}, {0xa63ca0?, 0xc000118c60})
(...)
lagmoellertim commented 9 months ago

Hey, could you try adding f16: true to your model config yaml? This worked for me.

neowisard commented 5 months ago

Tesla P40 doesn't have F16 support, only INT8 ot F32. F16 cores of P40 very slow.