Failed to load the model causes the segmentation violation error with code=0x1 addr=0x10 pc=0x7fb924380eb8

[liuyang77886]

环境信息(Environment information)：

root@autodl-container-a4da118bfa-64e0dc0a:~/autodl-tmp/LocalAI# nvidia-smi
Sun Jun 18 17:02:20 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.105.17   Driver Version: 525.105.17   CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            On   | 00000000:00:07.0 Off |                  Off |
| N/A   32C    P8    16W /  70W |      0MiB / 16384MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

root@autodl-container-a4da118bfa-64e0dc0a:~/autodl-tmp/LocalAI# python --version
Python 3.10.8

不论我调用ggml-gpt4all-j 或者 ggml-gpt4all-j-v1.3-groovy.bin 都报错(I got errors with models below)：

root@autodl-container-a4da118bfa-64e0dc0a:~/autodl-tmp/LocalAI# curl http://localhost:8080/v1/models
{"object":"list","data":[{"id":"ggml-gpt4all-j","object":"model"},{"id":"ggml-gpt4all-j-v1.3-groovy.bin","object":"model"}]}

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
     "model": "ggml-gpt4all-j",
     "messages": [{"role": "user", "content": "How are you?"}],
     "temperature": 0.9 
   }'

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
     "model": "ggml-gpt4all-j-v1.3-groovy.bin",
     "messages": [{"role": "user", "content": "How are you?"}],
     "temperature": 0.9 
   }'

root@autodl-container-a4da118bfa-64e0dc0a:~/autodl-tmp/LocalAI# ./local-ai-avx512-Linux-x86_64 
Starting LocalAI using 4 threads, with models path: /root/autodl-tmp/LocalAI/models

 ┌───────────────────────────────────────────────────┐ 
 │                   Fiber v2.46.0                   │ 
 │               http://127.0.0.1:8080               │ 
 │       (bound on host 0.0.0.0 and port 8080)       │ 
 │                                                   │ 
 │ Handlers ............ 23  Processes ........... 1 │ 
 │ Prefork ....... Disabled  PID .............. 1365 │ 
 └───────────────────────────────────────────────────┘ 

llama.cpp: loading model from /root/autodl-tmp/LocalAI/models/ggml-gpt4all-j-v1.3-groovy.bin
error loading model: unexpectedly reached end of file
llama_init_from_file: failed to load model
gptj_model_load: loading model from '/root/autodl-tmp/LocalAI/models/ggml-gpt4all-j-v1.3-groovy.bin' - please wait ...
gptj_model_load: n_vocab = 50400
gptj_model_load: n_ctx   = 2048
gptj_model_load: n_embd  = 4096
gptj_model_load: n_head  = 16
gptj_model_load: n_layer = 28
gptj_model_load: n_rot   = 64
gptj_model_load: f16     = 2
gptj_model_load: ggml ctx size = 5401.45 MB
gptj_model_load: kv self size  =  896.00 MB
gptj_model_load: ................................... done
gptj_model_load: model size =  3609.38 MB / num tensors = 285
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x7fb924380eb8]

runtime stack:
runtime.throw({0xe01743?, 0x803296a26218cd00?})
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/panic.go:1047 +0x5d fp=0x7fb91ffe6720 sp=0x7fb91ffe66f0 pc=0x44ad9d
runtime.sigpanic()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/signal_unix.go:821 +0x3e9 fp=0x7fb91ffe6780 sp=0x7fb91ffe6720 pc=0x460e49

goroutine 31 [syscall]:
runtime.cgocall(0x9dcbb0, 0xc00014cc98)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/cgocall.go:157 +0x5c fp=0xc00014cc70 sp=0xc00014cc38 pc=0x419b7c
github.com/nomic-ai/gpt4all/gpt4all-bindings/golang._Cfunc_model_prompt(0x7fb918015600, 0x7fb9180038b0, 0xc000544000, 0xa, 0x3f99999a, 0x400, 0x200, 0x50, 0x3f333333, 0x3f666666, ...)
        _cgo_gotypes.go:114 +0x45 fp=0xc00014cc98 sp=0xc00014cc70 pc=0x8a3b05
github.com/nomic-ai/gpt4all/gpt4all-bindings/golang.(*Model).Predict.func1(0x42a2c5?, 0xc0003a09f0?, {0xc000544000, 0x200?, 0x40?}, {0x400, 0xa, 0x200, 0x50, 0x1, ...})
        /home/runner/work/LocalAI/LocalAI/gpt4all/gpt4all-bindings/golang/gpt4all.go:56 +0x185 fp=0xc00014cd40 sp=0xc00014cc98 pc=0x8a4545
github.com/nomic-ai/gpt4all/gpt4all-bindings/golang.(*Model).Predict(0x0?, {0xc00040ab60, 0xc}, {0xc00014cf68, 0x4, 0xd0?})
        /home/runner/work/LocalAI/LocalAI/gpt4all/gpt4all-bindings/golang/gpt4all.go:56 +0x225 fp=0xc00014cea0 sp=0xc00014cd40 pc=0x8a41c5
github.com/go-skynet/LocalAI/api.ModelInference.func11()
        /home/runner/work/LocalAI/LocalAI/api/prediction.go:492 +0x270 fp=0xc00014cf98 sp=0xc00014cea0 pc=0x921230
github.com/go-skynet/LocalAI/api.ModelInference.func14()
        /home/runner/work/LocalAI/LocalAI/api/prediction.go:554 +0x1aa fp=0xc00014d050 sp=0xc00014cf98 pc=0x920a8a
github.com/go-skynet/LocalAI/api.ComputeChoices({0xc00040ab60, 0xc}, 0xc000176140, 0xc000142c80, 0xc0003fe820?, 0x1263a50, 0x4?)
        /home/runner/work/LocalAI/LocalAI/api/prediction.go:578 +0x224 fp=0xc00014d840 sp=0xc00014d050 pc=0x923dc4
github.com/go-skynet/LocalAI/api.chatEndpoint.func2(0xc0000de000)
        /home/runner/work/LocalAI/LocalAI/api/openai.go:439 +0x8c5 fp=0xc00014d9f0 sp=0xc00014d840 pc=0x91aee5
github.com/gofiber/fiber/v2.(*App).next(0xc000163200, 0xc0000de000)
        /home/runner/go/pkg/mod/github.com/gofiber/fiber/v2@v2.46.0/router.go:144 +0x1bf fp=0xc00014da98 sp=0xc00014d9f0 pc=0x86679f
github.com/gofiber/fiber/v2.(*Ctx).Next(0x14?)
        /home/runner/go/pkg/mod/github.com/gofiber/fiber/v2@v2.46.0/ctx.go:913 +0x53 fp=0xc00014dab8 sp=0xc00014da98 pc=0x851db3
github.com/gofiber/fiber/v2/middleware/recover.New.func1(0xd5fcc0?)
        /home/runner/go/pkg/mod/github.com/gofiber/fiber/v2@v2.46.0/middleware/recover/recover.go:43 +0xcb fp=0xc00014db30 sp=0xc00014dab8 pc=0x87320b
github.com/gofiber/fiber/v2.(*App).next(0xc000163200, 0xc0000de000)
        /home/runner/go/pkg/mod/github.com/gofiber/fiber/v2@v2.46.0/router.go:144 +0x1bf fp=0xc00014dbd8 sp=0xc00014db30 pc=0x86679f
github.com/gofiber/fiber/v2.(*App).handler(0xc000163200, 0x4a1197?)
        /home/runner/go/pkg/mod/github.com/gofiber/fiber/v2@v2.46.0/router.go:171 +0x87 fp=0xc00014dc38 sp=0xc00014dbd8 pc=0x8669e7
github.com/gofiber/fiber/v2.(*App).handler-fm(0xc0000d6000?)
        <autogenerated>:1 +0x2c fp=0xc00014dc58 sp=0xc00014dc38 pc=0x86bc0c
github.com/valyala/fasthttp.(*Server).serveConn(0xc00040c000, {0x12f6840?, 0xc0001263e8})
        /home/runner/go/pkg/mod/github.com/valyala/fasthttp@v1.47.0/server.go:2365 +0x11d3 fp=0xc00014dec8 sp=0xc00014dc58 pc=0x7ca073
github.com/valyala/fasthttp.(*Server).serveConn-fm({0x12f6840?, 0xc0001263e8?})
        <autogenerated>:1 +0x39 fp=0xc00014def0 sp=0xc00014dec8 pc=0x7d9939
github.com/valyala/fasthttp.(*workerPool).workerFunc(0xc000144aa0, 0xc00012e560)
        /home/runner/go/pkg/mod/github.com/valyala/fasthttp@v1.47.0/workerpool.go:224 +0xa9 fp=0xc00014dfa0 sp=0xc00014def0 pc=0x7d5b69
github.com/valyala/fasthttp.(*workerPool).getCh.func1()
        /home/runner/go/pkg/mod/github.com/valyala/fasthttp@v1.47.0/workerpool.go:196 +0x38 fp=0xc00014dfe0 sp=0xc00014dfa0 pc=0x7d58d8
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00014dfe8 sp=0xc00014dfe0 pc=0x47c9c1
created by github.com/valyala/fasthttp.(*workerPool).getCh
        /home/runner/go/pkg/mod/github.com/valyala/fasthttp@v1.47.0/workerpool.go:195 +0x1b0

goroutine 1 [IO wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc0000f3288 sp=0xc0000f3268 pc=0x44dab6
runtime.netpollblock(0x7fb9272fc328?, 0x41920f?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:527 +0xf7 fp=0xc0000f32c0 sp=0xc0000f3288 pc=0x446417
internal/poll.runtime_pollWait(0x7fb927316b98, 0x72)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:306 +0x89 fp=0xc0000f32e0 sp=0xc0000f32c0 pc=0x477169
internal/poll.(*pollDesc).wait(0xc00017e380?, 0x4?, 0x0)
        /opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0000f3308 sp=0xc0000f32e0 pc=0x4ee0d2
internal/poll.(*pollDesc).waitRead(...)
        /opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc00017e380)
        /opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_unix.go:614 +0x2bd fp=0xc0000f33b0 sp=0xc0000f3308 pc=0x4f39dd
net.(*netFD).accept(0xc00017e380)
        /opt/hostedtoolcache/go/1.20.4/x64/src/net/fd_unix.go:172 +0x35 fp=0xc0000f3468 sp=0xc0000f33b0 pc=0x57a855
net.(*TCPListener).accept(0xc0001245d0)
        /opt/hostedtoolcache/go/1.20.4/x64/src/net/tcpsock_posix.go:148 +0x25 fp=0xc0000f3490 sp=0xc0000f3468 pc=0x590c05
net.(*TCPListener).Accept(0xc0001245d0)
        /opt/hostedtoolcache/go/1.20.4/x64/src/net/tcpsock.go:297 +0x3d fp=0xc0000f34c0 sp=0xc0000f3490 pc=0x58fcfd
github.com/valyala/fasthttp.acceptConn(0xc00040c000, {0x12f3da0, 0xc0001245d0}, 0xc0000f36b8)
        /home/runner/go/pkg/mod/github.com/valyala/fasthttp@v1.47.0/server.go:1930 +0x62 fp=0xc0000f35a0 sp=0xc0000f34c0 pc=0x7c8542
github.com/valyala/fasthttp.(*Server).Serve(0xc00040c000, {0x12f3da0?, 0xc0001245d0})
        /home/runner/go/pkg/mod/github.com/valyala/fasthttp@v1.47.0/server.go:1823 +0x4f4 fp=0xc0000f36e8 sp=0xc0000f35a0 pc=0x7c7b54
github.com/gofiber/fiber/v2.(*App).Listen(0xc000163200, {0xde34aa?, 0x7?})
        /home/runner/go/pkg/mod/github.com/gofiber/fiber/v2@v2.46.0/listen.go:82 +0x110 fp=0xc0000f3748 sp=0xc0000f36e8 pc=0x85d890
main.main.func1(0xc0003ba160?)
        /home/runner/work/LocalAI/LocalAI/main.go:140 +0x6c6 fp=0xc0000f3990 sp=0xc0000f3748 pc=0x953366
github.com/urfave/cli/v2.(*Command).Run(0xc0003ba160, 0xc000164900, {0xc000118200, 0x1, 0x1})
        /home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.25.5/command.go:274 +0x9eb fp=0xc0000f3c30 sp=0xc0000f3990 pc=0x94084b
github.com/urfave/cli/v2.(*App).RunContext(0xc0003b6000, {0x12f4108?, 0xc00012a000}, {0xc000118200, 0x1, 0x1})
        /home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.25.5/app.go:332 +0x616 fp=0xc0000f3ca0 sp=0xc0000f3c30 pc=0x93d176
github.com/urfave/cli/v2.(*App).Run(...)
        /home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.25.5/app.go:309
main.main()
        /home/runner/work/LocalAI/LocalAI/main.go:144 +0xff6 fp=0xc0000f3f80 sp=0xc0000f3ca0 pc=0x952bd6
runtime.main()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:250 +0x207 fp=0xc0000f3fe0 sp=0xc0000f3f80 pc=0x44d687
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000f3fe8 sp=0xc0000f3fe0 pc=0x47c9c1

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000094fb0 sp=0xc000094f90 pc=0x44dab6
runtime.goparkunlock(...)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:387
runtime.forcegchelper()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:305 +0xb0 fp=0xc000094fe0 sp=0xc000094fb0 pc=0x44d8f0
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000094fe8 sp=0xc000094fe0 pc=0x47c9c1
created by runtime.init.6
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:293 +0x25

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000095780 sp=0xc000095760 pc=0x44dab6
runtime.goparkunlock(...)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:387
runtime.bgsweep(0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgcsweep.go:319 +0xde fp=0xc0000957c8 sp=0xc000095780 pc=0x439d1e
runtime.gcenable.func1()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:178 +0x26 fp=0xc0000957e0 sp=0xc0000957c8 pc=0x42ef86
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000957e8 sp=0xc0000957e0 pc=0x47c9c1
created by runtime.gcenable
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:178 +0x6b

goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc0000ba000?, 0x12ec2d8?, 0x0?, 0x0?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000095f70 sp=0xc000095f50 pc=0x44dab6
runtime.goparkunlock(...)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:387
runtime.(*scavengerState).park(0x1796100)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgcscavenge.go:400 +0x53 fp=0xc000095fa0 sp=0xc000095f70 pc=0x437bf3
runtime.bgscavenge(0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgcscavenge.go:633 +0x65 fp=0xc000095fc8 sp=0xc000095fa0 pc=0x4381e5
runtime.gcenable.func2()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:179 +0x26 fp=0xc000095fe0 sp=0xc000095fc8 pc=0x42ef26
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000095fe8 sp=0xc000095fe0 pc=0x47c9c1
created by runtime.gcenable
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:179 +0xaa

goroutine 18 [finalizer wait]:
runtime.gopark(0x1a0?, 0x1796de0?, 0xe0?, 0x24?, 0xc000094770?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000094628 sp=0xc000094608 pc=0x44dab6
runtime.runfinq()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mfinal.go:193 +0x107 fp=0xc0000947e0 sp=0xc000094628 pc=0x42dfc7
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000947e8 sp=0xc0000947e0 pc=0x47c9c1
created by runtime.createfing
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mfinal.go:163 +0x45

goroutine 19 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000090750 sp=0xc000090730 pc=0x44dab6
runtime.gcBgMarkWorker()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000907e0 sp=0xc000090750 pc=0x430cf1
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000907e8 sp=0xc0000907e0 pc=0x47c9c1
created by runtime.gcBgMarkStartWorkers
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1199 +0x25

goroutine 5 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000096750 sp=0xc000096730 pc=0x44dab6
runtime.gcBgMarkWorker()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000967e0 sp=0xc000096750 pc=0x430cf1
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000967e8 sp=0xc0000967e0 pc=0x47c9c1
created by runtime.gcBgMarkStartWorkers
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1199 +0x25

goroutine 20 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000090f50 sp=0xc000090f30 pc=0x44dab6
runtime.gcBgMarkWorker()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc000090fe0 sp=0xc000090f50 pc=0x430cf1
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000090fe8 sp=0xc000090fe0 pc=0x47c9c1
created by runtime.gcBgMarkStartWorkers
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1199 +0x25

goroutine 6 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000096f50 sp=0xc000096f30 pc=0x44dab6
runtime.gcBgMarkWorker()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc000096fe0 sp=0xc000096f50 pc=0x430cf1
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000096fe8 sp=0xc000096fe0 pc=0x47c9c1
created by runtime.gcBgMarkStartWorkers
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1199 +0x25

goroutine 34 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000504750 sp=0xc000504730 pc=0x44dab6
runtime.gcBgMarkWorker()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc0005047e0 sp=0xc000504750 pc=0x430cf1
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0005047e8 sp=0xc0005047e0 pc=0x47c9c1
created by runtime.gcBgMarkStartWorkers
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1199 +0x25

goroutine 50 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000500750 sp=0xc000500730 pc=0x44dab6
runtime.gcBgMarkWorker()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc0005007e0 sp=0xc000500750 pc=0x430cf1
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0005007e8 sp=0xc0005007e0 pc=0x47c9c1
created by runtime.gcBgMarkStartWorkers
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1199 +0x25

goroutine 35 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000504f50 sp=0xc000504f30 pc=0x44dab6
runtime.gcBgMarkWorker()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc000504fe0 sp=0xc000504f50 pc=0x430cf1
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000504fe8 sp=0xc000504fe0 pc=0x47c9c1
created by runtime.gcBgMarkStartWorkers
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1199 +0x25

goroutine 7 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000097750 sp=0xc000097730 pc=0x44dab6
runtime.gcBgMarkWorker()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000977e0 sp=0xc000097750 pc=0x430cf1
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000977e8 sp=0xc0000977e0 pc=0x47c9c1
created by runtime.gcBgMarkStartWorkers
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1199 +0x25

goroutine 21 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000091750 sp=0xc000091730 pc=0x44dab6
runtime.gcBgMarkWorker()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000917e0 sp=0xc000091750 pc=0x430cf1
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000917e8 sp=0xc0000917e0 pc=0x47c9c1
created by runtime.gcBgMarkStartWorkers
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1199 +0x25

goroutine 22 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000091f50 sp=0xc000091f30 pc=0x44dab6
runtime.gcBgMarkWorker()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc000091fe0 sp=0xc000091f50 pc=0x430cf1
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000091fe8 sp=0xc000091fe0 pc=0x47c9c1
created by runtime.gcBgMarkStartWorkers
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1199 +0x25

goroutine 8 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000097f50 sp=0xc000097f30 pc=0x44dab6
runtime.gcBgMarkWorker()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc000097fe0 sp=0xc000097f50 pc=0x430cf1
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000097fe8 sp=0xc000097fe0 pc=0x47c9c1
created by runtime.gcBgMarkStartWorkers
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1199 +0x25

goroutine 23 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000092750 sp=0xc000092730 pc=0x44dab6
runtime.gcBgMarkWorker()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000927e0 sp=0xc000092750 pc=0x430cf1
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000927e8 sp=0xc0000927e0 pc=0x47c9c1
created by runtime.gcBgMarkStartWorkers
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1199 +0x25

goroutine 9 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc0000ce750 sp=0xc0000ce730 pc=0x44dab6
runtime.gcBgMarkWorker()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000ce7e0 sp=0xc0000ce750 pc=0x430cf1
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000ce7e8 sp=0xc0000ce7e0 pc=0x47c9c1
created by runtime.gcBgMarkStartWorkers
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1199 +0x25

goroutine 24 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000092f50 sp=0xc000092f30 pc=0x44dab6
runtime.gcBgMarkWorker()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc000092fe0 sp=0xc000092f50 pc=0x430cf1
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000092fe8 sp=0xc000092fe0 pc=0x47c9c1
created by runtime.gcBgMarkStartWorkers
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1199 +0x25

goroutine 51 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000500f50 sp=0xc000500f30 pc=0x44dab6
runtime.gcBgMarkWorker()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc000500fe0 sp=0xc000500f50 pc=0x430cf1
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000500fe8 sp=0xc000500fe0 pc=0x47c9c1
created by runtime.gcBgMarkStartWorkers
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1199 +0x25

goroutine 36 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000505750 sp=0xc000505730 pc=0x44dab6
runtime.gcBgMarkWorker()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc0005057e0 sp=0xc000505750 pc=0x430cf1
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0005057e8 sp=0xc0005057e0 pc=0x47c9c1
created by runtime.gcBgMarkStartWorkers
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1199 +0x25

goroutine 37 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000505f50 sp=0xc000505f30 pc=0x44dab6
runtime.gcBgMarkWorker()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc000505fe0 sp=0xc000505f50 pc=0x430cf1
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000505fe8 sp=0xc000505fe0 pc=0x47c9c1
created by runtime.gcBgMarkStartWorkers
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1199 +0x25

goroutine 25 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000093750 sp=0xc000093730 pc=0x44dab6
runtime.gcBgMarkWorker()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000937e0 sp=0xc000093750 pc=0x430cf1
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000937e8 sp=0xc0000937e0 pc=0x47c9c1
created by runtime.gcBgMarkStartWorkers
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1199 +0x25

goroutine 38 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000506750 sp=0xc000506730 pc=0x44dab6
runtime.gcBgMarkWorker()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc0005067e0 sp=0xc000506750 pc=0x430cf1
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0005067e8 sp=0xc0005067e0 pc=0x47c9c1
created by runtime.gcBgMarkStartWorkers
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1199 +0x25

goroutine 26 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000093f50 sp=0xc000093f30 pc=0x44dab6
runtime.gcBgMarkWorker()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc000093fe0 sp=0xc000093f50 pc=0x430cf1
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000093fe8 sp=0xc000093fe0 pc=0x47c9c1
created by runtime.gcBgMarkStartWorkers
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1199 +0x25

goroutine 52 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000501750 sp=0xc000501730 pc=0x44dab6
runtime.gcBgMarkWorker()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc0005017e0 sp=0xc000501750 pc=0x430cf1
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0005017e8 sp=0xc0005017e0 pc=0x47c9c1
created by runtime.gcBgMarkStartWorkers
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1199 +0x25

goroutine 27 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc0000ca750 sp=0xc0000ca730 pc=0x44dab6
runtime.gcBgMarkWorker()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000ca7e0 sp=0xc0000ca750 pc=0x430cf1
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000ca7e8 sp=0xc0000ca7e0 pc=0x47c9c1
created by runtime.gcBgMarkStartWorkers
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1199 +0x25

goroutine 39 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000506f50 sp=0xc000506f30 pc=0x44dab6
runtime.gcBgMarkWorker()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc000506fe0 sp=0xc000506f50 pc=0x430cf1
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000506fe8 sp=0xc000506fe0 pc=0x47c9c1
created by runtime.gcBgMarkStartWorkers
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1199 +0x25

goroutine 28 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc0000caf50 sp=0xc0000caf30 pc=0x44dab6
runtime.gcBgMarkWorker()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000cafe0 sp=0xc0000caf50 pc=0x430cf1
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000cafe8 sp=0xc0000cafe0 pc=0x47c9c1
created by runtime.gcBgMarkStartWorkers
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1199 +0x25

goroutine 53 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000501f50 sp=0xc000501f30 pc=0x44dab6
runtime.gcBgMarkWorker()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc000501fe0 sp=0xc000501f50 pc=0x430cf1
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000501fe8 sp=0xc000501fe0 pc=0x47c9c1
created by runtime.gcBgMarkStartWorkers
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1199 +0x25

goroutine 54 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000502750 sp=0xc000502730 pc=0x44dab6
runtime.gcBgMarkWorker()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc0005027e0 sp=0xc000502750 pc=0x430cf1
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0005027e8 sp=0xc0005027e0 pc=0x47c9c1
created by runtime.gcBgMarkStartWorkers
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1199 +0x25

goroutine 55 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000502f50 sp=0xc000502f30 pc=0x44dab6
runtime.gcBgMarkWorker()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc000502fe0 sp=0xc000502f50 pc=0x430cf1
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000502fe8 sp=0xc000502fe0 pc=0x47c9c1
created by runtime.gcBgMarkStartWorkers
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1199 +0x25

goroutine 10 [GC worker (idle)]:
runtime.gopark(0x1b98f00?, 0x1?, 0xca?, 0x78?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc0000cef50 sp=0xc0000cef30 pc=0x44dab6
runtime.gcBgMarkWorker()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000cefe0 sp=0xc0000cef50 pc=0x430cf1
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000cefe8 sp=0xc0000cefe0 pc=0x47c9c1
created by runtime.gcBgMarkStartWorkers
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1199 +0x25

goroutine 11 [GC worker (idle)]:
runtime.gopark(0x142b0171f63f43?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc0000cf750 sp=0xc0000cf730 pc=0x44dab6
runtime.gcBgMarkWorker()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000cf7e0 sp=0xc0000cf750 pc=0x430cf1
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000cf7e8 sp=0xc0000cf7e0 pc=0x47c9c1
created by runtime.gcBgMarkStartWorkers
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1199 +0x25

goroutine 40 [GC worker (idle)]:
runtime.gopark(0x142b01726497b9?, 0x1?, 0xb8?, 0x81?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000507750 sp=0xc000507730 pc=0x44dab6
runtime.gcBgMarkWorker()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc0005077e0 sp=0xc000507750 pc=0x430cf1
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0005077e8 sp=0xc0005077e0 pc=0x47c9c1
created by runtime.gcBgMarkStartWorkers
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1199 +0x25

goroutine 12 [GC worker (idle)]:
runtime.gopark(0x142b017262532c?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc0000cff50 sp=0xc0000cff30 pc=0x44dab6
runtime.gcBgMarkWorker()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000cffe0 sp=0xc0000cff50 pc=0x430cf1
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000cffe8 sp=0xc0000cffe0 pc=0x47c9c1
created by runtime.gcBgMarkStartWorkers
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1199 +0x25

goroutine 41 [GC worker (idle)]:
runtime.gopark(0x142b017264e87a?, 0x3?, 0x9e?, 0xd2?, 0x0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000507f50 sp=0xc000507f30 pc=0x44dab6
runtime.gcBgMarkWorker()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc000507fe0 sp=0xc000507f50 pc=0x430cf1
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000507fe8 sp=0xc000507fe0 pc=0x47c9c1
created by runtime.gcBgMarkStartWorkers
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1199 +0x25

goroutine 29 [select]:
runtime.gopark(0xc0000cb710?, 0x2?, 0x0?, 0x0?, 0xc0000cb6ec?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc0000a0d40 sp=0xc0000a0d20 pc=0x44dab6
runtime.selectgo(0xc0000a0f10, 0xc0000cb6e8, 0x0?, 0x0, 0x0?, 0x1)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/select.go:327 +0x7be fp=0xc0000a0e80 sp=0xc0000a0d40 pc=0x45d29e
github.com/go-skynet/LocalAI/api.(*galleryApplier).start.func1()
        /home/runner/work/LocalAI/LocalAI/api/gallery.go:98 +0xe6 fp=0xc0000a0fe0 sp=0xc0000a0e80 pc=0x917d66
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000a0fe8 sp=0xc0000a0fe0 pc=0x47c9c1
created by github.com/go-skynet/LocalAI/api.(*galleryApplier).start
        /home/runner/work/LocalAI/LocalAI/api/gallery.go:96 +0xaa

goroutine 30 [sleep]:
runtime.gopark(0x142b0ac2ce3170?, 0xc0000cbf88?, 0x85?, 0xce?, 0xc000144ad0?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc0000cbf58 sp=0xc0000cbf38 pc=0x44dab6
time.Sleep(0x2540be400)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/time.go:195 +0x135 fp=0xc0000cbf98 sp=0xc0000cbf58 pc=0x479835
github.com/valyala/fasthttp.(*workerPool).Start.func2()
        /home/runner/go/pkg/mod/github.com/valyala/fasthttp@v1.47.0/workerpool.go:67 +0x56 fp=0xc0000cbfe0 sp=0xc0000cbf98 pc=0x7d5036
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000cbfe8 sp=0xc0000cbfe0 pc=0x47c9c1
created by github.com/valyala/fasthttp.(*workerPool).Start
        /home/runner/go/pkg/mod/github.com/valyala/fasthttp@v1.47.0/workerpool.go:59 +0xdd

goroutine 57 [sleep]:
runtime.gopark(0x142b0b08ecef63?, 0xd35780?, 0x10?, 0x4b?, 0xc11bcf9b5a928abb?)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc0000a1f88 sp=0xc0000a1f68 pc=0x44dab6
time.Sleep(0x3b9aca00)
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/time.go:195 +0x135 fp=0xc0000a1fc8 sp=0xc0000a1f88 pc=0x479835
github.com/valyala/fasthttp.updateServerDate.func1()
        /home/runner/go/pkg/mod/github.com/valyala/fasthttp@v1.47.0/header.go:2247 +0x1e fp=0xc0000a1fe0 sp=0xc0000a1fc8 pc=0x7d5fbe
runtime.goexit()
        /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000a1fe8 sp=0xc0000a1fe0 pc=0x47c9c1
created by github.com/valyala/fasthttp.updateServerDate
        /home/runner/go/pkg/mod/github.com/valyala/fasthttp@v1.47.0/header.go:2245 +0x25
root@autodl-container-a4da118bfa-64e0dc0a:~/autodl-tmp/LocalAI# 

[yunghoy]

Have the same issue with ggml-gpt4all-j.bin and

• https://quay.io/go-skynet/local-ai:master-cublas-cuda11
• https://quay.io/go-skynet/local-ai:master-cublas-cuda12
• https://quay.io/go-skynet/local-ai:master-cublas-cuda11-ffmpeg

• https://quay.io/go-skynet/local-ai:master-cublas-cuda12-ffmpeg

  2023-06-22 01:22:51 8:22AM DBG Loading model 'ggml-gpt4all-j.bin' greedly 2023-06-22 01:22:51 8:22AM DBG [llama] Attempting to load 2023-06-22 01:22:51 8:22AM DBG Loading model llama from ggml-gpt4all-j.bin 2023-06-22 01:22:51 8:22AM DBG Loading model in memory from file: /models/ggml-gpt4all-j.bin 2023-06-22 01:22:51 ggml_init_cublas: found 1 CUDA devices: 2023-06-22 01:22:51 Device 0: NVIDIA GeForce RTX 4090 2023-06-22 01:22:52 llama.cpp: loading model from /models/ggml-gpt4all-j.bin 2023-06-22 01:22:56 error loading model: unexpectedly reached end of file 2023-06-22 01:22:56 llama_init_from_file: failed to load model 2023-06-22 01:22:56 fatal error: unexpected signal during runtime execution 2023-06-22 01:22:56 [signal SIGSEGV: segmentation violation code=0x1 addr=0x4 pc=0xa82190]

[yunghoy]

Found the reason candidate. LocalAI tries to load model with different bindings. Once it fails it moves to the next binding candidate. However, this method cannot be applied with CUDA because CUDA throws Segment Fault. This terminates Docker container as SIG* kernel command was generated from the application as designed in Docker.

[guyguy333]

Having the same issue. @yunghoy did you find a workaround ?

[localai-bot]

:warning::warning::warning::warning::warning:

Hi! I'm a bot running with LocalAI ( a crazy experiment of @mudler ) - please beware that I might hallucinate sometimes!

_but.... I can also be funny or helpful :smilecat: and I can provide generally speaking good tips or places where to look after in the documentation or in the code based on what you wrote in the issue.

Don't engage in conversation with me, I don't support (yet) replying!

:warning::warning::warning::warning::warning:

The issue seems to be related to a segmentation fault in the LocalAI application. It could be caused by a number of reasons such as a programming error, memory allocation issues or a hardware failure. To help debug the issue further, you can try to reproduce the problem and gather more information about the environment and the code that is being executed at the time of the crash. You can also check the logs for any error messages or stack traces that may provide additional clues. Additionally, you can try to reproduce the issue on a different machine or platform to rule out any hardware-related issues.

Sources:

• https://localai.io/basics/build/index.html
• https://localai.io/basics/news/index.html
• https://localai.io/basics/getting_started/index.html