JamborJan
commented
6 months ago I am using nextcloud all in one and the Local AI container. I used the basic example there for the models:
# Stable Diffusion in NCNN with c++, supported txt2img and img2img
- url: github:go-skynet/model-gallery/stablediffusion.yaml
# Port of OpenAI's Whisper model in C/C++
- url: github:go-skynet/model-gallery/whisper-base.yaml
name: whisper-1
# A commercially licensable model based on GPT-J and trained by Nomic AI on the v0 GPT4All dataset.
- url: github:go-skynet/model-gallery/gpt4all-j.yaml
name: gpt4all-jThe container is up and running with these logs:
CPU info:
model name : Common KVM processor
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm rep_good nopl cpuid extd_apicid tsc_known_freq pni cx16 x2apic hypervisor cmp_legacy 3dnowprefetch vmmcall
CPU: no AVX found
CPU: no AVX2 found
CPU: no AVX512 found
nc: getaddrinfo for host "nextcloud-aio-nextcloud" port 9001: Name or service not known
Waiting for nextcloud to start
nc: getaddrinfo for host "nextcloud-aio-nextcloud" port 9001: Name or service not known
Waiting for nextcloud to start
nc: getaddrinfo for host "nextcloud-aio-nextcloud" port 9001: Name or service not known
Waiting for nextcloud to start
nc: getaddrinfo for host "nextcloud-aio-nextcloud" port 9001: Name or service not known
Waiting for nextcloud to start
nc: getaddrinfo for host "nextcloud-aio-nextcloud" port 9001: Name or service not known
Waiting for nextcloud to start
Waiting for nextcloud to start
Waiting for nextcloud to start
Waiting for nextcloud to start
Waiting for nextcloud to start
Waiting for nextcloud to start
Waiting for nextcloud to start
++ nproc
+ THREADS=16
+ export THREADS
+ set +x
10:55AM INF Starting LocalAI using 16 threads, with models path: /models
10:55AM INF LocalAI version: v1.30.0 (274ace289823a8bacb7b4987b5c961b62d5eee99)
┌───────────────────────────────────────────────────┐
│ Fiber v2.49.2 │
│ http://127.0.0.1:8080 │
│ (bound on host 0.0.0.0 and port 8080) │
│ │
│ Handlers ............ 70 Processes ........... 1 │
│ Prefork ....... Disabled PID ................ 39 │
└───────────────────────────────────────────────────┘When I run the test mentioned earlier in the threads:
LOCALAI=http://localhost:8080
curl $LOCALAI/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "gpt4all-j",
"messages": [{"role": "user", "content": "How are you?"}],
"temperature": 2
}'The command produces this error:
{"error":{"code":500,"message":"could not load model: rpc error: code = Unknown desc = failed loading model","type":""}}I get this error always when a request is either send via nextcloud ai assistant or the test curl command:
rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:42737: connect: connection refused"
rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:41939: connect: connection refused"
rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:45029: connect: connection refused"
rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:32875: connect: connection refused"
rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:32875: connect: connection refused"
rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:32875: connect: connection refused"
iammaguire
kilmarnock
SicLuceatLux
Punkado
LocalAI version:
V1.21 root@63429046747f:/build# ./local-ai --version LocalAI version 4548473 (4548473acf4f57ff149492272cc1fdba3521f83a) llmai-api-1 | 3:04AM DBG Loading model '
Environment, CPU architecture, OS, and Version: Intel X86 centos
Describe the bug Grpc error
To Reproduce
Expected behavior Output response
Logs openllama7b' greedly llmai-api-1 | 3:04AM DBG [llama] Attempting to load llmai-api-1 | 3:04AM DBG Loading model llama from openllama7b llmai-api-1 | 3:04AM DBG Loading model in memory from file: /models/openllama7b llmai-api-1 | 3:04AM DBG Loading GRPC Model%!(EXTRA string=llama, model.Options={llama openllama7b 4 /tmp/localai/backend_data 0xc0000400b0 0xc000296a20}) llmai-api-1 | 3:04AM DBG Loading GRPC Process%!(EXTRA string=/tmp/localai/backend_data/backend-assets/grpc/llama) llmai-api-1 | 3:04AM DBG GRPC Service for 'llama' (openllama7b) will be running at: 'localhost:37785' llmai-api-1 | 3:04AM DBG GRPC Service Started llmai-api-1 | rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1">127.0.0.1:37785: connect: connection refused" llmai-api-1 | 3:04AM DBG GRPC(llama-openllama7b-localhost:37785): stderr 2023/07/19 03:04:00 gRPC Server listening at 127.0.0.1:37785 llmai-api-1 | 3:04AM DBG GRPC Service Ready llmai-api-1 | 3:04AM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:} sizeCache:0 unknownFields:[] Model:/models/openllama7b ContextSize:512 Seed:0 NBatch:512 F16Memory:false MLock:false MMap:false VocabOnly:false LowVRAM:false Embeddings:false NUMA:false NGPULayers:0 MainGPU: TensorSplit: Threads:4 LibrarySearchPath:}
Additional context