search

mudler / LocalAI

:robot: The free, Open Source alternative to OpenAI, Claude and others. Self-hosted and local-first. Drop-in replacement for OpenAI, running on consumer-grade hardware. No GPU required. Runs gguf, transformers, diffusers and many more models architectures. Features: Generate Text, Audio, Video, Images, Voice Cloning, Distributed, P2P inference
https://localai.io
MIT License
26.16k stars 1.96k forks source link

Can't build LocalAI with llama.cpp with CUDA #3418

Open dimazig opened 2 months ago

dimazig commented 2 months ago 
dima@dima-Dell-G15-Special-Edition-5521:~/IdeaProjects/LocalAI$ make BUILD_TYPE=cublas BUILD_GRPC_FOR_BACKEND_LLAMA=ON CUDA_DOCKER_ARCH=all GRPC_BACKENDS=backend-assets/grpc/llama-cpp build 
go mod edit -replace github.com/donomii/go-rwkv.cpp=/home/dima/IdeaProjects/LocalAI/sources/go-rwkv.cpp
go mod edit -replace github.com/ggerganov/whisper.cpp=/home/dima/IdeaProjects/LocalAI/sources/whisper.cpp
go mod edit -replace github.com/ggerganov/whisper.cpp/bindings/go=/home/dima/IdeaProjects/LocalAI/sources/whisper.cpp/bindings/go
go mod edit -replace github.com/go-skynet/go-bert.cpp=/home/dima/IdeaProjects/LocalAI/sources/go-bert.cpp
go mod edit -replace github.com/M0Rf30/go-tiny-dream=/home/dima/IdeaProjects/LocalAI/sources/go-tiny-dream
go mod edit -replace github.com/mudler/go-piper=/home/dima/IdeaProjects/LocalAI/sources/go-piper
go mod edit -replace github.com/mudler/go-stable-diffusion=/home/dima/IdeaProjects/LocalAI/sources/go-stable-diffusion
go mod edit -replace github.com/go-skynet/go-llama.cpp=/home/dima/IdeaProjects/LocalAI/sources/go-llama.cpp
go mod download
mkdir -p pkg/grpc/proto
protoc --experimental_allow_proto3_optional -Ibackend/ --go_out=pkg/grpc/proto/ --go_opt=paths=source_relative --go-grpc_out=pkg/grpc/proto/ --go-grpc_opt=paths=source_relative \
    backend/backend.proto
mkdir -p backend-assets/grpc
I local-ai build info:
I BUILD_TYPE: cublas
I GO_TAGS: 
I LD_FLAGS: -s -w -X "github.com/mudler/LocalAI/internal.Version=v2.20.1-27-g8369614b" -X "github.com/mudler/LocalAI/internal.Commit=8369614b6ee2994a650da62d53db0409c4faaf4f"
I UPX: 
CGO_LDFLAGS="-lcublas -lcudart -L/usr/local/cuda/lib64/" go build -ldflags "-s -w -X "github.com/mudler/LocalAI/internal.Version=v2.20.1-27-g8369614b" -X "github.com/mudler/LocalAI/internal.Commit=8369614b6ee2994a650da62d53db0409c4faaf4f"" -tags "" -o local-ai ./
# github.com/mudler/LocalAI
too much data, last section SGOSTRING (2418399720, over 2e+09 bytes)
too much data, last section SGOFUNC (2420896144, over 2e+09 bytes)
too much data, last section SGCBITS (2420919432, over 2e+09 bytes)
too much data, last section SRODATA (2421594793, over 2e+09 bytes)
too much data, last section SFUNCTAB (2421594793, over 2e+09 bytes)
too much data, last section SELFROSECT (2421598264, over 2e+09 bytes)
too much data, last section STYPELINK (2421659600, over 2e+09 bytes)
too much data, last section SITABLINK (2421683624, over 2e+09 bytes)
.plt: pc-relative relocation address for .got.plt is too big: 0x911e50c2
.plt: pc-relative relocation address for .got.plt is too big: 0x911e50c4
.plt: pc-relative relocation address for .got.plt is too big: 0x911e50c2
.plt: pc-relative relocation address for .got.plt is too big: 0x911e50ba
.plt: pc-relative relocation address for .got.plt is too big: 0x911e50b2
.plt: pc-relative relocation address for .got.plt is too big: 0x911e50aa
.plt: pc-relative relocation address for .got.plt is too big: 0x911e50a2
.plt: pc-relative relocation address for .got.plt is too big: 0x911e509a
.plt: pc-relative relocation address for .got.plt is too big: 0x911e5092
.plt: pc-relative relocation address for .got.plt is too big: 0x911e508a
.plt: pc-relative relocation address for .got.plt is too big: 0x911e5082
.plt: pc-relative relocation address for .got.plt is too big: 0x911e507a
.plt: pc-relative relocation address for .got.plt is too big: 0x911e5072
/home/dima/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/pkg/tool/linux_amd64/link: too many errors
make: *** [Makefile:369: build] Error 1
dima@dima-Dell-G15-Special-Edition-5521:~/IdeaProjects/LocalAI$
dimazig commented 2 months ago

with make GRPC_BACKENDS=backend-assets/grpc/llama-cpp build

I'm getting the same error. There is something wrong with Go lang code

dimazig commented 2 months ago

tried to upgrade Go to 1.23 but getting the same error

CGO_LDFLAGS="-lcublas -lcudart -L/usr/local/cuda/lib64/" go build -ldflags "-s -w -X "github.com/mudler/LocalAI/internal.Version=v2.20.1-30-gbc684c25" -X "github.com/mudler/LocalAI/internal.Commit=bc684c259c5afb73e64bc6c145e2aa261562a468"" -tags "" -o local-ai ./
# github.com/mudler/LocalAI
too much data, last section SGOSTRING (2418459656, over 2e+09 bytes)
too much data, last section SGOFUNC (2420451420, over 2e+09 bytes)
too much data, last section SGCBITS (2420474736, over 2e+09 bytes)
too much data, last section SRODATA (2421160431, over 2e+09 bytes)
too much data, last section SFUNCTAB (2421160431, over 2e+09 bytes)
too much data, last section SELFROSECT (2421163896, over 2e+09 bytes)
too much data, last section STYPELINK (2421226020, over 2e+09 bytes)
too much data, last section SITABLINK (2421250096, over 2e+09 bytes)
.plt: pc-relative relocation address for .got.plt is too big: 0x911986e2
.plt: pc-relative relocation address for .got.plt is too big: 0x911986e4
.plt: pc-relative relocation address for .got.plt is too big: 0x911986e2
.plt: pc-relative relocation address for .got.plt is too big: 0x911986da
.plt: pc-relative relocation address for .got.plt is too big: 0x911986d2
.plt: pc-relative relocation address for .got.plt is too big: 0x911986ca
.plt: pc-relative relocation address for .got.plt is too big: 0x911986c2
.plt: pc-relative relocation address for .got.plt is too big: 0x911986ba
.plt: pc-relative relocation address for .got.plt is too big: 0x911986b2
.plt: pc-relative relocation address for .got.plt is too big: 0x911986aa
.plt: pc-relative relocation address for .got.plt is too big: 0x911986a2
.plt: pc-relative relocation address for .got.plt is too big: 0x9119869a
.plt: pc-relative relocation address for .got.plt is too big: 0x91198692
/usr/lib/go-1.23/pkg/tool/linux_amd64/link: too many errors
make: *** [Makefile:369: build] Error 1
dimazig commented 2 months ago

something's exceeding 2Gb limit

https://go-review.googlesource.com/c/go/+/11715/6/src/cmd/link/internal/ld/data.go

dimazig commented 2 months ago

If I use --trace then it shows a little more

too much data, last section SGOSTRING (2418459656, over 2e+09 bytes)
too much data, last section SGOFUNC (2420451420, over 2e+09 bytes)
too much data, last section SGCBITS (2420474736, over 2e+09 bytes)
too much data, last section SRODATA (2421160431, over 2e+09 bytes)
too much data, last section SFUNCTAB (2421160431, over 2e+09 bytes)
too much data, last section SELFROSECT (2421163896, over 2e+09 bytes)
too much data, last section STYPELINK (2421226020, over 2e+09 bytes)
too much data, last section SITABLINK (2421250096, over 2e+09 bytes)
reflect.TypeOf: pc-relative relocation address for go:itab.*reflect.rtype,reflect.Type is too big: 0x91735244
reflect.PtrTo: pc-relative relocation address for go:itab.*reflect.rtype,reflect.Type is too big: 0x91735211
reflect.PtrTo: pc-relative relocation address for go:itab.*reflect.rtype,reflect.Type is too big: 0x917351f9
reflect.PtrTo: pc-relative relocation address for go:itab.*reflect.rtype,reflect.Type is too big: 0x917351f4
reflect.implements: pc-relative relocation address for reflect.implements.jump53 is too big: 0x917128a7
reflect.MapOf: pc-relative relocation address for reflect.lookupCache is too big: 0x9259a69d
reflect.MapOf: pc-relative relocation address for reflect..typeAssert.2 is too big: 0x9254677c
reflect.MapOf: pc-relative relocation address for net..stmp_57 is too big: 0x916eddd1
reflect.MapOf: pc-relative relocation address for reflect.lookupCache is too big: 0x9259a56e
reflect.MapOf: pc-relative relocation address for reflect..typeAssert.3 is too big: 0x92546670
reflect.MapOf: pc-relative relocation address for runtime.writeBarrier is too big: 0x925c6962
reflect.MapOf: pc-relative relocation address for runtime.writeBarrier is too big: 0x925c6861
reflect.MapOf: pc-relative relocation address for runtime.writeBarrier is too big: 0x925c682e
/usr/lib/go-1.23/pkg/tool/linux_amd64/link: too many errors
make: *** [Makefile:369: build] Error 1
aderbalbotelho commented 1 month ago

I have the same error with cuda 12.6 and go1.23.1 on debian 12, nvidia driver 560.35.03-1

``go build -ldflags "-s -w -X "github.com/mudler/LocalAI/internal.Version=v2.22.0-88-g32db7879" -X "github.com/mudler/LocalAI/internal.Commit=32db7879915a3910671872ac9e5245e44e9e032e"" -tags "stablediffusion,tts,p2p" -o backend-assets/grpc/piper ./backend/go/tts/
I local-ai build info:
I BUILD_TYPE: cublas
I GO_TAGS: stablediffusion,tts,p2p
I LD_FLAGS: -s -w -X "github.com/mudler/LocalAI/internal.Version=v2.22.0-88-g32db7879" -X "github.com/mudler/LocalAI/internal.Commit=32db7879915a3910671872ac9e5245e44e9e032e"
I UPX: 
CGO_LDFLAGS="-lcublas -lcudart -L/usr/local/cuda/lib64/" go build -ldflags "-s -w -X "github.com/mudler/LocalAI/internal.Version=v2.22.0-88-g32db7879" -X "github.com/mudler/LocalAI/internal.Commit=32db7879915a3910671872ac9e5245e44e9e032e"" -tags "stablediffusion,tts,p2p" -o local-ai ./
# github.com/mudler/LocalAI
too much data, last section SGOSTRING (2091635592, over 2e+09 bytes)
too much data, last section SGOFUNC (2093749086, over 2e+09 bytes)
too much data, last section SGCBITS (2093773096, over 2e+09 bytes)
too much data, last section SRODATA (2094720599, over 2e+09 bytes)
too much data, last section SFUNCTAB (2094720599, over 2e+09 bytes)
too much data, last section SELFROSECT (2094724056, over 2e+09 bytes)
too much data, last section STYPELINK (2094790428, over 2e+09 bytes)
too much data, last section SITABLINK (2094816592, over 2e+09 bytes)
make: *** [Makefile:372: build] Error 1

``

aderbalbotelho commented 1 month ago

My parameters in ulimit

`# ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) 0
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) unlimited
max locked memory           (kbytes, -l) unlimited
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1024000
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 67108864
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 513870
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited
`
wimdeherdt commented 1 week ago

Same problem here. Anyone find a solution for this?

buergi commented 1 week ago

Same for me with CUDA 11.4 and driver 470.239.06.