Closed e576082c closed 10 months ago
It seems like the build is working. I don't see any build errors, your setup just doesn't support customtkinter? Are you saying that the customtkinter GUI worked for you previously but not anymore?
What seems to be the problem?
The "requirements.txt" file does certainly have a missing "packaging" dependency. (Yeah, the name of the missing dependency is "packaging" lol). But I think this is unrelated to the problem.
After I succeed building with make LLAMA_OPENBLAS=1 LLAMA_CLBLAST=1 LLAMA_CUBLAS=1
, and failing to first run, I then always manually install "packaging" into my venv, and then the GUI starts up normally.
The problem is with missing AVX2 support in new versions. At least the GUI says so. Could it be, that this is only a problem with the GUI reporting "NoAVX2 Mode (Old CPU), Failsafe Mode (Old CPU)", but under the hood, AVX2 Mode is actually ON and working? Please look at what I see.
Before:
After:
Ah okay, what you're seeing is actually not a bug but a feature - I should probably make the tooltip clearer. When building in Linux, koboldcpp applies the -march=native -mtune=native
flags. These cause the compiler to target the architecture you're currently building on, rather than a specific intrinsic flag. Thus, there's no point in building the non-avx2 versions - this has always been the case, and if you grabbed the *.so files from an older version like 1.44 you'd find the avx2 and non-avx2.so files are similarly functional. So an optimization was made to remove them.
Thanks, now I get it. This was only a big misunderstanding, because I have not done any performance tests, just blindly believed what the GUI told me.
I checked it out, and the new versions are not slower than version 1.44.2, so it's probably true, that AVX2 support is ON, and only the GUI was deterring me from using the newer versions.
The new versions, if not absolutely necessary, do not build with old, vintage CPU support, and this causes the number of backends to drop from 6 to 4. I thought, that "more backends = more accelerators", so "less backends = less speed", but now I understand that's not how it works. Thanks for clarifying.
I suppose, this non-issue can be closed as solved.
Yeah, in future, I will hide those useless options in linux, to avoid confusing users
Expected Behavior
Building new versions from source must succeed with AVX2 ON. I could build past versions up to 1.44.2 with AVX2 ON.
Current Behavior
Does not build with AVX2 since version 1.45.2. Versions 1.46.1 and 1.47.2 also do not build with AVX2 ON. Maybe something is wrong with the makefile. Or it was not documented whether I need to do anything special to make the new versions build with AVX2.
Environment and Context
The problem should not be with my PC. Just to be sure, I re-downloaded the source code of 1.44.2, 1.45.2, 1.46.1, 1.47.2, and then I built all of them again with
make LLAMA_OPENBLAS=1 LLAMA_CLBLAST=1 LLAMA_CUBLAS=1
. All versions failed to build with AVX2, except the good old 1.44.2. Version 1.44.2 works fine, and I am happy with it. All newer version also build, but only in "NoAVX2 Mode (Old CPU), Failsafe Mode (Old CPU)". This makes me sad.OS: Devuan GNU/Linux, Daedalus version Kernel: linux-xanmod-x64v3 CPU: AMD Ryzen 5 3600 (it has avx2) RAM: 64GB VRAM: 12GB, NVIDIA GPU (should be unrelated)
Failure Information (for bugs)
I just run
make LLAMA_OPENBLAS=1 LLAMA_CLBLAST=1 LLAMA_CUBLAS=1
. Throws a lot of gibberish. I understand nothing from it. Please see the attached log files. All builds eventually succeed, but only version 1.44.2 seem to have AVX2 Mode ON.Steps to Reproduce
$ python3 -m venv ./venv
$ source ./venv/bin/activate
$ pip install -r requirements.txt
make LLAMA_OPENBLAS=1 LLAMA_CLBLAST=1 LLAMA_CUBLAS=1
(Maybe I am missing something from here?)$ python3 koboldcpp.py
$ pip install customtkinter packaging
$ python3 koboldcpp.py
Extra environment info (I did not know that this needs torch. Does it? It seem to work without it!):
Runtime info must be unrelated. Just check, that the GUI says "NoAVX2 Mode (Old CPU), Failsafe Mode (Old CPU)".
Failure Logs
See the attachments. 1.44.2-good.log 1.45.2-bad.log 1.46.1-bad.log 1.47.2-bad.log