search

nomic-ai / gpt4all

GPT4All: Chat with Local LLMs on Any Device
https://gpt4all.io
MIT License
68.16k stars 7.48k forks source link

support older Intel CPUs without AVX #1540

Open arteze opened 9 months ago

arteze commented 9 months ago

System Info

Hardware

uname -a
Linux puppypc22006 5.11.11-lxpup64 #1 SMP Tue Mar 30 15:47:34 BST 2021 x86_64 Pentium(R) Dual-Core  CPU      E5700  @ 3.00GHz GenuineIntel GNU/Linux

When open chat

/mnt/sda2/gpt/gpt/bin/chat
Failed to open log file, logging to stdout...
[Debug] (Fri Oct 20 03:27:42 2023): checking directory path "/mnt/sda2/gpt/gpt/plugins/platforms" ...
[Debug] (Fri Oct 20 03:27:42 2023): checking directory path "/mnt/sda2/gpt/gpt/plugins/platforms" ...
[Debug] (Fri Oct 20 03:27:42 2023): looking at "/mnt/sda2/gpt/gpt/plugins/platforms/libqxcb.so"
[Debug] (Fri Oct 20 03:27:42 2023): looking at "/mnt/sda2/gpt/gpt/plugins/platforms/libqxcb.so"
[Debug] (Fri Oct 20 03:27:42 2023): Found metadata in lib /mnt/sda2/gpt/gpt/plugins/platforms/libqxcb.so, metadata=
{
    "IID": "org.qt-project.Qt.QPA.QPlatformIntegrationFactoryInterface.5.3",
    "MetaData": {
        "Keys": [
            "xcb"
        ]
    },
    "archlevel": 1,
    "className": "QXcbIntegrationPlugin",
    "debug": false,
    "version": 394496
}

[Debug] (Fri Oct 20 03:27:42 2023): Found metadata in lib /mnt/sda2/gpt/gpt/plugins/platforms/libqxcb.so, metadata=
{
    "IID": "org.qt-project.Qt.QPA.QPlatformIntegrationFactoryInterface.5.3",
    "MetaData": {
        "Keys": [
            "xcb"
        ]
    },
    "archlevel": 1,
    "className": "QXcbIntegrationPlugin",
    "debug": false,
    "version": 394496
}

[Debug] (Fri Oct 20 03:27:42 2023): Got keys from plugin meta data QList("xcb")
[Debug] (Fri Oct 20 03:27:42 2023): Got keys from plugin meta data QList("xcb")
[Debug] (Fri Oct 20 03:27:42 2023): checking directory path "/mnt/sda2/gpt/gpt/bin/platforms" ...
[Debug] (Fri Oct 20 03:27:42 2023): checking directory path "/mnt/sda2/gpt/gpt/bin/platforms" ...
[Debug] (Fri Oct 20 03:27:42 2023): "/mnt/sda2/gpt/gpt/plugins/platforms/libqxcb.so" loaded library
[Debug] (Fri Oct 20 03:27:42 2023): "/mnt/sda2/gpt/gpt/plugins/platforms/libqxcb.so" loaded library
[Debug] (Fri Oct 20 03:27:42 2023): checking directory path "/mnt/sda2/gpt/gpt/plugins/platformthemes" ...
[Debug] (Fri Oct 20 03:27:42 2023): checking directory path "/mnt/sda2/gpt/gpt/plugins/platformthemes" ...
[Debug] (Fri Oct 20 03:27:42 2023): checking directory path "/mnt/sda2/gpt/gpt/bin/platformthemes" ...
[Debug] (Fri Oct 20 03:27:42 2023): checking directory path "/mnt/sda2/gpt/gpt/bin/platformthemes" ...
[Debug] (Fri Oct 20 03:27:42 2023): checking directory path "/mnt/sda2/gpt/gpt/plugins/platforminputcontexts" ...
[Debug] (Fri Oct 20 03:27:42 2023): checking directory path "/mnt/sda2/gpt/gpt/plugins/platforminputcontexts" ...
[Debug] (Fri Oct 20 03:27:42 2023): looking at "/mnt/sda2/gpt/gpt/plugins/platforminputcontexts/libqtvirtualkeyboardplugin.so"
[Debug] (Fri Oct 20 03:27:42 2023): looking at "/mnt/sda2/gpt/gpt/plugins/platforminputcontexts/libqtvirtualkeyboardplugin.so"
[Debug] (Fri Oct 20 03:27:42 2023): Found metadata in lib /mnt/sda2/gpt/gpt/plugins/platforminputcontexts/libqtvirtualkeyboardplugin.so, metadata=
{
    "IID": "org.qt-project.Qt.QPlatformInputContextFactoryInterface.5.1",
    "MetaData": {
        "Keys": [
            "qtvirtualkeyboard"
        ]
    },
    "archlevel": 1,
    "className": "QVirtualKeyboardPlugin",
    "debug": false,
    "version": 394496
}

[Debug] (Fri Oct 20 03:27:42 2023): Found metadata in lib /mnt/sda2/gpt/gpt/plugins/platforminputcontexts/libqtvirtualkeyboardplugin.so, metadata=
{
    "IID": "org.qt-project.Qt.QPlatformInputContextFactoryInterface.5.1",
    "MetaData": {
        "Keys": [
            "qtvirtualkeyboard"
        ]
    },
    "archlevel": 1,
    "className": "QVirtualKeyboardPlugin",
    "debug": false,
    "version": 394496
}

[Debug] (Fri Oct 20 03:27:42 2023): Got keys from plugin meta data QList("qtvirtualkeyboard")
[Debug] (Fri Oct 20 03:27:42 2023): Got keys from plugin meta data QList("qtvirtualkeyboard")
[Debug] (Fri Oct 20 03:27:42 2023): looking at "/mnt/sda2/gpt/gpt/plugins/platforminputcontexts/libcomposeplatforminputcontextplugin.so"
[Debug] (Fri Oct 20 03:27:42 2023): looking at "/mnt/sda2/gpt/gpt/plugins/platforminputcontexts/libcomposeplatforminputcontextplugin.so"
[Debug] (Fri Oct 20 03:27:42 2023): Found metadata in lib /mnt/sda2/gpt/gpt/plugins/platforminputcontexts/libcomposeplatforminputcontextplugin.so, metadata=
{
    "IID": "org.qt-project.Qt.QPlatformInputContextFactoryInterface.5.1",
    "MetaData": {
        "Keys": [
            "compose",
            "xim"
        ]
    },
    "archlevel": 1,
    "className": "QComposePlatformInputContextPlugin",
    "debug": false,
    "version": 394496
}

[Debug] (Fri Oct 20 03:27:42 2023): Found metadata in lib /mnt/sda2/gpt/gpt/plugins/platforminputcontexts/libcomposeplatforminputcontextplugin.so, metadata=
{
    "IID": "org.qt-project.Qt.QPlatformInputContextFactoryInterface.5.1",
    "MetaData": {
        "Keys": [
            "compose",
            "xim"
        ]
    },
    "archlevel": 1,
    "className": "QComposePlatformInputContextPlugin",
    "debug": false,
    "version": 394496
}

[Debug] (Fri Oct 20 03:27:42 2023): Got keys from plugin meta data QList("compose", "xim")
[Debug] (Fri Oct 20 03:27:42 2023): Got keys from plugin meta data QList("compose", "xim")
[Debug] (Fri Oct 20 03:27:42 2023): looking at "/mnt/sda2/gpt/gpt/plugins/platforminputcontexts/libibusplatforminputcontextplugin.so"
[Debug] (Fri Oct 20 03:27:42 2023): looking at "/mnt/sda2/gpt/gpt/plugins/platforminputcontexts/libibusplatforminputcontextplugin.so"
[Debug] (Fri Oct 20 03:27:42 2023): Found metadata in lib /mnt/sda2/gpt/gpt/plugins/platforminputcontexts/libibusplatforminputcontextplugin.so, metadata=
{
    "IID": "org.qt-project.Qt.QPlatformInputContextFactoryInterface.5.1",
    "MetaData": {
        "Keys": [
            "ibus"
        ]
    },
    "archlevel": 1,
    "className": "QIbusPlatformInputContextPlugin",
    "debug": false,
    "version": 394496
}

[Debug] (Fri Oct 20 03:27:42 2023): Found metadata in lib /mnt/sda2/gpt/gpt/plugins/platforminputcontexts/libibusplatforminputcontextplugin.so, metadata=
{
    "IID": "org.qt-project.Qt.QPlatformInputContextFactoryInterface.5.1",
    "MetaData": {
        "Keys": [
            "ibus"
        ]
    },
    "archlevel": 1,
    "className": "QIbusPlatformInputContextPlugin",
    "debug": false,
    "version": 394496
}

[Debug] (Fri Oct 20 03:27:42 2023): Got keys from plugin meta data QList("ibus")
[Debug] (Fri Oct 20 03:27:42 2023): Got keys from plugin meta data QList("ibus")
[Debug] (Fri Oct 20 03:27:42 2023): checking directory path "/mnt/sda2/gpt/gpt/bin/platforminputcontexts" ...
[Debug] (Fri Oct 20 03:27:42 2023): checking directory path "/mnt/sda2/gpt/gpt/bin/platforminputcontexts" ...
[Debug] (Fri Oct 20 03:27:42 2023): "/mnt/sda2/gpt/gpt/plugins/platforminputcontexts/libcomposeplatforminputcontextplugin.so" loaded library
[Debug] (Fri Oct 20 03:27:42 2023): "/mnt/sda2/gpt/gpt/plugins/platforminputcontexts/libcomposeplatforminputcontextplugin.so" loaded library
Instrucción ilegal

Information

Reproduction

Open chat

Expected behavior

Open

arteze commented 9 months ago

Now I have this problem.

Failed to open log file, logging to stdout...
[Debug] (Fri Oct 20 15:36:41 2023): checking directory path "/bin/platforms" ...
[Debug] (Fri Oct 20 15:36:41 2023): checking directory path "/bin/platforms" ...
[Debug] (Fri Oct 20 15:36:41 2023): checking directory path "/usr/lib/qt/plugins/platforms" ...
[Debug] (Fri Oct 20 15:36:41 2023): checking directory path "/usr/lib/qt/plugins/platforms" ...
[Warning] (Fri Oct 20 15:36:41 2023): Could not find the Qt platform plugin "xcb" in "/usr/lib/qt/plugins/platforms"
[Warning] (Fri Oct 20 15:36:41 2023): Could not find the Qt platform plugin "xcb" in "/usr/lib/qt/plugins/platforms"
[Fatal] (Fri Oct 20 15:36:41 2023): This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

[Fatal] (Fri Oct 20 15:36:41 2023): This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Abortado
cebtenzzre commented 9 months ago

This is a very old CPU, which we do not support. GPT4All requires a processor that supports AVX instructions, which are only available in CPUs from 2016 2011 and newer.

arteze commented 9 months ago

Is there no way to emulate it if I don't have that instruction?

cebtenzzre commented 9 months ago

We could in theory make a build without AVX instructions, since llama.cpp supports it. But it would be extremely slow. My 4 year old Ryzen 5 3600 is 18x as fast as that and I got impatient with the CPU-only performance of AI models very quickly. You would probably be looking at many seconds per word generated.

arteze commented 9 months ago

Yes, I would like to see that compilation made. Let it detect if I have the instruction and if I don't have it, let it use that compilation.

draeath commented 9 months ago

"We could in theory make a build without AVX instructions, since llama.cpp supports it. But it would be extremely slow."

It might not be as bad as you expect. I'm able to run this on a Raspberry Pi 4, and I see roughly 0.5-1.5 words output per second (llama-2-7b-chat.ggmlv3.q4_0.bin on an 8gb RAM model, using backend v2.4.19 built with -march=native)

arteze commented 9 months ago

These are the specifications of my computer:

inxi
CPU~Dual core Pentium E5700 (-MCP-) speed/max~2939/2954 MHz Kernel~5.11.11-lxpup64 x86_64 Up~1 day Mem~1427.5/1990.7MB HDD~95.6GB(75.0% used) Procs~187 Client~Shell inxi~2.3.8
manyoso commented 9 months ago

@cebtenzzre we can't make individual builds like this... only way I'd support it is if we could at runtime detect they don't have AVX and use that with the same build of llama.cpp. I'm regretting that we every supported plain 'ol AVX because it has never worked well and has been a big maintenance burden.

AuqW9cQ commented 9 months ago

I have the i7-3770 Processor, according to the specifications AVX is supported, although the release date is stated as Q2'12.

When I try to start GPT4All I get the following message: ~/gpt4all/bin ./chat Failed to open log file, logging to stdout... terminate called after throwing an instance of 'vk::InitializationFailedError' what(): vk::Instance::enumeratePhysicalDevices: ErrorInitializationFailed zsh: IOT instruction (core dumped) ./chat Is this the same problem with "missing"/too old AVX support?

cebtenzzre commented 9 months ago

what(): vk::Instance::enumeratePhysicalDevices: ErrorInitializationFailed

This is not related to your CPU, it's a Vulkan-related problem. Please open a new issue. Include your OS, GPU, and the output of vulkaninfo.

brankoradovanovic-mcom commented 8 months ago

This is a very old CPU, which we do not support. GPT4All requires a processor that supports AVX instructions, which are only available in CPUs from 2016 and newer.

Is it AVX or rather AVX2? These were actually introduced in 2011 and 2013 respectively.

blacklightpy commented 8 months ago

My CPU was manufactured in 2019, but it doesn't have AVX/AVX2 because it's Pentium Gold (4417U Kaby Lake R), and is not Core series. Dual Core multithreading 2.3 GHz CPU. Not that bad, and has 12GB of RAM.

fanoush commented 4 months ago

My CPU was manufactured in 2019, but it doesn't have AVX/AVX2 because it's Pentium Gold (4417U Kaby Lake R), and is not Core series. Dual Core multithreading 2.3 GHz CPU. Not that bad, and has 12GB of RAM.

Mine is Pentium Silver N6005 launched in 2021 and no AVX too. Quad core up to 3.30 GHz - still pretty fast machine. Have 32GB of DDR4 memory in it.

BTW how important is AVX anyway when the stuff runs mainly on GPU? Which may actually be a bigger problem for my machine, the GPU is Intel® UHD Graphics 900 MHz with 32 Execution Units, how fast and usable that could be with 16GB of shared GPU RAM (provided that intel GPUs get gpt4all support in future)?

cosmic-snow commented 4 months ago

BTW how important is AVX anyway when the stuff runs mainly on GPU? Which may actually be a bigger problem for my machine, the GPU is Intel® UHD Graphics 900 MHz with 32 Execution Units, how fast and usable that could be with 16GB of shared GPU RAM (provided that intel GPUs get gpt4all support in future)?

If you have enough VRAM in your GPU to load the whole model in it, then AVX is not very important. For context: the project initially started without GPU support, and running these things on CPU without that would be incredibly slow.

Now if you search through the issues, I think someone somewhere has managed to find some kind of emulator for AVX, I can't really remember where I saw that, though. Sorry.

You might be able to get it to work somehow, but just note that if you try that, you're on your own.

cebtenzzre commented 4 months ago

If you have enough VRAM in your GPU to load the whole model in it, then AVX is not very important.

We do not currently support Intel GPUs, and even if we did, we would be targeting Intel Arc and Xe graphics. Entry-level integrated GPUs from Intel are not very good for compute; in practice they are often slower than the CPU they are paired with.

cosmic-snow commented 4 months ago

We do not currently support Intel GPUs, and even if we did, we would be targeting Intel Arc and Xe graphics.

I stand corrected on this, then.

The take-away in any case is that while you might be able to get it to run in some way (emulator maybe?), performance would be bad. And unless/until someone steps up to the task, you would be on your own with any problems that might arise. (Read: it's not a common problem and there are other things that take priority)

arteze commented 4 months ago

I was able to install it using the Garuda Linux Chaotic repository, and this time it was able to open, although the CPU incompatibility message appears. Screenshot

Before I couldn't even open the window, now at least I can see it.

arteze commented 4 months ago

I show the specifications I have for the CPU:

cat /proc/cpuinfo 
processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model       : 23
model name  : Pentium(R) Dual-Core  CPU      E5700  @ 3.00GHz
stepping    : 10
microcode   : 0xa07
cpu MHz     : 2435.512
cache size  : 2048 KB
physical id : 0
siblings    : 2
core id     : 0
cpu cores   : 2
apicid      : 0
initial apicid  : 0
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl cpuid aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm xsave lahf_lm pti tpr_shadow vnmi flexpriority vpid dtherm
vmx flags   : vnmi flexpriority tsc_offset vtpr vapic
bugs        : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit
bogomips    : 6037.03
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor   : 1
vendor_id   : GenuineIntel
cpu family  : 6
model       : 23
model name  : Pentium(R) Dual-Core  CPU      E5700  @ 3.00GHz
stepping    : 10
microcode   : 0xa07
cpu MHz     : 2463.650
cache size  : 2048 KB
physical id : 0
siblings    : 2
core id     : 1
cpu cores   : 2
apicid      : 1
initial apicid  : 1
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl cpuid aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm xsave lahf_lm pti tpr_shadow vnmi flexpriority vpid dtherm
vmx flags   : vnmi flexpriority tsc_offset vtpr vapic
bugs        : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit
bogomips    : 6037.03
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:
cosmic-snow commented 4 months ago

Sorry, but unless you manage to make AVX appear in the flags line somehow, it won't work. I'd say the issue is still open in case someone would want to work on that, but in general and as mentioned previously it's low priority.

arteze commented 4 months ago

I saw an alternative whose name is Alpaca.

Could that engine be used instead of Llama? Would that solve the problem?

Another way that occurs to me is to use a Qemu library to emulate AVX.

navr32 commented 4 months ago

I have take too and lost too many time to try this on gpt4all..but now other project run well on old processor : ollama ...and koboldcpp...wich is better for old hardware because run too with old amd vulkan card without any avx at all..so the card like mine Vulkan0: AMD Radeon RX 480 Graphics (RADV POLARIS10) .run with 2tok/s with some gguf model 13b But with your hardware you can just run very little model slowly...but you can.