search

zylon-ai / private-gpt

Interact with your documents using the power of GPT, 100% privately, no data leaks
https://privategpt.dev
Apache License 2.0
53.65k stars 7.21k forks source link

Illegal instruction (core dumped) #203

Closed liyimeng closed 7 months ago

liyimeng commented 1 year ago 
root@fd13db08a5d7:/usr/src/app# python ./privateGPT.py 
llama.cpp: loading model from models/ggml-model-q4_0.bin
llama.cpp: can't use mmap because tensors are not aligned; convert to new format to avoid this
llama_model_load_internal: format     = 'ggml' (old version with low tokenizer quality and no mmap support)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 1000
llama_model_load_internal: n_embd     = 4096
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 32
llama_model_load_internal: n_layer    = 32
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: ftype      = 2 (mostly Q4_0)
llama_model_load_internal: n_ff       = 11008
llama_model_load_internal: n_parts    = 1
llama_model_load_internal: model size = 7B
llama_model_load_internal: ggml ctx size = 4113748.20 KB
llama_model_load_internal: mem required  = 5809.33 MB (+ 2052.00 MB per state)
...................................................................................................
.
llama_init_from_file: kv self size  = 1000.00 MB
AVX = 1 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 | 
Using embedded DuckDB with persistence: data will be stored in: db
Illegal instruction (core dumped)
maozdemir commented 1 year ago

How did you end up with this? Provide more information.

liyimeng commented 1 year ago

I just follow the instructions in README, all the steps when well, pip install python ingest.py ... when I try the last step python ./privateGPT.py I get this.

I run the code on an old xeon CPU (2012), will it be something missing on the CPU? I run 8 cores + 16GB ram

I run in docker with image python:3

MaxAkbar commented 1 year ago

Yeah, same here. Followed the instructions with the sample.

EDIT: After waiting for a bit, it worked but was very slow and did get some kind of an error. Here are the results.


llama.cpp: loading model from D:\PrivateGPT\models\ggml-model-q4_0.bin
llama.cpp: can't use mmap because tensors are not aligned; convert to new format to avoid this
llama_model_load_internal: format     = 'ggml' (old version with low tokenizer quality and no mmap support)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 1000
llama_model_load_internal: n_embd     = 4096
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 32
llama_model_load_internal: n_layer    = 32
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: ftype      = 2 (mostly Q4_0)
llama_model_load_internal: n_ff       = 11008
llama_model_load_internal: n_parts    = 1
llama_model_load_internal: model size = 7B
llama_model_load_internal: ggml ctx size = 4113748.20 KB
llama_model_load_internal: mem required  = 5809.33 MB (+ 2052.00 MB per state)
...................................................................................................
.
llama_init_from_file: kv self size  = 1000.00 MB
AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 | 
Using embedded DuckDB with persistence: data will be stored in: db
gptj_model_load: loading model from 'D:\PrivateGPT\models\ggml-gpt4all-j-v1.3-groovy.bin' - please wait ...
gptj_model_load: n_vocab = 50400
gptj_model_load: n_ctx   = 2048
gptj_model_load: n_embd  = 4096
gptj_model_load: n_head  = 16
gptj_model_load: n_layer = 28
gptj_model_load: n_rot   = 64
gptj_model_load: f16     = 2
gptj_model_load: ggml ctx size = 4505.45 MB
gptj_model_load: memory_size =   896.00 MB, n_mem = 57344
gptj_model_load: ................................... done
gptj_model_load: model size =  3609.38 MB / num tensors = 285

Enter a query: What did the president say about russia? 

llama_print_timings:        load time =   504.66 ms
llama_print_timings:      sample time =     0.00 ms /     1 runs   (    0.00 ms per run)
llama_print_timings: prompt eval time =   646.91 ms /    10 tokens (   64.69 ms per token)
llama_print_timings:        eval time =     0.00 ms /     1 runs   (    0.00 ms per run)
llama_print_timings:       total time =   650.76 ms
gpt_tokenize: unknown token 'Γ'
gpt_tokenize: unknown token 'Ç'
gpt_tokenize: unknown token 'Ö'
gpt_tokenize: unknown token 'Γ'
gpt_tokenize: unknown token 'Ç'
gpt_tokenize: unknown token 'Ö'
gpt_tokenize: unknown token 'Γ'
gpt_tokenize: unknown token 'Ç'
gpt_tokenize: unknown token 'Ö'
gpt_tokenize: unknown token 'Γ'
gpt_tokenize: unknown token 'Ç'
gpt_tokenize: unknown token '£'
gpt_tokenize: unknown token 'Γ'
gpt_tokenize: unknown token 'Ç'
gpt_tokenize: unknown token '¥'
gpt_tokenize: unknown token 'Γ'
gpt_tokenize: unknown token 'Ç'
gpt_tokenize: unknown token 'Ö'
gpt_tokenize: unknown token 'Γ'
gpt_tokenize: unknown token 'Ç'
gpt_tokenize: unknown token 'Ö'
gpt_tokenize: unknown token 'Γ'
gpt_tokenize: unknown token 'Ç'
gpt_tokenize: unknown token 'Ö'
gpt_tokenize: unknown token 'Γ'
gpt_tokenize: unknown token 'Ç'
We are choking off Russia’s access to technology that will sap its economic strength and weaken its military for years to come.

Tonight I say to the Russian oligarchs and corrupt leaders who have bilked billions of dollars off this violent regime no more.

> source_documents\state_of_the_union.txt:
While it shouldn’t have taken something so terrible for people around the world to see what’s at stake now everyone sees it clearly.

We see the unity among leaders of nations and a more unified Europe a more unified West. And we see unity among the people who are gathering in cities in large crowds around the world even in Russia to demonstrate their support for Ukraine.    

> source_documents\state_of_the_union.txt:
Please rise if you are able and show that, Yes, we the United States of America stand with the Ukrainian people.

Throughout our history we’ve learned this lesson when dictators do not pay a price for their aggression they cause more chaos.

They keep moving.

And the costs and the threats to America and the world keep rising.

That’s why the NATO Alliance was created to secure peace and stability in Europe after World War 2.

> source_documents\state_of_the_union.txt:
We prepared extensively and carefully.

We spent months building a coalition of other freedom-loving nations from Europe and the Americas to Asia and Africa to confront Putin.

I spent countless hours unifying our European allies. We shared with the world in advance what we knew Putin was planning and precisely how he would try to falsely justify his aggression.

We countered Russia’s lies with truth.

And now that he has acted the free world is holding him accountable.

Enter a query:```
apcameron commented 1 year ago

I am getting the same error. I then tried to update to the latest ggml format but I still get the error.

 python privateGPT.py
llama.cpp: loading model from models/ggml-model-q4_0.bin
llama_model_load_internal: format     = ggjt v2 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 1000
llama_model_load_internal: n_embd     = 4096
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 32
llama_model_load_internal: n_layer    = 32
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: ftype      = 2 (mostly Q4_0)
llama_model_load_internal: n_ff       = 11008
llama_model_load_internal: n_parts    = 1
llama_model_load_internal: model size = 7B
llama_model_load_internal: ggml ctx size =  72.75 KB
llama_model_load_internal: mem required  = 5809.34 MB (+ 2052.00 MB per state)
llama_init_from_file: kv self size  = 1000.00 MB
AVX = 1 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 | 
Using embedded DuckDB with persistence: data will be stored in: db
Illegal instruction (core dumped)

Then Looking a dmesg I saw the following [ 1006.633763] traps: python[7629] trap invalid opcode ip:7f50f5f1f776 sp:7fff6f8ac370 error:0 in _pyllamacpp.cpython-310-x86_64-linux-gnu.so[7f50f5edb000+64000] [ 1914.619589] traps: python[7773] trap invalid opcode ip:7f7869b1f776 sp:7ffc6d97bfe0 error:0 in _pyllamacpp.cpython-310-x86_64-linux-gnu.so[7f7869adb000+64000] [ 2127.314608] traps: python[7826] trap invalid opcode ip:7f1fdb287776 sp:7ffd8f856810 error:0 in _pyllamacpp.cpython-310-x86_64-linux-gnu.so[7f1fdb243000+64000] [ 2524.098079] traps: python[8023] trap invalid opcode ip:7f3335916776 sp:7ffdc2830a00 error:0 in _pyllamacpp.cpython-310-x86_64-linux-gnu.so[7f33358d2000+64000] [ 2742.237852] traps: python[8070] trap invalid opcode ip:7fa4cdf08776 sp:7ffd1c4eb990 error:0 in _pyllamacpp.cpython-310-x86_64-linux-gnu.so[7fa4cdec4000+64000]

maozdemir commented 1 year ago

The iasue is not about GGML, since the most recent release on pip of llama-cpp-python is not the most upstream one.

If you can run it wirh valgrind (which will significantly reduce the speed) we can help better.

getvasanth commented 1 year ago

I did try running the valgrind, this is the latest code.. any pointer will help, trying to run on a ubuntu vm with python3.10

valgrind python3.10 privateGPT.py ==312712== Memcheck, a memory error detector ==312712== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. ==312712== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info ==312712== Command: python3.10 privateGPT.py ==312712== ==312712== Invalid read of size 8 ==312712== at 0x40286C0: strncmp (strcmp.S:170) ==312712== by 0x400668D: is_dst (dl-load.c:216) ==312712== by 0x400810E: _dl_dst_count (dl-load.c:253) ==312712== by 0x400810E: expand_dynamic_string_token (dl-load.c:395) ==312712== by 0x40082B7: fillin_rpath.isra.0 (dl-load.c:483) ==312712== by 0x4008602: decompose_rpath (dl-load.c:654) ==312712== by 0x400A8C8: cache_rpath (dl-load.c:696) ==312712== by 0x400A8C8: cache_rpath (dl-load.c:677) ==312712== by 0x400A8C8: _dl_map_object (dl-load.c:2117) ==312712== by 0x4003494: openaux (dl-deps.c:64) ==312712== by 0x4B11C27: _dl_catch_exception (dl-error-skeleton.c:208) ==312712== by 0x4003C7B: _dl_map_object_deps (dl-deps.c:248) ==312712== by 0x400EA0E: dl_open_worker_begin (dl-open.c:592) ==312712== by 0x4B11C27: _dl_catch_exception (dl-error-skeleton.c:208) ==312712== by 0x400DF99: dl_open_worker (dl-open.c:782) ==312712== Address 0x4c768cd is 365 bytes inside a block of size 372 alloc'd ==312712== at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==312712== by 0x40271FF: malloc (rtld-malloc.h:56) ==312712== by 0x40271FF: strdup (strdup.c:42) ==312712== by 0x4008594: decompose_rpath (dl-load.c:629) ==312712== by 0x400A8C8: cache_rpath (dl-load.c:696) ==312712== by 0x400A8C8: cache_rpath (dl-load.c:677) ==312712== by 0x400A8C8: _dl_map_object (dl-load.c:2117) ==312712== by 0x4003494: openaux (dl-deps.c:64) ==312712== by 0x4B11C27: _dl_catch_exception (dl-error-skeleton.c:208) ==312712== by 0x4003C7B: _dl_map_object_deps (dl-deps.c:248) ==312712== by 0x400EA0E: dl_open_worker_begin (dl-open.c:592) ==312712== by 0x4B11C27: _dl_catch_exception (dl-error-skeleton.c:208) ==312712== by 0x400DF99: dl_open_worker (dl-open.c:782) ==312712== by 0x4B11C27: _dl_catch_exception (dl-error-skeleton.c:208) ==312712== by 0x400E34D: _dl_open (dl-open.c:883) ==312712== ==312712== Invalid read of size 8 ==312712== at 0x40286C8: strncmp (strcmp.S:172) ==312712== by 0x400668D: is_dst (dl-load.c:216) ==312712== by 0x400810E: _dl_dst_count (dl-load.c:253) ==312712== by 0x400810E: expand_dynamic_string_token (dl-load.c:395) ==312712== by 0x40082B7: fillin_rpath.isra.0 (dl-load.c:483) ==312712== by 0x4008602: decompose_rpath (dl-load.c:654) ==312712== by 0x400A8C8: cache_rpath (dl-load.c:696) ==312712== by 0x400A8C8: cache_rpath (dl-load.c:677) ==312712== by 0x400A8C8: _dl_map_object (dl-load.c:2117) ==312712== by 0x4003494: openaux (dl-deps.c:64) ==312712== by 0x4B11C27: _dl_catch_exception (dl-error-skeleton.c:208) ==312712== by 0x4003C7B: _dl_map_object_deps (dl-deps.c:248) ==312712== by 0x400EA0E: dl_open_worker_begin (dl-open.c:592) ==312712== by 0x4B11C27: _dl_catch_exception (dl-error-skeleton.c:208) ==312712== by 0x400DF99: dl_open_worker (dl-open.c:782) ==312712== Address 0x4c768d5 is 1 bytes after a block of size 372 alloc'd ==312712== at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==312712== by 0x40271FF: malloc (rtld-malloc.h:56) ==312712== by 0x40271FF: strdup (strdup.c:42) ==312712== by 0x4008594: decompose_rpath (dl-load.c:629) ==312712== by 0x400A8C8: cache_rpath (dl-load.c:696) ==312712== by 0x400A8C8: cache_rpath (dl-load.c:677) ==312712== by 0x400A8C8: _dl_map_object (dl-load.c:2117) ==312712== by 0x4003494: openaux (dl-deps.c:64) ==312712== by 0x4B11C27: _dl_catch_exception (dl-error-skeleton.c:208) ==312712== by 0x4003C7B: _dl_map_object_deps (dl-deps.c:248) ==312712== by 0x400EA0E: dl_open_worker_begin (dl-open.c:592) ==312712== by 0x4B11C27: _dl_catch_exception (dl-error-skeleton.c:208) ==312712== by 0x400DF99: dl_open_worker (dl-open.c:782) ==312712== by 0x4B11C27: _dl_catch_exception (dl-error-skeleton.c:208) ==312712== by 0x400E34D: _dl_open (dl-open.c:883) ==312712== ==312712== Invalid read of size 8 ==312712== at 0x40286C0: strncmp (strcmp.S:170) ==312712== by 0x400668D: is_dst (dl-load.c:216) ==312712== by 0x4007F79: _dl_dst_substitute (dl-load.c:295) ==312712== by 0x40082B7: fillin_rpath.isra.0 (dl-load.c:483) ==312712== by 0x4008602: decompose_rpath (dl-load.c:654) ==312712== by 0x400A8C8: cache_rpath (dl-load.c:696) ==312712== by 0x400A8C8: cache_rpath (dl-load.c:677) ==312712== by 0x400A8C8: _dl_map_object (dl-load.c:2117) ==312712== by 0x4003494: openaux (dl-deps.c:64) ==312712== by 0x4B11C27: _dl_catch_exception (dl-error-skeleton.c:208) ==312712== by 0x4003C7B: _dl_map_object_deps (dl-deps.c:248) ==312712== by 0x400EA0E: dl_open_worker_begin (dl-open.c:592) ==312712== by 0x4B11C27: _dl_catch_exception (dl-error-skeleton.c:208) ==312712== by 0x400DF99: dl_open_worker (dl-open.c:782) ==312712== Address 0x4c768cd is 365 bytes inside a block of size 372 alloc'd ==312712== at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==312712== by 0x40271FF: malloc (rtld-malloc.h:56) ==312712== by 0x40271FF: strdup (strdup.c:42) ==312712== by 0x4008594: decompose_rpath (dl-load.c:629) ==312712== by 0x400A8C8: cache_rpath (dl-load.c:696) ==312712== by 0x400A8C8: cache_rpath (dl-load.c:677) ==312712== by 0x400A8C8: _dl_map_object (dl-load.c:2117) ==312712== by 0x4003494: openaux (dl-deps.c:64) ==312712== by 0x4B11C27: _dl_catch_exception (dl-error-skeleton.c:208) ==312712== by 0x4003C7B: _dl_map_object_deps (dl-deps.c:248) ==312712== by 0x400EA0E: dl_open_worker_begin (dl-open.c:592) ==312712== by 0x4B11C27: _dl_catch_exception (dl-error-skeleton.c:208) ==312712== by 0x400DF99: dl_open_worker (dl-open.c:782) ==312712== by 0x4B11C27: _dl_catch_exception (dl-error-skeleton.c:208) ==312712== by 0x400E34D: _dl_open (dl-open.c:883) ==312712== ==312712== Invalid read of size 8 ==312712== at 0x40286C8: strncmp (strcmp.S:172) ==312712== by 0x400668D: is_dst (dl-load.c:216) ==312712== by 0x4007F79: _dl_dst_substitute (dl-load.c:295) ==312712== by 0x40082B7: fillin_rpath.isra.0 (dl-load.c:483) ==312712== by 0x4008602: decompose_rpath (dl-load.c:654) ==312712== by 0x400A8C8: cache_rpath (dl-load.c:696) ==312712== by 0x400A8C8: cache_rpath (dl-load.c:677) ==312712== by 0x400A8C8: _dl_map_object (dl-load.c:2117) ==312712== by 0x4003494: openaux (dl-deps.c:64) ==312712== by 0x4B11C27: _dl_catch_exception (dl-error-skeleton.c:208) ==312712== by 0x4003C7B: _dl_map_object_deps (dl-deps.c:248) ==312712== by 0x400EA0E: dl_open_worker_begin (dl-open.c:592) ==312712== by 0x4B11C27: _dl_catch_exception (dl-error-skeleton.c:208) ==312712== by 0x400DF99: dl_open_worker (dl-open.c:782) ==312712== Address 0x4c768d5 is 1 bytes after a block of size 372 alloc'd ==312712== at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==312712== by 0x40271FF: malloc (rtld-malloc.h:56) ==312712== by 0x40271FF: strdup (strdup.c:42) ==312712== by 0x4008594: decompose_rpath (dl-load.c:629) ==312712== by 0x400A8C8: cache_rpath (dl-load.c:696) ==312712== by 0x400A8C8: cache_rpath (dl-load.c:677) ==312712== by 0x400A8C8: _dl_map_object (dl-load.c:2117) ==312712== by 0x4003494: openaux (dl-deps.c:64) ==312712== by 0x4B11C27: _dl_catch_exception (dl-error-skeleton.c:208) ==312712== by 0x4003C7B: _dl_map_object_deps (dl-deps.c:248) ==312712== by 0x400EA0E: dl_open_worker_begin (dl-open.c:592) ==312712== by 0x4B11C27: _dl_catch_exception (dl-error-skeleton.c:208) ==312712== by 0x400DF99: dl_open_worker (dl-open.c:782) ==312712== by 0x4B11C27: _dl_catch_exception (dl-error-skeleton.c:208) ==312712== by 0x400E34D: _dl_open (dl-open.c:883) ==312712== ==312712== Warning: set address range perms: large range [0x2a7fa000, 0x3b8d5000) (noaccess) ==312712== Warning: set address range perms: large range [0x2a800000, 0x3b6db000) (defined) ==312712== Warning: set address range perms: large range [0x59c87000, 0x6de28000) (noaccess) ==312712== Warning: set address range perms: large range [0x59e00000, 0x6dda1000) (defined) ==312712== Warning: set address range perms: large range [0x6dda1000, 0x86c54000) (defined) ==312712== Warning: set address range perms: large range [0x6ede2000, 0x81cd6000) (defined) ==312712== Warning: set address range perms: large range [0x86c54000, 0xacd14000) (defined) ==312712== Warning: set address range perms: large range [0x89bf6000, 0xac003000) (defined) Traceback (most recent call last): File "/home/user/workspace/ai/privateGPT/privateGPT.py", line 57, in main() File "/home/user/workspace/ai/privateGPT/privateGPT.py", line 21, in main embeddings = HuggingFaceEmbeddings(model_name=embeddings_model_name) File "/home/user/.local/lib/python3.10/site-packages/langchain/embeddings/huggingface.py", line 46, in init import sentence_transformers File "/home/user/.local/lib/python3.10/site-packages/sentence_transformers/init.py", line 3, in from .datasets import SentencesDataset, ParallelSentencesDataset File "/home/user/.local/lib/python3.10/site-packages/sentence_transformers/datasets/init.py", line 1, in from .DenoisingAutoEncoderDataset import DenoisingAutoEncoderDataset File "/home/user/.local/lib/python3.10/site-packages/sentence_transformers/datasets/DenoisingAutoEncoderDataset.py", line 5, in import nltk File "/home/user/.local/lib/python3.10/site-packages/nltk/init.py", line 133, in from nltk.collocations import File "/home/user/.local/lib/python3.10/site-packages/nltk/collocations.py", line 36, in from nltk.metrics import ( File "/home/user/.local/lib/python3.10/site-packages/nltk/metrics/init.py", line 18, in from nltk.metrics.association import ( File "/home/user/.local/lib/python3.10/site-packages/nltk/metrics/association.py", line 26, in from scipy.stats import fisher_exact File "/home/user/.local/lib/python3.10/site-packages/scipy/stats/init.py", line 485, in from ._stats_py import File "/home/user/.local/lib/python3.10/site-packages/scipy/stats/_stats_py.py", line 46, in from . import distributions File "/home/user/.local/lib/python3.10/site-packages/scipy/stats/distributions.py", line 10, in from . import _continuous_distns File "/home/user/.local/lib/python3.10/site-packages/scipy/stats/_continuous_distns.py", line 32, in import scipy.stats._boost as _boost File "/home/user/.local/lib/python3.10/site-packages/scipy/stats/_boost/init.py", line 1, in from scipy.stats._boost.beta_ufunc import ( SystemError: initialization of beta_ufunc raised unreported exception ==312712== ==312712== HEAP SUMMARY: ==312712== in use at exit: 22,089,026 bytes in 73,233 blocks ==312712== total heap usage: 1,832,659 allocs, 1,759,426 frees, 430,167,687 bytes allocated ==312712== ==312712== LEAK SUMMARY: ==312712== definitely lost: 1,728 bytes in 12 blocks ==312712== indirectly lost: 0 bytes in 0 blocks ==312712== possibly lost: 391,240 bytes in 293 blocks ==312712== still reachable: 21,696,058 bytes in 72,928 blocks ==312712== of which reachable via heuristic: ==312712== stdstring : 386,085 bytes in 9,992 blocks ==312712== suppressed: 0 bytes in 0 blocks ==312712== Rerun with --leak-check=full to see details of leaked memory ==312712== ==312712== For lists of detected and suppressed errors, rerun with: -s ==312712== ERROR SUMMARY: 22 errors from 4 contexts (suppressed: 0 from 0)

maozdemir commented 1 year ago

Oh well it feels like it's about memory, not sure about that though.

Maybe you should try to compile llama-cpp-python and langchain on your local machine. Check my repos for the Jupyter Notebook that contains the code that can guide you...

andrediasbarbosa commented 1 year ago

Indeed Gentlemen, after having unsuccessfully tried to compile this project in Windows, I have switched to WSL and am now getting a "illegal Instruction" error - an HW error for sure, however its quite a new processor. Intel Celeron 1.5GHz CPU J3455

jonarmani commented 1 year ago

I also get `Illegal instruction" on several machines. I don't think it's a hardware resource issue, as it fails the same on a 24-core Intel Xeon E5-2620 with 32GB RAM.

I'm using Python 3.11.3 in pyenv.

python -vvv privateGPT.py reveals it tripping up on ~/.pyenv/versions/privateGPT/lib/python3.11/site-packages/_pyllamacpp.cpython-311-x86_64-linux-gnu.so

liyimeng commented 1 year ago

@jonarmani I have a similar cpu like yours

ossie-git commented 1 year ago

I ran into this on an older machine. But when I tried it on a newer CPU, it worked successfully. I think the difference is if the CPU supports the AVX2 instruction. To quickly check for this:

$ cat /proc/cpuinfo | grep avx2
ubuntinux commented 1 year ago

Your CPU needs to support avx2 instructions, otherwise executing privateGPT script will give you the error: "Illegal instruction (core dumped)". Tested on two of my computers where one has an AMD CPU with avx instruction set (Illegal instruction (core dumped)), and on another INTEL CPU with avx and avx2 instruction set. (it worked). Solutions: https://tech.amikelive.com/node-887/how-to-resolve-error-illegal-instruction-core-dumped-when-running-import-tensorflow-in-a-python-program/

andrediasbarbosa commented 1 year ago

Brilliant, many thanks, however this project does not rely on TensorFlow -> would be great knowing what other libraries could possible be causing this error and which should we downgrade.

On Fri, May 19, 2023 at 10:52 AM ubuntinux @.***> wrote:

Your CPU needs to support avx2 instructions, otherwise executing privateGPT script will give you the error: "Illegal instruction (core dumped)". Tested on two of my computers where one has an AMD CPU with avx instruction set (Illegal instruction (core dumped)), and on another INTEL CPU with avx and avx2 instruction set. (it worked). Solutions: https://tech.amikelive.com/node-887/how-to-resolve-error-illegal-instruction-core-dumped-when-running-import-tensorflow-in-a-python-program/

— Reply to this email directly, view it on GitHub https://github.com/imartinez/privateGPT/issues/203#issuecomment-1554321796, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABMBTKEGLVOUJDLJOGCJOXLXG47GXANCNFSM6AAAAAAYDMJHC4 . You are receiving this because you commented.Message ID: @.***>

apcameron commented 1 year ago

Following this guide https://github.com/nomic-ai/pygpt4all/issues/71 If it still fails git clone --recursive https://github.com/abdeladim-s/pygptj.git and install this with pip install e .

jonarmani commented 1 year ago

I've tried with all compiler options set to OFF in the CMakeLists.txt for pyllamacpp, and am still seeing the error. 🙁

apcameron commented 1 year ago

Did you try updating pygptj as well? If you are running linux dmesg may give you a clue as to where it is failing

Lars-Wunderlich commented 1 year ago

Got the same error on a M1 mac

~/git/privateGPT main ✗ 4h41m ✖ ◒ ⍉ ▶ python3.10 privateGPT.py
Using embedded DuckDB with persistence: data will be stored in: db [1] 72993 illegal hardware instruction python3.10 privateGPT.py (venv) (base)

PyGPT-J A simple Command Line Interface to test the package Version: 2.0.3

[+] Running model models/ggml-gpt4all-j-v1.3-groovy.bin gptj_model_load: loading model from 'models/ggml-gpt4all-j-v1.3-groovy.bin' - please wait ... gptj_model_load: n_vocab = 50400 gptj_model_load: n_ctx = 2048 gptj_model_load: n_embd = 4096 gptj_model_load: n_head = 16 gptj_model_load: n_layer = 28 gptj_model_load: n_rot = 64 gptj_model_load: f16 = 2 [1] 73146 illegal hardware instruction pygptj models/ggml-gpt4all-j-v1.3-groovy.bin (venv) (base)

jonarmani commented 1 year ago

dmesg tells me it's this llama cpp library: python[31999] trap invalid opcode ip:7f8d8fb6559f sp:7ffd0b9559a0 error:0 in _pyllamacpp.cpython-311-x86_64-linux-gnu.so[7f8d8fb55000+5b000]

but again, I've modified its CMakeLists.txt to turn all CPU opts OFF and rebuilt with python -m pip install -e .

I'm using pyenv, which is new to me, so maybe I'm doing something weird there. But I'm in the activated environment the whole time, not issuing any pyenv or related commands. EDIT: this output is the same as the folder I built it in above:

>>> import pkg_resources
>>> pkg_resources.require("pyllamacpp")
[pyllamacpp 1.0.7 (/home/jon/pyllamacpp/pyllamacpp)]
digitalw00t commented 1 year ago

Oh it's definately that. I ran this through gdb: draeician@vivy:~/git/privateGPT$ . venv/bin/activate (venv) draeician@vivy:~/git/privateGPT$ gdb python --->lots of start up message<--- (gdb) run privateGPT.py --->lots of [New Thread 0x7fffd2ff3640 (LWP 129334)] messages<--- [New Thread 0x7ffe92ffd640 (LWP 129612)]

Thread 1 "python" received signal SIGILL, Illegal instruction. 0x00007ffed3723776 in ?? () from /home/draeician/git/privateGPT/venv/lib/python3.10/site-packages/_pyllamacpp.cpython-310-x86_64-linux-gnu.so (gdb) exit A debugging session is active.

vachu commented 1 year ago

Following this guide nomic-ai/pygpt4all#71 If it still fails git clone --recursive https://github.com/abdeladim-s/pygptj.git and install this with pip install e .

Tried both the options but still getting the error. I am running privateGPT in a Manjaro VirtualBox VM, with 4 CPU cores assigned and 10GB of RAM. The Host CPU is Intel Core vPRO i9

image

WIIN-AI commented 1 year ago

I am also Observing : Illegal instruction (core dumped). Note: This is in RHEL container..

+1

vachu commented 1 year ago

Following this guide nomic-ai/pygpt4all#71 If it still fails git clone --recursive https://github.com/abdeladim-s/pygptj.git and install this with pip install e .

Tried both the options but still getting the error. I am running privateGPT in a Manjaro VirtualBox VM, with 4 CPU cores assigned and 10GB of RAM. The Host CPU is Intel Core vPRO i9

image

Hi all,

I think I have found a solution - more of a workaround. The root cause seems to be Hyper-V is somehow enabled (albeit surreptitiously) on my Windows 10 Host, due to which the VirtualBox Guest (Manjaro Linux) doesn't seem to have access to CPU Instructions like AVX2.

So, I migrated to WSL2 Ubuntu 22.04 and from there, I was able to build this app out of the box. Suits my needs, for the time being.

PS: Not sure why Hyper-V is still "active" despite turning it off explicitly. Also, I was not aware that VirtualBox 7.0.x indeed works with Hyper-V on. With 6.x versions, I vividly remember I was not able to guests with Hyper-V on

maozdemir commented 1 year ago

Following this guide nomic-ai/pygpt4all#71 If it still fails git clone --recursive https://github.com/abdeladim-s/pygptj.git and install this with pip install e .

Tried both the options but still getting the error. I am running privateGPT in a Manjaro VirtualBox VM, with 4 CPU cores assigned and 10GB of RAM. The Host CPU is Intel Core vPRO i9 image

Hi all,

I think I have found a solution - more of a workaround. The root cause seems to be Hyper-V is somehow enabled (albeit surreptitiously) on my Windows 10 Host, due to which the VirtualBox Guest (Manjaro Linux) doesn't seem to have access to CPU Instructions like AVX2.

So, I migrated to WSL2 Ubuntu 22.04 and from there, I was able to build this app out of the box. Suits my needs, for the time being.

PS: Not sure why Hyper-V is still "active" despite turning it off explicitly. Also, I was not aware that VirtualBox 7.0.x indeed works with Hyper-V on. With 6.x versions, I vividly remember I was not able to guests with Hyper-V on

Out of curiosity, Hyper-V VM users don't seem to be having such an issue, why are you using vbox? WSL2 also utilises Hyper-V afaik, so I assume you'd have no issues on it.

vachu commented 1 year ago

Following this guide nomic-ai/pygpt4all#71 If it still fails git clone --recursive https://github.com/abdeladim-s/pygptj.git and install this with pip install e .

Tried both the options but still getting the error. I am running privateGPT in a Manjaro VirtualBox VM, with 4 CPU cores assigned and 10GB of RAM. The Host CPU is Intel Core vPRO i9 image

Hi all, I think I have found a solution - more of a workaround. The root cause seems to be Hyper-V is somehow enabled (albeit surreptitiously) on my Windows 10 Host, due to which the VirtualBox Guest (Manjaro Linux) doesn't seem to have access to CPU Instructions like AVX2. So, I migrated to WSL2 Ubuntu 22.04 and from there, I was able to build this app out of the box. Suits my needs, for the time being. PS: Not sure why Hyper-V is still "active" despite turning it off explicitly. Also, I was not aware that VirtualBox 7.0.x indeed works with Hyper-V on. With 6.x versions, I vividly remember I was not able to guests with Hyper-V on

Out of curiosity, Hyper-V VM users don't seem to be having such an issue, why are you using vbox? WSL2 also utilises Hyper-V afaik, so I assume you'd have no issues on it.

I need to learn the nitty-gritties myself but from whatever I have understood, Microsoft's Hyper-V is a "Type 1" Hypervisor that has all / max control of the underlying hardware. All the guest OS-es don't have as much access / control of the underlying hardware and this could be the reason for advanced CPU Instruction Sets like AVX2 being unavailable to guest OS-es of other apps like VBox. That doesn't seem to be the case for direct guest OS-es of Hyper-V itself.

Before installing VBox, I had ensured that I had disabled Hyper-V completely thru the "Turn Windows features on or off" yet it looks like Hyper-V is/was still running "surreptitiously" - the Turtle icon at the bottom-right of the VBox window indicated that. I abandoned my fight to get rid of that Turtle icon and migrated to WSL2 instead. Btw, I had to use VBox due to my professional work

maozdemir commented 1 year ago

Following this guide nomic-ai/pygpt4all#71 If it still fails git clone --recursive https://github.com/abdeladim-s/pygptj.git and install this with pip install e .

Tried both the options but still getting the error. I am running privateGPT in a Manjaro VirtualBox VM, with 4 CPU cores assigned and 10GB of RAM. The Host CPU is Intel Core vPRO i9 image

Hi all, I think I have found a solution - more of a workaround. The root cause seems to be Hyper-V is somehow enabled (albeit surreptitiously) on my Windows 10 Host, due to which the VirtualBox Guest (Manjaro Linux) doesn't seem to have access to CPU Instructions like AVX2. So, I migrated to WSL2 Ubuntu 22.04 and from there, I was able to build this app out of the box. Suits my needs, for the time being. PS: Not sure why Hyper-V is still "active" despite turning it off explicitly. Also, I was not aware that VirtualBox 7.0.x indeed works with Hyper-V on. With 6.x versions, I vividly remember I was not able to guests with Hyper-V on

Out of curiosity, Hyper-V VM users don't seem to be having such an issue, why are you using vbox? WSL2 also utilises Hyper-V afaik, so I assume you'd have no issues on it.

I need to learn the nitty-gritties myself but from whatever I have understood, Microsoft's Hyper-V is a "Type 1" Hypervisor that has all / max control of the underlying hardware. All the guest OS-es don't have as much access / control of the underlying hardware and this could be the reason for advanced CPU Instruction Sets like AVX2 being unavailable to guest OS-es of other apps like VBox. That doesn't seem to be the case for direct guest OS-es of Hyper-V itself.

Before installing VBox, I had ensured that I had disabled Hyper-V completely thru the "Turn Windows features on or off" yet it looks like Hyper-V is/was still running "surreptitiously" - the Turtle icon at the bottom-right of the VBox window indicated that. I abandoned my fight to get rid of that Turtle icon and migrated to WSL2 instead. Btw, I had to use VBox due to my professional work

alright, then you can try this: https://stackoverflow.com/questions/65780506/how-to-enable-avx-avx2-in-virtualbox-6-1-16-with-ubuntu-20-04-64bit or disable the windows antivirus completely, as far as i know that's what causes the instruction blockage.