search

janhq / jan

Jan is an open source alternative to ChatGPT that runs 100% offline on your computer. Multiple engine support (llama.cpp, TensorRT-LLM)
https://jan.ai/
GNU Affero General Public License v3.0
23.09k stars 1.34k forks source link

bug: No model can start #3898

Open fuzz4 opened 4 days ago

fuzz4 commented 4 days ago

Jan version

0.5.7

Describe the Bug

tried to start different models but none starts, tried with many, same result. running Jan on a imac with Monterey. thanks for your feedback. P.S. seen same issue but NVDIA card does not apply in my case since have a built in radeon.

Steps to Reproduce

No response

Screenshots / Logs

No response

What is your OS?

imtuyethan commented 4 days ago

@fuzz4 Can I get more info on:

Thank you!

fuzz4 commented 2 days ago

Hi, here are the infos: Mac: iMac (Retina 5K, 27-inch, 2017), 3,4 GHz Intel Core i5 quad-core, Memory: 40 GB 2400 MHz DDR4, graphic card : Radeon Pro 570 4 GB, Running Macos Monterey

logs: 2024-10-30T21:19:19.012Z [CORTEX]::Debug: Cortex engine path: jan-data-folder/extensions/@janhq/inference-cortex-extension/dist/bin/mac-x64 2024-10-30T21:19:19.012Z [CORTEX] PATH: /usr/bin:/bin:/usr/sbin:/sbin::jan-data-folder/engines/@janhq/inference-cortex-extension/1.0.20:jan-data-folder/extensions/@janhq/inference-cortex-extension/dist/bin/mac-x64 2024-10-30T21:19:19.131Z [CORTEX]:: cortex is ready 2024-10-30T21:19:19.131Z [CORTEX]:: Loading model with params {"cpu_threads":4,"ctx_len":4096,"prompt_template":"<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n{system_message}<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n","llama_model_path":"jan-data-folder/models/llama3.2-3b-instruct/Llama-3.2-3B-Instruct-Q8_0.gguf","ngl":33,"system_prompt":"<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n","user_prompt":"<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n","ai_prompt":"<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n","model":"llama3.2-3b-instruct"} 2024-10-30T21:19:19.162Z [CORTEX]:: 20241030 21:19:19.114096 UTC 1671673 INFO cortex-cpp version: 0.5.0 - main.cc:73 20241030 21:19:19.114976 UTC 1671673 INFO Server started, listening at: 127.0.0.1:3928 - main.cc:78 20241030 21:19:19.114978 UTC 1671673 INFO Please load your model - main.cc:79 20241030 21:19:19.114993 UTC 1671673 INFO Number of thread is:4 - main.cc:86 20241030 21:19:19.140852 UTC 1671676 INFO CPU instruction set: fpu = 1| mmx = 1| sse = 1| sse2 = 1| sse3 = 1| ssse3 = 1| sse4_1 = 1| sse4_2 = 1| pclmulqdq = 1| avx = 1| avx2 = 1| avx512_f = 0| avx512_dq = 0| avx512_ifma = 0| avx512_pf = 0| avx512_er = 0| avx512_cd = 0| avx512_bw = 0| has_avx512_vl = 0| has_avx512_vbmi = 0| has_avx512_vbmi2 = 0| avx512_vnni = 0| avx512_bitalg = 0| avx512_vpopcntdq = 0| avx512_4vnniw = 0| avx512_4fmaps = 0| avx512_vp2intersect = 0| aes = 1| f16c = 1| - server.cc:288 20241030 21:19:19.161951 UTC 1671676 ERROR Could not load engine: Could not load library "jan-data-folder/extensions/@janhq/inference-cortex-extension/dist/bin/mac-x64/engines/cortex.llamacpp/libengine.dylib" dlopen(jan-data-folder/extensions/@janhq/inference-cortex-extension/dist/bin/mac-x64/engines/cortex.llamacpp/libengine.dylib, 0x0006): Symbol not found: (_cblas_sgemm$NEWLAPACK$ILP64) Referenced from: 'jan-data-folder/extensions/@janhq/inference-cortex-extension/dist/bin/mac-x64/engines/cortex.llamacpp/libengine.dylib' Expected in: '/System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate' - server.cc:299

2024-10-30T21:19:19.170Z [CORTEX]:: Validating model llama3.2-3b-instruct 2024-10-30T21:19:19.170Z [CORTEX]:: Load model success with response {} 2024-10-30T21:19:19.173Z [CORTEX]:: Validate model state with response 409 2024-10-30T21:19:19.175Z [CORTEX]:: Validate model state failed with response {"message":"Engine is not loaded yet"} and status is "Conflict" 2024-10-30T21:19:19.175Z [CORTEX]::Error: Validate model status failed 2024-10-30T21:19:44.513Z [CORTEX]:: Request to kill cortex 2024-10-30T21:19:44.518Z [CORTEX]:: 20241030 21:19:19.172832 UTC 1671677 WARN Engine is not loaded yet - server.cc:141 20241030 21:19:44.516016 UTC 1671678 INFO Program is exitting, goodbye! - processManager.cc:8

2024-10-30T21:19:44.520Z [CORTEX]:: cortex exited with code: 0 2024-10-30T21:19:44.521Z [CORTEX]:: cortex process is terminated 2024-10-30T21:19:44.555Z [CORTEX]::CPU information - 4 2024-10-30T21:19:44.555Z [CORTEX]:: Request to kill cortex 2024-10-30T21:19:44.558Z [CORTEX]:: cortex process is terminated 2024-10-30T21:19:44.559Z [CORTEX]:: Spawning cortex subprocess... 2024-10-30T21:19:44.559Z [CORTEX] PATH: /usr/bin:/bin:/usr/sbin:/sbin::jan-data-folder/engines/@janhq/inference-cortex-extension/1.0.20:jan-data-folder/extensions/@janhq/inference-cortex-extension/dist/bin/mac-x64:jan-data-folder/extensions/@janhq/inference-cortex-extension/dist/bin/mac-x64 2024-10-30T21:19:44.559Z [CORTEX]::Debug: Cortex engine path: jan-data-folder/extensions/@janhq/inference-cortex-extension/dist/bin/mac-x64 2024-10-30T21:19:44.559Z [CORTEX]:: Spawn cortex at path: jan-data-folder/extensions/@janhq/inference-cortex-extension/dist/bin/mac-x64/cortex-cpp, and args: 1,127.0.0.1,3928 2024-10-30T21:19:44.666Z [CORTEX]:: cortex is ready 2024-10-30T21:19:44.666Z [CORTEX]:: Loading model with params {"cpu_threads":4,"ctx_len":4096,"prompt_template":"<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n{system_message}<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n","llama_model_path":"jan-data-folder/models/llama3.2-3b-instruct/Llama-3.2-3B-Instruct-Q8_0.gguf","ngl":33,"system_prompt":"<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n","user_prompt":"<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n","ai_prompt":"<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n","model":"llama3.2-3b-instruct"} 2024-10-30T21:19:44.669Z [CORTEX]:: 20241030 21:19:44.575440 UTC 1671905 INFO cortex-cpp version: 0.5.0 - main.cc:73 20241030 21:19:44.575998 UTC 1671905 INFO Server started, listening at: 127.0.0.1:3928 - main.cc:78 20241030 21:19:44.575999 UTC 1671905 INFO Please load your model - main.cc:79 20241030 21:19:44.576004 UTC 1671905 INFO Number of thread is:4 - main.cc:86 20241030 21:19:44.668513 UTC 1671909 INFO CPU instruction set: fpu = 1| mmx = 1| sse = 1| sse2 = 1| sse3 = 1| ssse3 = 1| sse4_1 = 1| sse4_2 = 1| pclmulqdq = 1| avx = 1| avx2 = 1| avx512_f = 0| avx512_dq = 0| avx512_ifma = 0| avx512_pf = 0| avx512_er = 0| avx512_cd = 0| avx512_bw = 0| has_avx512_vl = 0| has_avx512_vbmi = 0| has_avx512_vbmi2 = 0| avx512_vnni = 0| avx512_bitalg = 0| avx512_vpopcntdq = 0| avx512_4vnniw = 0| avx512_4fmaps = 0| avx512_vp2intersect = 0| aes = 1| f16c = 1| - server.cc:288 20241030 21:19:44.669369 UTC 1671909 ERROR Could not load engine: Could not load library "jan-data-folder/extensions/@janhq/inference-cortex-extension/dist/bin/mac-x64/engines/cortex.llamacpp/libengine.dylib" dlopen(jan-data-folder/extensions/@janhq/inference-cortex-extension/dist/bin/mac-x64/engines/cortex.llamacpp/libengine.dylib, 0x0006): Symbol not found: (_cblas_sgemm$NEWLAPACK$ILP64) Referenced from: 'jan-data-folder/extensions/@janhq/inference-cortex-extension/dist/bin/mac-x64/engines/cortex.llamacpp/libengine.dylib' Expected in: '/System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate' - server.cc:299

2024-10-30T21:19:44.670Z [CORTEX]:: Load model success with response {} 2024-10-30T21:19:44.671Z [CORTEX]:: Validating model llama3.2-3b-instruct 2024-10-30T21:19:44.673Z [CORTEX]:: Validate model state with response 409 2024-10-30T21:19:44.673Z [CORTEX]:: Validate model state failed with response {"message":"Engine is not loaded yet"} and status is "Conflict" 2024-10-30T21:19:44.674Z [CORTEX]::Error: Validate model status failed

Thank you!

louis-jan commented 1 day ago

cc @vansangpfiev

dlopen(jan-data-folder/extensions/@janhq/inference-cortex-extension/dist/bin/mac-x64/engines/cortex.llamacpp/libengine.dylib, 0x0006): Symbol not found: (_cblas_sgemm$NEWLAPACK$ILP64)