bug: The model can't start / The specified module could not be found

[mike2003]

• [X] I have searched the existing issues

Current behavior

The model can't start In log "The specified module could not be found." But i have c:\Users\mik\AppData\Roaming\Jan\data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu\engines\cortex.llamacpp\engine.dll

Minimum reproduction step

1. Download JAN
2. Start
3. Select llama 3.1 instruct for download
4. wait
5. Start descusion

Expected behavior

The conversation has begun.

Screenshots / Logs

20240904 14:15:26.852000 UTC 19800 INFO Server started, listening at: 127.0.0.1:3928 - main.cc:78 20240904 14:15:26.852000 UTC 19800 INFO Please load your model - main.cc:79 20240904 14:15:26.852000 UTC 19800 INFO Number of thread is:12 - main.cc:86 20240904 14:15:26.948000 UTC 23908 INFO CPU instruction set: fpu = 1| mmx = 1| sse = 1| sse2 = 1| sse3 = 1| ssse3 = 1| sse4_1 = 1| sse4_2 = 1| pclmulqdq = 1| avx = 1| avx2 = 1| avx512_f = 1| avx512_dq = 1| avx512_ifma = 1| avx512_pf = 0| avx512_er = 0| avx512_cd = 1| avx512_bw = 1| has_avx512_vl = 1| has_avx512_vbmi = 1| has_avx512_vbmi2 = 1| avx512_vnni = 1| avx512_bitalg = 1| avx512_vpopcntdq = 1| avx512_4vnniw = 0| avx512_4fmaps = 0| avx512_vp2intersect = 0| aes = 1| f16c = 1| - server.cc:288 20240904 14:15:26.949000 UTC 23908 ERROR Could not load engine: Could not load library "C:\Users\mik\AppData\Roaming\Jan\data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu/engines/cortex.llamacpp/engine.dll" The specified module could not be found.

• server.cc:299

2024-09-04T14:15:26.951Z [CORTEX]::Debug: Load model success with response {} 2024-09-04T14:15:26.951Z [CORTEX]::Debug: Validating model llama3.1-8b-instruct 2024-09-04T14:15:26.953Z [CORTEX]::Debug: Validate model state with response 409 2024-09-04T14:15:26.954Z [CORTEX]::Error: Validate model status failed 2024-09-04T14:15:26.954Z [CORTEX]::Debug: Validate model state failed with response {"message":"Engine is not loaded yet"} and status is "Conflict"

Jan version

jan-win-x64-0.5.3

In which operating systems have you tested?

• [ ] macOS
• [X] Windows
• [ ] Linux

Environment details

Win 11 x64

[mike2003]

data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu/engines/cortex.llamacpp/engine.dll"

Why are the slashes going in different directions????

[dan-homebrew]

@louis-jan Can you take a look at this - if this is a Windows/Linux file path issue, this means that we're breaking on all Windows right now...

[vansangpfiev]

"C:\Users\mik\AppData\Roaming\Jan\data\extensions@janhq\inference-cortex-extension\dist\bin\win-cpu/

It is some logging format issue. The specified module could not be found. means dll dependencies were missed. @mike2003 Could you please list all the files in C:\Users\mik\AppData\Roaming\Jan\data\extensions@janhq\inference-cortex-extension\dist\bin\win-cpu\ directory?

[mike2003]

c:\Users\mik\AppData\Roaming\Jan\data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu\cortex-cpp.exe
c:\Users\mik\AppData\Roaming\Jan\data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu\vcruntime140.dll
c:\Users\mik\AppData\Roaming\Jan\data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu\engines\cortex.llamacpp\vcruntime140.dll
c:\Users\mik\AppData\Roaming\Jan\data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu\vcruntime140_1.dll
c:\Users\mik\AppData\Roaming\Jan\data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu\engines\cortex.llamacpp\vcruntime140_1.dll
c:\Users\mik\AppData\Roaming\Jan\data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu\engines\cortex.llamacpp\vcomp140.dll
c:\Users\mik\AppData\Roaming\Jan\data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu\msvcp140.dll
c:\Users\mik\AppData\Roaming\Jan\data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu\engines\cortex.llamacpp\msvcp140.dll
c:\Users\mik\AppData\Roaming\Jan\data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu\engines\cortex.llamacpp\engine.dll

[vansangpfiev]

c:\Users\mik\AppData\Roaming\Jan\data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu\cortex-cpp.exe
c:\Users\mik\AppData\Roaming\Jan\data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu\vcruntime140.dll
c:\Users\mik\AppData\Roaming\Jan\data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu\engines\cortex.llamacpp\vcruntime140.dll
c:\Users\mik\AppData\Roaming\Jan\data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu\vcruntime140_1.dll
c:\Users\mik\AppData\Roaming\Jan\data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu\engines\cortex.llamacpp\vcruntime140_1.dll
c:\Users\mik\AppData\Roaming\Jan\data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu\engines\cortex.llamacpp\vcomp140.dll
c:\Users\mik\AppData\Roaming\Jan\data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu\msvcp140.dll
c:\Users\mik\AppData\Roaming\Jan\data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu\engines\cortex.llamacpp\msvcp140.dll
c:\Users\mik\AppData\Roaming\Jan\data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu\engines\cortex.llamacpp\engine.dll

Seems like we did not copy vcomp140.dll file to execution file directory. Could you please copy

c:\Users\mik\AppData\Roaming\Jan\data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu\engines\cortex.llamacpp\vcomp140.dll

into c:\Users\mik\AppData\Roaming\Jan\data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu\ and try again?

[louis-jan]

c:\Users\mik\AppData\Roaming\Jan\data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu\cortex-cpp.exe
c:\Users\mik\AppData\Roaming\Jan\data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu\vcruntime140.dll
c:\Users\mik\AppData\Roaming\Jan\data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu\engines\cortex.llamacpp\vcruntime140.dll
c:\Users\mik\AppData\Roaming\Jan\data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu\vcruntime140_1.dll
c:\Users\mik\AppData\Roaming\Jan\data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu\engines\cortex.llamacpp\vcruntime140_1.dll
c:\Users\mik\AppData\Roaming\Jan\data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu\engines\cortex.llamacpp\vcomp140.dll
c:\Users\mik\AppData\Roaming\Jan\data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu\msvcp140.dll
c:\Users\mik\AppData\Roaming\Jan\data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu\engines\cortex.llamacpp\msvcp140.dll
c:\Users\mik\AppData\Roaming\Jan\data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu\engines\cortex.llamacpp\engine.dll

Seems like we did not copy vcomp140.dll file to execution file directory. Could you please copy

c:\Users\mik\AppData\Roaming\Jan\data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu\engines\cortex.llamacpp\vcomp140.dll

into c:\Users\mik\AppData\Roaming\Jan\data\extensions\@janhq\inference-cortex-extension\dist\bin\win-cpu\ and try again?

Hi @dan-homebrew @vansangpfiev, from this clue, I did try to investigate from the extension bundle and it turned out that we've missed copying the DLLs from the engine to the cortex-cpp folder that lead to the issue. Not all of the users' machines have that DLL file. That affect all of the window users that do not have Visual C++ Redistributable installed (cmmiw)

Let me try to articulate my findings:

• The vcomp140.dll file is required for the latest engine update (llama.cpp v01.25). The file is shipped along with the engine, but it is located within the engine folder, which does not have any effect.
• Our legacy Cortex bundle script from Jan assumes that all DLL files are bundled inside cortex-cpp.
• cortex-cpp will just work with dll files placed in the cortex-cpp folder.

There are two possible fixes:

• Jan copies DLL files when decompressing the engine (the extension's build script). A hot fix from Jan's extension.
• The engine can load DLL files in place or Cortex-CPP bundles vcomp140.dll. Neither is an easy fix.

cc @hiento09

Update from @vansangpfiev:

• vcomp140.dll file missing is a legacy issue

[mike2003]

Good! Started!

[dan-homebrew]

@louis-jan @vansangpfiev Great job - nice find!

@vansangpfiev Will our revised approach to cortex.cpp bundling dependencies in each engine, solve for this?

• i.e. discussion in https://github.com/janhq/cortex.cpp/issues/1046
• llama.cpp will bundle all C++ dependencies, including vcomp140.dll

Or will we need to copy all Engine .dll dependencies to the main Jan execution file folder?

[vansangpfiev]

@louis-jan @vansangpfiev Great job - nice find!

@vansangpfiev Will our revised approach to cortex.cpp bundling dependencies in each engine, solve for this?

• i.e. discussion in Discussion: Cortex.cpp Engine Dependencies Architecture (e.g. CUDA Toolkit) cortex.cpp#1046
• llama.cpp will bundle all C++ dependencies, including vcomp140.dll

Or will we need to copy all Engine .dll dependencies to the main Jan execution file folder?

Our approach to bundle dependencies will solve this problem. However, I would recommend a hot fix as Louis suggested for a short term solution.

[dan-homebrew]

@louis-jan Can we queue this hotfix for Sprint 19, to be shipped with the next release? (be it v0.5.4 or v0.6.0)

[louis-jan]

@dan-homebrew. Roger that!

[benapetr]

this is closed but I am still hitting this in 0.5.4, is there any known workaround? do I need to just install the VC redist?

[louis-jan]

Hi @benapetr could you please share your device specs and the app log?

[benapetr]

I figured it out, it was missing CUDA 12.0 libraries, but the error log was identical to first post here.

It would be great if the error message contained specific reason why it failed and not just "error", perhaps report which dll / dependency is missing?

[louis-jan]

I figured it out, it was missing CUDA 12.0 libraries, but the error log was identical to first post here.

It would be great if the error message contained specific reason why it failed and not just "error", perhaps report which dll / dependency is missing?

Hi @benapetr, I think there should be an error message like this displayed, along with a further guidance message. This is about the UX level where it displays a proper message to UI.

From the log level, where we're working on improvements, to providing graceful messages.