search

lmstudio-ai / mlx-engine

👾🍎 Apple MLX engine for LM Studio
MIT License
224 stars 22 forks source link

Error (Unknown ArrayValue filter: rejectattr) when trying to generate from a 4-bit quantized MLX model #21

Open chimezie opened 3 weeks ago

chimezie commented 3 weeks ago

I'm running LM Studio verison 0.3.5. on an Apple M1 Ultra with 32 GB of memory.

As mentioned in Discord, I have a model of my own finetuned from Nemo that was quantized (after fusing the LoRA adapter) using mlx-lm, ensuring I was using v4.45.2 of transformers in the command line. I was able to generate from it in the command line using demo.py of mlx-engine (after a few changes to sync it with caching updates to mlx: https://github.com/lmstudio-ai/mlx-engine/pull/20 )

However, when I try to load the quantized model, I get the following error:

2024-10-30 08:11:25 [DEBUG]
1 Error predicting: Error: Unknown ArrayValue filter: rejectattr
    at te.evaluateFilterExpression (/Applications/LM Studio.app/Contents/Resources/app/.webpack/main/llmworker.js:28:113250)
    at te.evaluate (/Applications/LM Studio.app/Contents/Resources/app/.webpack/main/llmworker.js:28:120023)
    at te.evaluateFilterExpression (/Applications/LM Studio.app/Contents/Resources/app/.webpack/main/llmworker.js:28:110233)
    at te.evaluate (/Applications/LM Studio.app/Contents/Resources/app/.webpack/main/llmworker.js:28:120023)
    at te.evaluateFilterExpression (/Applications/LM Studio.app/Contents/Resources/app/.webpack/main/llmworker.js:28:110233)
    at te.evaluate (/Applications/LM Studio.app/Contents/Resources/app/.webpack/main/llmworker.js:28:120023)
    at te.evaluateFor (/Applications/LM Studio.app/Contents/Resources/app/.webpack/main/llmworker.js:28:117002)
    at te.evaluate (/Applications/LM Studio.app/Contents/Resources/app/.webpack/main/llmworker.js:28:119122)
    at te.evaluateBlock (/Applications/LM Studio.app/Contents/Resources/app/.webpack/main/llmworker.js:28:114336)
    at te.evalProgram (/Applications/LM Studio.app/Contents/Resources/app/.webpack/main/llmworker.js:28:114253)
neilmehta24 commented 3 weeks ago

If the demo.py generation works for your quantized model, I expect that the next release of LM Studio will resolve this issue. Expect a new release here soon https://lmstudio.ai/beta-releases

chimezie commented 3 weeks ago

Excellent. Looking forward to it