ValueError after following llm-mlc install instructions

[X52p]

I followed these steps:

pipx install llm
llm install llm-mlc
llm mlc pip install --pre --force-reinstall mlc-ai-nightly mlc-chat-nightly -f https://mlc.ai/wheels
llm mlc setup
llm mlc download-model Llama-2-13b-chat --alias llama2
llm -m llama2 'difference between a llama and an alpaca'

This results in this error:

Error: Traceback (most recent call last):
  File "D:\a\package\package\tvm\src\runtime\relax_vm\ndarray_cache_support.cc", line 262
ValueError: Check failed: shard_rec.nbytes == raw_data.length() (81920000 vs. 133) : Parameters are not loaded properly. Please check your parameter shards and git lfs installation

I also tried other models (for example Llama-2-7b-chat) resulting in the same error.

Software:

• Windows 10
• python 3.12.0 and python 3.9 (I tried both versions)

Hardware:

• AMD Ryzen1950X
• GTX 1080 Ti

Am I missing a step? Is my hardware incompatible?

[rupurt]

Running into a similar error

ValueError: Encountered an corrupted parameter shard. It means it is not downloaded completely or downloading is interrupted. Please try to download again.

[emmceemoore]

I believe your issue is related to git LFS. I was getting a similar error and noticed that the shards were much smaller than expected (per the error message 😅). I checked out the shards and they looked like this:

cat ~/Library/Application Support/io.datasette.llm/mlc/dist/prebuilt/mlc-chat-Llama-2-70b-chat-hf-q4f16_1/params_shard_99.bin

version https://git-lfs.github.com/spec/v1
oid sha256:a08539a7c31e9ad931f2c4bdb913e8542c2feaa801168af6950890e278f95a08
size 234881024

For me, the solution was to set programs.git.lfs.enable to true in my nix config, remove the directory, and re-download the model.

You may want to check your git/LFS settings to see if they're setup so that the shards (and not just pointers to the shards) are being downloaded.

[rupurt]

Thank you @emmceemoore. Combined with my fix here that did the trick!