Closed federicoparra closed 7 months ago
The model is IMPRESSIVE. https://huggingface.co/spaces/stabilityai/stablelm-2-1_6b-zephyr
It is VERY WELL trained to follow instructions, like summarization, but because it was trained in several languages it's BRILLIANT at translation (English to Spanish, to French, those I tried because I speak them).
So here we have a model almost as small as Phi-1.5 but that is fine tuned out of the box and multilingual.
We need this :)
@jeethu @Sing-Li I know you already support larger StableLM models. I wonder what it would entail to support this one. Do let me know if I can help in anything, I do need this model running on the Orange Pi!
I took a brief look at it yesterday. The architecture is almost identical to the larger Stable LM 3B model, with only one difference: They've added bias
terms to Q
, K
and V
projections in attention. A bigger concern IMO is the tokenizer. They're using the QWen tokenizer (tiktoken based) and tokenizers-cpp currently only supports Huggingface and sentencepiece based tokenizers.
So in order to support this model, it would entail including support for a new tokenizer (QWen) in tokenizers-cpp, correct? Is this trivial or complex? I'm not proficient in CPP but can still write some CPP.
The good news is that QWen’s tokenizer is already supported in MLC ;))
Ok so that means it is just adding the bias terms to Q, K and V in attention?
@junrushao anyway I can help to bring this model onto the fold? Let me know!
@junrushao @jeethu Hi guys, sorry for bothering you I'm assuming you must be busy. Is there anything I can do to make it work on my own? I really need this for a project I'm working on. thanks a lot!
The good news is that QWen’s tokenizer is already supported in MLC ;))
@junrushao Are you sure is the same? I do not know QWen model deeply but the StableLM seems to be using tiktoken.
super().__init__(errors=errors, **kwargs)
self._tiktoken_config = _arcade100k(vocab_file)
self.tokenizer = tiktoken.Encoding(**self._tiktoken_config)
Hi guys, any news on this by any chance? @jeethu @junrushao
I took a brief look at it yesterday. The architecture is almost identical to the larger Stable LM 3B model, with only one difference: They've added
bias
terms toQ
,K
andV
projections in attention. A bigger concern IMO is the tokenizer. They're using the QWen tokenizer (tiktoken based) and tokenizers-cpp currently only supports Huggingface and sentencepiece based tokenizers.
I managed to enable t stablelm16_model.zip he bias terms in the model (file attached), but since the mlc_chat convert_weight is not working even with StableLM3b I don't know how to proceed any further
hey @federicoparra lets try! why don't you create a fork and a PR? It would be easier to work with it
I see this https://github.com/mlc-ai/mlc-llm/issues/1616 was not yet resolved.
I'm trying first making StableLM 3B work using instructions here https://github.com/mlc-ai/mlc-llm/pull/1008: python build.py --model path/to/stablelm-3b-4e1t --quantization q4f16_1 --target metal --use-safetensors
So far I'm having an error: safetensors_rust.SafetensorError: Error while deserializing header: HeaderTooLarge
I'm trying to figure it out.
Once/if I'm able to get compiled weights/model using this approach (which I had never used before! and doesn't seem to be in the documentation of MLC) then I'll try to do same with the 1.6B model (with the model file altered, as I shared earlier, to have bias for Q, K and V projections in attention.
I'll post here with results...
It was a silly LFS error (not downloading safetensor raw file but just pointer) it should work now I'll report a little later
The build process is ending abruptly on Google Colab, configured with: %pip install --pre -U -f https://mlc.ai/wheels mlc-ai-nightly-cu122 %pip install --pre -U -f https://mlc.ai/wheels mlc-chat-nightly-cu122 mlc-ai-nightly-cu122
Using path "/content/stablelm-3b-4e1t" for model "stablelm-3b-4e1t" Target configured: cuda -keys=cuda,gpu -arch=sm_75 -max_num_threads=1024 -max_shared_memory_per_block=49152 -max_threads_per_block=1024 -registers_per_block=65536 -thread_warp_size=32 Automatically using target for weight quantization: cuda -keys=cuda,gpu -arch=sm_75 -max_num_threads=1024 -max_shared_memory_per_block=49152 -max_threads_per_block=1024 -registers_per_block=65536 -thread_warp_size=32 Get old param: 0% 0/260 [00:00<?, ?tensors/s] Set new param: 0% 0/390 [00:00<?, ?tensors/s]Start computing and quantizing weights... This may take a while. Get old param: 0% 1/260 [00:02<11:09, 2.58s/tensors]^C
I took a brief look at it yesterday. The architecture is almost identical to the larger Stable LM 3B model, with only one difference: They've added
bias
terms toQ
,K
andV
projections in attention. A bigger concern IMO is the tokenizer. They're using the QWen tokenizer (tiktoken based) and tokenizers-cpp currently only supports Huggingface and sentencepiece based tokenizers.
I also don't understand this: if the only difference between a 3B+ nodes model and a 1.6B nodes model would be the adding of 3 nodes (bias), then the resulting variant would be (a bit) larger, not almost half smaller (in terms of nodes) ?
I also don't understand this: if the only difference between a 3B+ nodes model and a 1.6B nodes model would be the adding of 3 nodes (bias), then the resulting variant would be (a bit) larger, not almost half smaller (in terms of nodes) ?
You can have the same, or nearly the same model architecture and have a different number of transformer layers, and the width of hidden and intermediate layers in the MLP. This is also the case with Llama 2 models. Please look at num_hidden_layers
, hidden_size
and intermediate_size
fields in the config.json
file for both StableLM models. They're different (the smaller model has smaller values). I found the difference between the two by loading up the PyTorch models and comparing the layers. Also, unfortunately, I've been way too tied up with other things to take a look at this.
Ok I have sort of good news, bad news folks, @jeethu @DavidGOrtega @junrushao :
First good news: if you open a google colab (with high ram option, you need premium for that) and you do this: !git clone --recursive https://github.com/mlc-ai/mlc-llm.git %pip install --pre -U -f https://mlc.ai/wheels mlc-ai-nightly-cu122 %pip install --pre -U -f https://mlc.ai/wheels mlc-chat-nightly-cu122 mlc-ai-nightly-cu122 !git lfs install !git clone https://huggingface.co/stabilityai/stablelm-2-zephyr-1_6b !python /content/mlc-llm/build.py --model /content/stablelm-2-zephyr-1_6b/ --quantization q4f16_1 --use-safetensors --target llvm
This, strangely, works! without any modification, at least in the sense that it produces the quantized weights and the .so binary file.
Bad news: unfortunately the same on my own Orange Pi 5 targetting Mali does not work:
$python build.py --model ../models/stablelm-2-zephyr-1_6b --quantization q4f16_1 --u
se-safetensors --target mali
Using path "../models/stablelm-2-zephyr-1_6b" for model "stablelm-2-zephyr-1_6b"
Target configured: opencl -keys=mali,opencl,gpu -device=mali -max_function_args=128 -max_num_threads=256 -max_shared_memory_per_block=16384 -max_threads_per_block=256 -texture_spatial_limit=16384 -thread_warp_size=1
arm_release_ver: g13p0-01eac0, rk_so_ver: 3
arm_release_ver of this libmali is 'g6p0-01eac0', rk_so_ver is '7'.
Automatically using target for weight quantization: opencl -keys=opencl,gpu -max_function_args=128 -max_num_threads=256 -max_shared_memory_per_block=16384 -max_threads_per_block=256 -texture_spatial_limit=16384 -thread_warp_size=1
Get old param: 0%| | 0/196 [00:00<?, ?tensors/sTraceback (most recent call last): | 0/294 [00:00<?, ?tensors/s]
File "/home/federico/Documents/code/mlc-llm/build.py", line 4, in scale
is directly accessed by host memory (it is not contained in a thread environment or in the function arguments.
Variable A
is directly accessed by host memory (it is not contained in a thread environment or in the function arguments.
Variable w_gathered
is directly accessed by host memory (it is not contained in a thread environment or in the function arguments.
Variable w_gathered
is directly accessed by host memory (it is not contained in a thread environment or in the function arguments.
Variable w_gathered
is directly accessed by host memory (it is not contained in a thread environment or in the function arguments.
Variable scale
is directly accessed by host memory (it is not contained in a thread environment or in the function arguments.
Variable A
is directly accessed by host memory (it is not contained in a thread environment or in the function arguments.
File "/home/federico/Documents/code/tvm-unity/src/tir/analysis/verify_memory.cc", line 205
RuntimeError: Memory verification failed with the following errors:
@T.prim_func def encode4(A: T.Buffer((2048, 5632), "float16"), w_gathered: T.Buffer((2048, 704), "uint32"), scale: T.Buffer((2048, 176), "float16")): T.func_attr({"target": T.target({"host": {"keys": ["cpu"], "kind": "llvm", "tag": ""}, "keys": ["opencl", "gpu"], "kind": "opencl", "max_function_args": 128, "max_num_threads": 256, "max_shared_memory_per_block": 16384, "max_threads_per_block": 256, "tag": "", "texture_spatial_limit": 16384, "thread_warp_size": 1}), "tir.is_scheduled": T.bool(True), "tir.noalias": T.bool(True)}) max_abs_value = T.allocate([360448], "float16", "global") max_abs_value_1 = T.Buffer((360448,), "float16", data=max_abs_value) A_1 = T.Buffer((11534336,), "float16", data=A.data) for i, j, k in T.grid(2048, 176, 32): cse_var_1: T.int32 = i 176 + j if k == 0: max_abs_value_1[cse_var_1] = T.float16(-65504) max_abs_value_1[cse_var_1] = T.max(max_abs_value_1[cse_var_1], T.fabs(A_1[i 5632 + j 32 + k])) scale_1 = T.Buffer((360448,), "float16", data=scale.data) for i, j in T.grid(2048, 176): cse_var_2: T.int32 = i 176 + j scale_1[cse_var_2] = T.max(max_abs_value_1[cse_var_2], T.float16(0.0001)) T.float16(0.14285714285714285) for i, j, k in T.grid(2048, 704, 8): cse_var_3: T.int32 = i 704 + j w_gathered_1 = T.Buffer((1441792,), "uint32", data=w_gathered.data) if k == 0: w_gathered_1[cse_var_3] = T.uint32(0) w_gathered_1[cse_var_3] = T.bitwise_or(w_gathered_1[cse_var_3], T.shift_left(T.Cast("uint32", T.min(T.max(T.round(A_1[i 5632 + j 8 + k] / scale_1[i 176 + j // 4] + T.float16(7)), T.float16(0)), T.float16(14))), T.Cast("uint32", k) T.uint32(4)))
If I repeat the above procedure, in colab and using cuda as target when building, and then, in colab, cmake mlc-llm and use mlc_chat_cli...
Use MLC config: "/content/mlc-llm/build/dist/stablelm-2-zephyr-1_6b-q4f16_1/params/mlc-chat-config.json"
Use model weights: "/content/mlc-llm/build/dist/stablelm-2-zephyr-1_6b-q4f16_1/params/ndarray-cache.json"
Use model library: "/content/mlc-llm/build/dist/stablelm-2-zephyr-1_6b-q4f16_1/stablelm-2-zephyr-1_6b-q4f16_1-cuda.so"
You can use the following special commands:
/help print the special commands
/exit quit the cli
/stats print out the latest stats (token/sec)
/reset restart a fresh chat
/reload [model] reload model model
from disk, or reload the current model if model
is not specified
Loading model... [21:37:57] /content/mlc-llm/cpp/tokenizers.cc:81: Cannot find any tokenizer under: /content/mlc-llm/build/dist/stablelm-2-zephyr-1_6b-q4f16_1/params Stack trace: [bt] (0) /content/mlc-llm/build/tvm/libtvm_runtime.so(tvm::runtime::Backtrace[abi:cxx11]()+0x2c) [0x7b2b47212f1c] [bt] (1) /content/mlc-llm/build/mlc_chat_cli(tvm::runtime::detail::LogFatal::Entry::Finalize()+0x3b) [0x55fa53c9fe0b] [bt] (2) /content/mlc-llm/build/libmlc_llm.so(+0x2bc2d6) [0x7b2b4764b2d6] [bt] (3) /content/mlc-llm/build/libmlc_llm.so(mlc::llm::Tokenizer::FromPath(tvm::runtime::String const&)+0x8c1) [0x7b2b4764bba1] [bt] (4) /content/mlc-llm/build/libmlc_llm.so(mlc::llm::LLMChat::Reload(tvm::runtime::TVMArgValue, tvm::runtime::String, tvm::runtime::String)+0x598) [0x7b2b475c3ad8] [bt] (5) /content/mlc-llm/build/libmlc_llm.so(+0x23783c) [0x7b2b475c683c] [bt] (6) /content/mlc-llm/build/mlc_chat_cli(+0x141cc) [0x55fa53ca31cc] [bt] (7) /content/mlc-llm/build/mlc_chat_cli(+0xdddf) [0x55fa53c9cddf] [bt] (8) /content/mlc-llm/build/mlc_chat_cli(+0x99f9) [0x55fa53c989f9]
AMAZING! I'll try
Works great! Compiled the branch and tried it, a really nice conversation in Spanish! Go MLC! 👏👏👏
⚙️ Request New Models
https://stability.ai/news/introducing-stable-lm-2
Model weights: https://huggingface.co/stabilityai/stablelm-2-zephyr-1_6b
Just came out! New version of StableLM with only 1.6b nodes, and MULTILINGUAL capacity, that in theory outperforms Phi-1.5 (and approaches Mistral 7B in other languages) !!!