EricLBuehler
commented
2 months ago Hey @Remember20240719! Can you please run with RUST_BACKTRACE=1?
Closed Remember20240719 closed 2 months ago
EricLBuehler
commented
2 months ago Hey @Remember20240719! Can you please run with RUST_BACKTRACE=1?
EricLBuehler
commented
2 months ago @Remember20240719 fixed it in #657! Please confirm that it works now after git pull.
EricLBuehler
commented
2 months ago @Remember20240719 closing as complete via #657.
Remember20240719
commented
2 months ago That was quick, thanks!
This time, the command runs normally, but crashes when given a prompt:
git HEAD a9b8b2ec4af6a7aafee952a2e35d05281fcd83e0
RUST_BACKTRACE=1 ./target/release/./mistralrs-server -i gguf --quantized-model-id $D/models/Mistral-Nemo-Instruct-2407-GGUF --quantized-filename Mistral-Nemo-Instruct-2407-Q4_K_L.gguf
2024-08-02T02:19:34.960621Z INFO mistralrs_server: avx: true, neon: false, simd128: false, f16c: true
2024-08-02T02:19:34.960657Z INFO mistralrs_server: Sampling method: penalties -> temperature -> topk -> topp -> minp -> multinomial
2024-08-02T02:19:34.960674Z INFO mistralrs_server: Model kind is: quantized from gguf (no adapters)
2024-08-02T02:19:34.960804Z INFO mistralrs_core::pipeline::paths: Loading `Mistral-Nemo-Instruct-2407-Q4_K_L.gguf` locally at `$D/models/Mistral-Nemo-Instruct-2407-GGUF/Mistral-Nemo-Instruct-2407-Q4_K_L.gguf`
2024-08-02T02:19:34.960895Z INFO mistralrs_core::pipeline::gguf: Loading model `$D/models/Mistral-Nemo-Instruct-2407-GGUF` on cpu.
2024-08-02T02:19:35.540106Z INFO mistralrs_core::pipeline::gguf: Model config:
general.architecture: llama
general.basename: Mistral-Nemo
general.file_type: 15
general.finetune: Instruct
general.languages: en, fr, de, es, it, pt, ru, zh, ja
general.license: apache-2.0
general.name: Mistral Nemo Instruct 2407
general.quantization_version: 2
general.size_label: 12B
general.type: model
general.version: 2407
llama.attention.head_count: 32
llama.attention.head_count_kv: 8
llama.attention.key_length: 128
llama.attention.layer_norm_rms_epsilon: 0.00001
llama.attention.value_length: 128
llama.block_count: 40
llama.context_length: 1024000
llama.embedding_length: 5120
llama.feed_forward_length: 14336
llama.rope.dimension_count: 128
llama.rope.freq_base: 1000000
llama.vocab_size: 131072
quantize.imatrix.chunks_count: 128
quantize.imatrix.dataset: /training_dir/calibration_datav3.txt
quantize.imatrix.entries_count: 280
quantize.imatrix.file: /models_out/Mistral-Nemo-Instruct-2407-GGUF/Mistral-Nemo-Instruct-2407.imatrix
2024-08-02T02:19:35.540602Z INFO mistralrs_core::pipeline::gguf: Debug is enabled, wrote the names and information about each tensor to `mistralrs_gguf_tensors.txt`.
2024-08-02T02:19:35.874385Z INFO mistralrs_core::gguf::gguf_tokenizer: GGUF tokenizer model is `gpt2`, kind: `Bpe`, num tokens: 131072, num added tokens: 0, num merges: 269443, num scores: 0
2024-08-02T02:19:35.874402Z INFO mistralrs_core::gguf::gguf_tokenizer: Tokenizer: Tokenizer(TokenizerImpl { normalizer: None, pre_tokenizer: Some(ByteLevel(ByteLevel { add_prefix_space: false, trim_offsets: true, use_regex: true })), model: BPE(BPE { dropout: None, unk_token: Some("<unk>"), continuing_subword_prefix: None, end_of_word_suffix: None, fuse_unk: false, byte_fallback: false, vocab: 131072, merges: 269443, ignore_merges: false }), post_processor: Some(Template(TemplateProcessing { single: Template([SpecialToken { id: "<s>", type_id: 0 }, Sequence { id: A, type_id: 0 }]), pair: Template([SpecialToken { id: "<s>", type_id: 0 }, Sequence { id: A, type_id: 0 }, Sequence { id: B, type_id: 1 }]), added_single: 1, added_pair: 1, special_tokens: Tokens({"<s>": SpecialToken { id: "<s>", ids: [1], tokens: ["<s>"] }}) })), decoder: Some(ByteLevel(ByteLevel { add_prefix_space: true, trim_offsets: true, use_regex: true })), added_vocabulary: AddedVocabulary { added_tokens_map: {"<s>": 1, "</s>": 2, "<unk>": 0}, added_tokens_map_r: {0: AddedToken { content: "<unk>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true }, 2: AddedToken { content: "</s>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true }, 1: AddedToken { content: "<s>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true }}, added_tokens: [], special_tokens: [AddedToken { content: "<s>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true }, AddedToken { content: "</s>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true }, AddedToken { content: "<unk>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true }], special_tokens_set: {"</s>", "<s>", "<unk>"}, split_trie: (AhoCorasick(dfa::DFA(
D 000000: \x00-\x0E => 0
F 000016:
* 000032: \x00-\x0E => 0
matches: 1
* 000048: \x00-\x0E => 0
matches: 2
* 000064: \x00-\x0E => 0
matches: 0
>000080: \x00-\x02 => 80, \x03 => 208, \x04-\x0E => 80
000096: \x00-\x02 => 0, \x03 => 208, \x04-\x0E => 0
000112: \x00-\x02 => 80, \x03 => 208, \x04-\n => 80, \x0B => 128, \x0C-\x0E => 80
000128: \x00-\x02 => 80, \x03 => 208, \x04 => 80, \x05 => 32, \x06-\x0E => 80
000144: \x00-\x02 => 80, \x03 => 208, \x04 => 80, \x05 => 64, \x06-\x0E => 80
000160: \x00-\x02 => 80, \x03 => 208, \x04-\x08 => 80, \t => 176, \n-\x0E => 80
000176: \x00-\x02 => 80, \x03 => 208, \x04-\x06 => 80, \x07 => 192, \x08-\x0E => 80
000192: \x00-\x02 => 80, \x03 => 208, \x04 => 80, \x05 => 48, \x06-\x0E => 80
000208: \x00 => 80, \x01 => 112, \x02 => 80, \x03 => 208, \x04-\n => 80, \x0B => 144, \x0C => 80, \r => 160, \x0E => 80
match kind: LeftmostLongest
prefilter: true
state length: 14
pattern length: 3
shortest pattern length: 3
longest pattern length: 5
alphabet length: 15
stride: 16
byte classes: ByteClasses(0 => [0-46], 1 => [47], 2 => [48-59], 3 => [60], 4 => [61], 5 => [62], 6 => [63-106], 7 => [107], 8 => [108-109], 9 => [110], 10 => [111-114], 11 => [115], 12 => [116], 13 => [117], 14 => [118-255])
memory usage: 992
)
), [1, 2, 0]), split_normalized_trie: (AhoCorasick(dfa::DFA(
D 000000: \x00 => 0
F 000001:
>000002: \x00 => 2
000003: \x00 => 0
match kind: LeftmostLongest
prefilter: false
state length: 4
pattern length: 0
shortest pattern length: 18446744073709551615
longest pattern length: 0
alphabet length: 1
stride: 1
byte classes: ByteClasses(0 => [0-255])
memory usage: 16
)
), []), encode_special_tokens: false }, truncation: None, padding: None })
2024-08-02T02:19:35.897140Z INFO mistralrs_core::gguf::chat_template: Discovered and using GGUF chat template: `{%- if messages[0]['role'] == 'system' %}\n {%- set system_message = messages[0]['content'] %}\n {%- set loop_messages = messages[1:] %}\n{%- else %}\n {%- set loop_messages = messages %}\n{%- endif %}\n\n{{- bos_token }}\n{%- for message in loop_messages %}\n {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}\n {{- raise_exception('After the optional system message, conversation roles must alternate user/assistant/user/assistant/...') }}\n {%- endif %}\n {%- if message['role'] == 'user' %}\n {%- if loop.last and system_message is defined %}\n {{- '[INST] ' + system_message + '\n\n' + message['content'] + '[/INST]' }}\n {%- else %}\n {{- '[INST] ' + message['content'] + '[/INST]' }}\n {%- endif %}\n {%- elif message['role'] == 'assistant' %}\n {{- ' ' + message['content'] + eos_token}}\n {%- else %}\n {{- raise_exception('Only user and assistant roles are supported, with the exception of an initial optional system message!') }}\n {%- endif %}\n{%- endfor %}\n`
2024-08-02T02:20:29.804507Z INFO mistralrs_core::pipeline::paths: Using literal chat template.
2024-08-02T02:20:30.167965Z INFO mistralrs_core::pipeline::chat_template: bos_toks = "<s>", eos_toks = "</s>", unk_tok = <unk>
2024-08-02T02:20:30.194591Z INFO mistralrs_server: Model loaded.
2024-08-02T02:20:30.194725Z INFO mistralrs_server::interactive_mode: Starting interactive loop with sampling params: SamplingParams { temperature: Some(0.1), top_k: Some(32), top_p: Some(0.1), min_p: Some(0.05), top_n_logprobs: 0, frequency_penalty: Some(0.1), presence_penalty: Some(0.1), stop_toks: None, max_len: Some(4096), logits_bias: None, n_choices: 1 }
> Dags. Do you like dags?
2024-08-02T02:20:44.803643Z ERROR mistralrs_core::engine: prompt step - Model failed with error: WithBacktrace { inner: ShapeMismatchBinaryOp { lhs: [1, 71, 32, 128], rhs: [1, 71, 5120], op: "reshape" }, backtrace: Backtrace [{ fn: "candle_core::error::Error::bt" }, { fn: "candle_core::tensor::Tensor::reshape" }, { fn: "mistralrs_core::models::quantized_llama::ModelWeights::forward" }, { fn: "<mistralrs_core::pipeline::gguf::GGUFPipeline as mistralrs_core::pipeline::Pipeline>::forward_inputs" }, { fn: "mistralrs_core::pipeline::Pipeline::step::{{closure}}" }, { fn: "mistralrs_core::engine::Engine::run::{{closure}}" }, { fn: "std::sys_common::backtrace::__rust_begin_short_backtrace" }, { fn: "core::ops::function::FnOnce::call_once{{vtable.shim}}" }, { fn: "std::sys::pal::unix::thread::Thread::new::thread_start" }, { fn: "start_thread" }, { fn: "__GI___clone3" }] }
2024-08-02T02:20:44.804331Z ERROR mistralrs_server::interactive_mode: Got a model error: "shape mismatch in reshape, lhs: [1, 71, 32, 128], rhs: [1, 71, 5120]
0: candle_core::error::Error::bt
1: candle_core::tensor::Tensor::reshape
2: mistralrs_core::models::quantized_llama::ModelWeights::forward
3: <mistralrs_core::pipeline::gguf::GGUFPipeline as mistralrs_core::pipeline::Pipeline>::forward_inputs
4: mistralrs_core::pipeline::Pipeline::step::{{closure}}
5: mistralrs_core::engine::Engine::run::{{closure}}
6: std::sys_common::backtrace::__rust_begin_short_backtrace
7: core::ops::function::FnOnce::call_once{{vtable.shim}}
8: std::sys::pal::unix::thread::Thread::new::thread_start
9: start_thread
10: __GI___clone3", response: ChatCompletionResponse { id: "0", choices: [Choice { finish_reason: "error", index: 0, message: ResponseMessage { content: Some(""), role: "assistant", tool_calls: [] }, logprobs: None }], created: 1722565244, model: "$D/models/Mistral-Nemo-Instruct-2407-GGUF", system_fingerprint: "local", object: "chat.completion", usage: Usage { completion_tokens: 0, prompt_tokens: 71, total_tokens: 71, avg_tok_per_sec: 1203.3899, avg_prompt_tok_per_sec: inf, avg_compl_tok_per_sec: NaN, total_time_sec: 0.059, total_prompt_time_sec: 0.0, total_completion_time_sec: 0.0 } }
Describe the bug
Hello, I found in PR #595 that Mistral Nemo Instruct 2407 was supported. It is working really well (using ISQ on HF safetensors).
Are GGUF supported too?
Using the Q8_0 from https://huggingface.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF/tree/main:
Server:
Client:
Latest commit or version
Latest commit 38fb9423cb30a996dbc991a2294ace97443008ae