Closed 17Reset closed 2 months ago
Please describe all the steps you have followed to help to reproduce your issue, clarify which llama.cpp release you are using and attach output and llama.log.
I am using the latest version of llamacpp and here is the output of my usage:
ubuntu@ubuntu:/mnt/llm/llm_toolkit$ ./main --model ../llm_quantized/smaug2_72b_q8.gguf --n-gpu-layers 81 --ctx-size 4096 --keep -1 --multiline-input --prompt "Analyze and list the global performance advantages of the U.S. Navy's AN/SPY-7 series of radars, focusing on their unique technical characteristics and performance indicators, such as detection range, accuracy, and anti-jamming capability, and compare them with existing similar radar systems from other countries to highlight the advancements of AN/SPY-7.
Please provide a detailed description, highlighting the advantages of the AN/SPY-7 series of radars in terms of detection range, accuracy and anti-jamming capability, and comparing it with similar radar systems in other countries.
Your answer should include the unique technical characteristics and performance indicators of the AN/SPY-7 series radar, as well as advantages over other radar systems. Please focus on describing the global performance benefits of AN/SPY-7 series radars and provide specific and detailed information."
Log start
main: build = 2678 (132f5579)
main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu
main: seed = 1713237780
llama_model_loader: loaded meta data with 23 key-value pairs and 963 tensors from ../llm_quantized/smaug2_72b_q8.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.name str = .
llama_model_loader: - kv 2: llama.vocab_size u32 = 152064
llama_model_loader: - kv 3: llama.context_length u32 = 32768
llama_model_loader: - kv 4: llama.embedding_length u32 = 8192
llama_model_loader: - kv 5: llama.block_count u32 = 80
llama_model_loader: - kv 6: llama.feed_forward_length u32 = 24576
llama_model_loader: - kv 7: llama.rope.dimension_count u32 = 128
llama_model_loader: - kv 8: llama.attention.head_count u32 = 64
llama_model_loader: - kv 9: llama.attention.head_count_kv u32 = 64
llama_model_loader: - kv 10: llama.attention.layer_norm_rms_epsilon f32 = 0.000001
llama_model_loader: - kv 11: llama.rope.freq_base f32 = 1000000.000000
llama_model_loader: - kv 12: general.file_type u32 = 7
llama_model_loader: - kv 13: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 14: tokenizer.ggml.tokens arr[str,152064] = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 15: tokenizer.ggml.scores arr[f32,152064] = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv 16: tokenizer.ggml.token_type arr[i32,152064] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 17: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv 18: tokenizer.ggml.bos_token_id u32 = 151644
llama_model_loader: - kv 19: tokenizer.ggml.eos_token_id u32 = 151645
llama_model_loader: - kv 20: tokenizer.ggml.padding_token_id u32 = 151643
llama_model_loader: - kv 21: tokenizer.chat_template str = {% for message in messages %}{% if lo...
llama_model_loader: - kv 22: general.quantization_version u32 = 2
llama_model_loader: - type f32: 402 tensors
llama_model_loader: - type q8_0: 561 tensors
llm_load_vocab: special tokens definition check successful ( 421/152064 ).
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = BPE
llm_load_print_meta: n_vocab = 152064
llm_load_print_meta: n_merges = 151387
llm_load_print_meta: n_ctx_train = 32768
llm_load_print_meta: n_embd = 8192
llm_load_print_meta: n_head = 64
llm_load_print_meta: n_head_kv = 64
llm_load_print_meta: n_layer = 80
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_embd_head_k = 128
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 1
llm_load_print_meta: n_embd_k_gqa = 8192
llm_load_print_meta: n_embd_v_gqa = 8192
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-06
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 24576
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx = 32768
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: model type = 65B
llm_load_print_meta: model ftype = Q8_0
llm_load_print_meta: model params = 72.29 B
llm_load_print_meta: model size = 74.95 GiB (8.91 BPW)
llm_load_print_meta: general.name = .
llm_load_print_meta: BOS token = 151644 '<|im_start|>'
llm_load_print_meta: EOS token = 151645 '<|im_end|>'
llm_load_print_meta: PAD token = 151643 '<|endoftext|>'
llm_load_print_meta: LF token = 148848 'ÄĬ'
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: CUDA_USE_TENSOR_CORES: yes
ggml_cuda_init: found 2 CUDA devices:
Device 0: NVIDIA RTX 6000 Ada Generation, compute capability 8.9, VMM: yes
Device 1: NVIDIA RTX 6000 Ada Generation, compute capability 8.9, VMM: yes
llm_load_tensors: ggml ctx size = 1.10 MiB
llm_load_tensors: offloading 80 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 81/81 layers to GPU
llm_load_tensors: CPU buffer size = 1262.25 MiB
llm_load_tensors: CUDA0 buffer size = 36250.41 MiB
llm_load_tensors: CUDA1 buffer size = 39234.12 MiB
..............................................................................................
llama_new_context_with_model: n_ctx = 4096
llama_new_context_with_model: n_batch = 2048
llama_new_context_with_model: n_ubatch = 512
llama_new_context_with_model: freq_base = 1000000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init: CUDA0 KV buffer size = 5248.00 MiB
llama_kv_cache_init: CUDA1 KV buffer size = 4992.00 MiB
llama_new_context_with_model: KV self size = 10240.00 MiB, K (f16): 5120.00 MiB, V (f16): 5120.00 MiB
llama_new_context_with_model: CUDA_Host output buffer size = 0.58 MiB
llama_new_context_with_model: pipeline parallelism enabled (n_copies=4)
llama_new_context_with_model: CUDA0 compute buffer size = 672.01 MiB
llama_new_context_with_model: CUDA1 compute buffer size = 672.02 MiB
llama_new_context_with_model: CUDA_Host compute buffer size = 48.02 MiB
llama_new_context_with_model: graph nodes = 2806
llama_new_context_with_model: graph splits = 3
system_info: n_threads = 40 / 80 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |
sampling:
repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
top_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 0.800
mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
sampling order:
CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temperature
generate: n_ctx = 4096, n_batch = 2048, n_predict = -1, n_keep = 177
Analyze and list the global performance advantages of the U.S. Navy's AN/SPY-7 series of radars, focusing on their unique technical characteristics and performance indicators, such as detection range, accuracy, and anti-jamming capability, and compare them with existing similar radar systems from other countries to highlight the advancements of AN/SPY-7.
Please provide a detailed description, highlighting the advantages of the AN/SPY-7 series of radars in terms of detection range, accuracy and anti-jamming capability, and comparing it with similar radar systems in other countries.
Your answer should include the unique technical characteristics and performance indicators of the AN/SPY-7 series radar, as well as advantages over other radar systems. Please focus on describing the global performance benefits of AN/SPY-7 series radars and provide specific and detailed information. [end of text]
llama_print_timings: load time = 17973.16 ms
llama_print_timings: sample time = 0.09 ms / 1 runs ( 0.09 ms per token, 11627.91 tokens per second)
llama_print_timings: prompt eval time = 487.10 ms / 177 tokens ( 2.75 ms per token, 363.38 tokens per second)
llama_print_timings: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
llama_print_timings: total time = 493.02 ms / 178 tokens
Log end
and detailed information. [end of text]
You want text generation to continue? From readme:
... generated text may be shorter if an End-of-Sequence (EOS) token is encountered. ... In non-interactive mode, the program will end. ... If you want the model to keep going without ever producing End-of-Sequence on its own, try --ignore-eos
.
This only achieves prompt word completion, however the Prompt does not have any Response and keeps outputting spaces.
This issue was closed because it has been inactive for 14 days since being marked as stale.
When I use llamacpp to reason about my Smuag-34B's model, there is no output when the input prompt has 150tokens, but the output is normal when scaled down to about 100.