Closed pliablepixels closed 12 months ago
Hi there, while trying to train based on the example, I keep getting GGML_ASSERT: /Users/pp/fiddle/llama.cpp/ggml-alloc.c:116: tensor->data == NULL - any thoughts on how I can triage? The models look ok to me. I tried with falcon/llama
GGML_ASSERT: /Users/pp/fiddle/llama.cpp/ggml-alloc.c:116: tensor->data == NULL
(ml) [pp@pps-2023-MBP:~/fiddle/llama.cpp/build]$ ./bin/train-text-from-scratch --vocab-model ../models/ggml-vocab-falcon.gguf --ctx 64 --embd 256 --head 8 --layer 16 --checkpoint-in chk-shakespeare-256x16-LATEST.gguf --checkpoint-out chk-shakespeare-256x16-ITERATION.gguf --model-out ggml-shakespeare-256x16-f32-ITERATION.gguf --train-data "shakespeare.txt" -t 6 -b 16 --seed 1 --adam-iter 256 --no-checkpointing main: seed: 1 llama_model_loader: loaded meta data with 17 key-value pairs and 0 tensors from ../models/ggml-vocab-falcon.gguf (version GGUF V2 (latest)) llama_model_loader: - kv 0: general.architecture str llama_model_loader: - kv 1: general.name str llama_model_loader: - kv 2: falcon.context_length u32 llama_model_loader: - kv 3: falcon.tensor_data_layout str llama_model_loader: - kv 4: falcon.embedding_length u32 llama_model_loader: - kv 5: falcon.feed_forward_length u32 llama_model_loader: - kv 6: falcon.block_count u32 llama_model_loader: - kv 7: falcon.attention.head_count u32 llama_model_loader: - kv 8: falcon.attention.head_count_kv u32 llama_model_loader: - kv 9: falcon.attention.layer_norm_epsilon f32 llama_model_loader: - kv 10: general.file_type u32 llama_model_loader: - kv 11: tokenizer.ggml.model str llama_model_loader: - kv 12: tokenizer.ggml.tokens arr llama_model_loader: - kv 13: tokenizer.ggml.scores arr llama_model_loader: - kv 14: tokenizer.ggml.token_type arr llama_model_loader: - kv 15: tokenizer.ggml.merges arr llama_model_loader: - kv 16: tokenizer.ggml.eos_token_id u32 llm_load_print_meta: format = GGUF V2 (latest) llm_load_print_meta: arch = falcon llm_load_print_meta: vocab type = BPE llm_load_print_meta: n_vocab = 65024 llm_load_print_meta: n_merges = 64784 llm_load_print_meta: n_ctx_train = 2048 llm_load_print_meta: n_embd = 4544 llm_load_print_meta: n_head = 71 llm_load_print_meta: n_head_kv = 1 llm_load_print_meta: n_layer = 32 llm_load_print_meta: n_rot = 64 llm_load_print_meta: n_gqa = 71 llm_load_print_meta: f_norm_eps = 1.0e-05 llm_load_print_meta: f_norm_rms_eps = 0.0e+00 llm_load_print_meta: n_ff = 18176 llm_load_print_meta: freq_base_train = 10000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: model type = 7B llm_load_print_meta: model ftype = mostly F16 llm_load_print_meta: model params = 0.00 B llm_load_print_meta: model size = 0.00 MiB (nan BPW) llm_load_print_meta: general.name = Falcon llm_load_print_meta: BOS token = 11 '<|endoftext|>' llm_load_print_meta: EOS token = 11 '<|endoftext|>' llm_load_print_meta: LF token = 138 'Ä' llama_model_load: vocab only - skipping tensors llama_new_context_with_model: n_ctx = 512 llama_new_context_with_model: freq_base = 10000.0 llama_new_context_with_model: freq_scale = 1 main: init model GGML_ASSERT: /Users/pp/fiddle/llama.cpp/ggml-alloc.c:116: tensor->data == NULL Abort trap: 6
Models:
(ml) [pp@pps-2023-MBP:~/fiddle/llama.cpp/build]$ ls -l ../models/ total 101338696 -rw-r--r--@ 1 pp staff 23237177504 Oct 12 07:35 codellama-34b-instruct.Q5_K_S.gguf -rw-r--r--@ 1 pp staff 23838797984 Oct 12 07:32 codellama-34b.Q5_K_M.gguf -rw-r--r--@ 1 pp staff 4783256256 Oct 12 06:16 codellama-7b.Q5_K_M.gguf -rw-r--r-- 1 pp staff 4825676 Oct 9 17:28 ggml-vocab-aquila.gguf -rw-r--r-- 1 pp staff 2547782 Oct 9 17:28 ggml-vocab-falcon.gguf -rw-r--r-- 1 pp staff 595423 Oct 9 17:28 ggml-vocab-llama.gguf
Fixed via #3618
ggerganov
commented
12 months ago
Hi there, while trying to train based on the example, I keep getting
GGML_ASSERT: /Users/pp/fiddle/llama.cpp/ggml-alloc.c:116: tensor->data == NULL- any thoughts on how I can triage? The models look ok to me. I tried with falcon/llamaModels: