search

THUDM / ChatGLM2-6B

ChatGLM2-6B: An Open Bilingual Chat LLM | 开源双语对话语言模型
Other
15.73k stars 1.85k forks source link

[BUG/Help] <title> #464

Open shnyyds opened 1 year ago

shnyyds commented 1 year ago

Is there an existing issue for this?

Current Behavior

微调报错 ValueError: None is not in list $ sh train.sh 08/16/2023 16:29:49 - WARNING - main - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False 08/16/2023 16:29:49 - INFO - main - Training/evaluation parameters Seq2SeqTrainingArguments( _n_gpu=1, adafactor=False, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, auto_find_batch_size=False, bf16=False, bf16_full_eval=False, data_seed=None, dataloader_drop_last=False, dataloader_num_workers=0, dataloader_pin_memory=True, ddp_backend=None, ddp_bucket_cap_mb=None, ddp_find_unused_parameters=None, ddp_timeout=1800, debug=[], deepspeed=None, disable_tqdm=False, do_eval=False, do_predict=False, do_train=True, eval_accumulation_steps=None, eval_delay=0, eval_steps=None, evaluation_strategy=no, fp16=False, fp16_backend=auto, fp16_full_eval=False, fp16_opt_level=O1, fsdp=[], fsdp_config={'fsdp_min_num_params': 0, 'xla': False, 'xla_fsdp_grad_ckpt': False}, fsdp_min_num_params=0, fsdp_transformer_layer_cls_to_wrap=None, full_determinism=False, generation_config=None, generation_max_length=None, generation_num_beams=None, gradient_accumulation_steps=16, gradient_checkpointing=False, greater_is_better=None, group_by_length=False, half_precision_backend=auto, hub_model_id=None, hub_private_repo=False, hub_strategy=every_save, hub_token=, ignore_data_skip=False, include_inputs_for_metrics=False, jit_mode_eval=False, label_names=None, label_smoothing_factor=0.0, learning_rate=0.02, length_column_name=length, load_best_model_at_end=False, local_rank=0, log_level=passive, log_level_replica=warning, log_on_each_node=True, logging_dir=output/adgen-chatglm2-6b-pt-128-2e-2\runs\Aug16_16-29-49_DESKTOP-JQ9P2MC, logging_first_step=False, logging_nan_inf_filter=True, logging_steps=10, logging_strategy=steps, lr_scheduler_type=linear, max_grad_norm=1.0, max_steps=3000, metric_for_best_model=None, mp_parameters=, no_cuda=False, num_train_epochs=3.0, optim=adamw_hf, optim_args=None, output_dir=output/adgen-chatglm2-6b-pt-128-2e-2, overwrite_output_dir=True, past_index=-1, per_device_eval_batch_size=1, per_device_train_batch_size=1, predict_with_generate=True, prediction_loss_only=False, push_to_hub=False, push_to_hub_model_id=None, push_to_hub_organization=None, push_to_hub_token=, ray_scope=last, remove_unused_columns=True, report_to=[], resume_from_checkpoint=None, run_name=output/adgen-chatglm2-6b-pt-128-2e-2, save_on_each_node=False, save_safetensors=False, save_steps=1000, save_strategy=steps, save_total_limit=None, seed=42, sharded_ddp=[], skip_memory_metrics=True, sortish_sampler=False, tf32=None, torch_compile=False, torch_compile_backend=None, torch_compile_mode=None, torchdynamo=None, tpu_metrics_debug=False, tpu_num_cores=None, use_ipex=False, use_legacy_prediction_loop=False, use_mps_device=False, warmup_ratio=0.0, warmup_steps=0, weight_decay=0.0, xpu_backend=None, ) C:\ProgramData\anaconda3\envs\ChatGLM\lib\site-packages\datasets\load.py:2072: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( [INFO|configuration_utils.py:667] 2023-08-16 16:29:50,852 >> loading configuration file chatglm2-6b\config.json [INFO|configuration_utils.py:667] 2023-08-16 16:29:50,868 >> loading configuration file chatglm2-6b\config.json [INFO|configuration_utils.py:725] 2023-08-16 16:29:50,871 >> Model config ChatGLMConfig { "_name_or_path": "chatglm2-6b", "add_bias_linear": false, "add_qkv_bias": true, "apply_query_key_layer_scaling": true, "apply_residual_connection_post_layernorm": false, "architectures": [ "ChatGLMModel" ], "attention_dropout": 0.0, "attention_softmax_in_fp32": true, "auto_map": { "AutoConfig": "configuration_chatglm.ChatGLMConfig", "AutoModel": "modeling_chatglm.ChatGLMForConditionalGeneration", "AutoModelForCausalLM": "modeling_chatglm.ChatGLMForConditionalGeneration", "AutoModelForSeq2SeqLM": "modeling_chatglm.ChatGLMForConditionalGeneration" }, "bias_dropout_fusion": true, "eos_token_id": 2, "ffn_hidden_size": 13696, "fp32_residual_connection": false, "hidden_dropout": 0.0, "hidden_size": 4096, "kv_channels": 128, "layernorm_epsilon": 1e-05, "model_type": "chatglm", "multi_query_attention": true, "multi_query_group_num": 2, "num_attention_heads": 32, "num_layers": 28, "original_rope": true, "pad_token_id": 0, "padded_vocab_size": 65024, "post_layer_norm": true, "pre_seq_len": null, "prefix_projection": false, "quantization_bit": 0, "rmsnorm": true, "seq_length": 32768, "tie_word_embeddings": false, "torch_dtype": "float16", "transformers_version": "4.29.1", "use_cache": true, "vocab_size": 65024 }

[INFO|tokenization_utils_base.py:1808] 2023-08-16 16:29:50,881 >> loading file tokenizer.model [INFO|tokenization_utils_base.py:1808] 2023-08-16 16:29:50,882 >> loading file added_tokens.json [INFO|tokenization_utils_base.py:1808] 2023-08-16 16:29:50,882 >> loading file special_tokens_map.json [INFO|tokenization_utils_base.py:1808] 2023-08-16 16:29:50,882 >> loading file tokenizer_config.json [INFO|modeling_utils.py:2513] 2023-08-16 16:29:51,038 >> loading weights file chatglm2-6b\pytorch_model.bin.index.json [INFO|configuration_utils.py:577] 2023-08-16 16:29:51,041 >> Generate config GenerationConfig { "_from_model_config": true, "eos_token_id": 2, "pad_token_id": 0, "transformers_version": "4.29.1" }

Loading checkpoint shards: 100%|██████████| 7/7 [00:16<00:00, 2.42s/it] [INFO|modeling_utils.py:3185] 2023-08-16 16:30:08,574 >> All model checkpoint weights were used when initializing ChatGLMForConditionalGeneration.

[WARNING modeling_utils.py:3187] 2023-08-16 16:30:08,574 >> Some weights of ChatGLMForConditionalGeneration were not initialized from the model checkpoint at chatglm2-6b and are newly initialized: ['transformer.prefix_encoder.embedding.weight'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. [INFO modeling_utils.py:2821] 2023-08-16 16:30:08,579 >> Generation config file not found, using a generation config created from the model config. Running tokenizer on train dataset: 0% 0/114599 [00:00<?, ? examples/s] +--------------------- Traceback (most recent call last) ---------------------+ C:\Users\1\IdeaProjects\sgdd-python\chatglm\sgdd-chat-glm-6b\ptuning\main.p y:430 in
427
428
429 if name == "main":
> 430 main()
431
C:\Users\1\IdeaProjects\sgdd-python\chatglm\sgdd-chat-glm-6b\ptuning\main.p
y:248 in main
245 max_train_samples = min(len(train_dataset), data_args.max
246 train_dataset = train_dataset.select(range(max_train_samp
247 with training_args.main_process_first(desc="train dataset map
> 248 train_dataset = train_dataset.map(
249 preprocess_function_train,
250 batched=True,
251 num_proc=data_args.preprocessing_num_workers,
C:\ProgramData\anaconda3\envs\ChatGLM\lib\site-packages\datasets\arrow_data
set.py:592 in wrapper
589 else:
590 self: "Dataset" = kwargs.pop("self")
591 # apply actual function
> 592 out: Union["Dataset", "DatasetDict"] = func(self, *args, **k
593 datasets: List["Dataset"] = list(out.values()) if isinstance
594 for dataset in datasets:
595 # Remove task templates if a column mapping of the templ
C:\ProgramData\anaconda3\envs\ChatGLM\lib\site-packages\datasets\arrow_data
set.py:557 in wrapper
554 "output_all_columns": self._output_all_columns,
555 }
556 # apply actual function
> 557 out: Union["Dataset", "DatasetDict"] = func(self, *args, **k
558 datasets: List["Dataset"] = list(out.values()) if isinstance
559 # re-apply format to the output
560 for dataset in datasets:
C:\ProgramData\anaconda3\envs\ChatGLM\lib\site-packages\datasets\arrow_data
set.py:3097 in map
3094 total=pbar_total,
3095 desc=desc or "Map",
3096 ) as pbar:
> 3097 for rank, done, content in Dataset._map_single(*
3098 if done:
3099 shards_done += 1
3100 logger.debug(f"Finished processing shard
C:\ProgramData\anaconda3\envs\ChatGLM\lib\site-packages\datasets\arrow_data
set.py:3474 in _map_single
3471 range(*(slice(i, i + batch_size).indices
3472 ) # Something simpler?
3473 try:
> 3474 batch = apply_function_on_filtered_input
3475 batch,
3476 indices,
3477 check_same_num_examples=len(shard.li
C:\ProgramData\anaconda3\envs\ChatGLM\lib\site-packages\datasets\arrow_data
set.py:3353 in apply_function_on_filtered_inputs
3350 additional_args += (effective_indices,)
3351 if with_rank:
3352 additional_args += (rank,)
> 3353 processed_inputs = function(fn_args, additional_args,
3354 if isinstance(processed_inputs, LazyDict):
3355 processed_inputs = {
3356 k: v for k, v in processed_inputs.data.items() i
C:\Users\1\IdeaProjects\sgdd-python\chatglm\sgdd-chat-glm-6b\ptuning\main.p
y:219 in preprocess_function_train
216
217 input_ids = tokenizer.build_inputs_with_special_token
218
> 219 context_length = input_ids.index(tokenizer.bostoken
220 mask_position = context_length - 1
221 labels = [-100] * context_length + input_ids[mask_pos
222

+-----------------------------------------------------------------------------+ ValueError: None is not in list

Expected Behavior

无

Steps To Reproduce

$ sh train.sh 08/16/2023 16:29:49 - WARNING - main - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False 08/16/2023 16:29:49 - INFO - main - Training/evaluation parameters Seq2SeqTrainingArguments( _n_gpu=1, adafactor=False, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, auto_find_batch_size=False, bf16=False, bf16_full_eval=False, data_seed=None, dataloader_drop_last=False, dataloader_num_workers=0, dataloader_pin_memory=True, ddp_backend=None, ddp_bucket_cap_mb=None, ddp_find_unused_parameters=None, ddp_timeout=1800, debug=[], deepspeed=None, disable_tqdm=False, do_eval=False, do_predict=False, do_train=True, eval_accumulation_steps=None, eval_delay=0, eval_steps=None, evaluation_strategy=no, fp16=False, fp16_backend=auto, fp16_full_eval=False, fp16_opt_level=O1, fsdp=[], fsdp_config={'fsdp_min_num_params': 0, 'xla': False, 'xla_fsdp_grad_ckpt': False}, fsdp_min_num_params=0, fsdp_transformer_layer_cls_to_wrap=None, full_determinism=False, generation_config=None, generation_max_length=None, generation_num_beams=None, gradient_accumulation_steps=16, gradient_checkpointing=False, greater_is_better=None, group_by_length=False, half_precision_backend=auto, hub_model_id=None, hub_private_repo=False, hub_strategy=every_save, hub_token=, ignore_data_skip=False, include_inputs_for_metrics=False, jit_mode_eval=False, label_names=None, label_smoothing_factor=0.0, learning_rate=0.02, length_column_name=length, load_best_model_at_end=False, local_rank=0, log_level=passive, log_level_replica=warning, log_on_each_node=True, logging_dir=output/adgen-chatglm2-6b-pt-128-2e-2\runs\Aug16_16-29-49_DESKTOP-JQ9P2MC, logging_first_step=False, logging_nan_inf_filter=True, logging_steps=10, logging_strategy=steps, lr_scheduler_type=linear, max_grad_norm=1.0, max_steps=3000, metric_for_best_model=None, mp_parameters=, no_cuda=False, num_train_epochs=3.0, optim=adamw_hf, optim_args=None, output_dir=output/adgen-chatglm2-6b-pt-128-2e-2, overwrite_output_dir=True, past_index=-1, per_device_eval_batch_size=1, per_device_train_batch_size=1, predict_with_generate=True, prediction_loss_only=False, push_to_hub=False, push_to_hub_model_id=None, push_to_hub_organization=None, push_to_hub_token=, ray_scope=last, remove_unused_columns=True, report_to=[], resume_from_checkpoint=None, run_name=output/adgen-chatglm2-6b-pt-128-2e-2, save_on_each_node=False, save_safetensors=False, save_steps=1000, save_strategy=steps, save_total_limit=None, seed=42, sharded_ddp=[], skip_memory_metrics=True, sortish_sampler=False, tf32=None, torch_compile=False, torch_compile_backend=None, torch_compile_mode=None, torchdynamo=None, tpu_metrics_debug=False, tpu_num_cores=None, use_ipex=False, use_legacy_prediction_loop=False, use_mps_device=False, warmup_ratio=0.0, warmup_steps=0, weight_decay=0.0, xpu_backend=None, ) C:\ProgramData\anaconda3\envs\ChatGLM\lib\site-packages\datasets\load.py:2072: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( [INFO|configuration_utils.py:667] 2023-08-16 16:29:50,852 >> loading configuration file chatglm2-6b\config.json [INFO|configuration_utils.py:667] 2023-08-16 16:29:50,868 >> loading configuration file chatglm2-6b\config.json [INFO|configuration_utils.py:725] 2023-08-16 16:29:50,871 >> Model config ChatGLMConfig { "_name_or_path": "chatglm2-6b", "add_bias_linear": false, "add_qkv_bias": true, "apply_query_key_layer_scaling": true, "apply_residual_connection_post_layernorm": false, "architectures": [ "ChatGLMModel" ], "attention_dropout": 0.0, "attention_softmax_in_fp32": true, "auto_map": { "AutoConfig": "configuration_chatglm.ChatGLMConfig", "AutoModel": "modeling_chatglm.ChatGLMForConditionalGeneration", "AutoModelForCausalLM": "modeling_chatglm.ChatGLMForConditionalGeneration", "AutoModelForSeq2SeqLM": "modeling_chatglm.ChatGLMForConditionalGeneration" }, "bias_dropout_fusion": true, "eos_token_id": 2, "ffn_hidden_size": 13696, "fp32_residual_connection": false, "hidden_dropout": 0.0, "hidden_size": 4096, "kv_channels": 128, "layernorm_epsilon": 1e-05, "model_type": "chatglm", "multi_query_attention": true, "multi_query_group_num": 2, "num_attention_heads": 32, "num_layers": 28, "original_rope": true, "pad_token_id": 0, "padded_vocab_size": 65024, "post_layer_norm": true, "pre_seq_len": null, "prefix_projection": false, "quantization_bit": 0, "rmsnorm": true, "seq_length": 32768, "tie_word_embeddings": false, "torch_dtype": "float16", "transformers_version": "4.29.1", "use_cache": true, "vocab_size": 65024 }

[INFO|tokenization_utils_base.py:1808] 2023-08-16 16:29:50,881 >> loading file tokenizer.model [INFO|tokenization_utils_base.py:1808] 2023-08-16 16:29:50,882 >> loading file added_tokens.json [INFO|tokenization_utils_base.py:1808] 2023-08-16 16:29:50,882 >> loading file special_tokens_map.json [INFO|tokenization_utils_base.py:1808] 2023-08-16 16:29:50,882 >> loading file tokenizer_config.json [INFO|modeling_utils.py:2513] 2023-08-16 16:29:51,038 >> loading weights file chatglm2-6b\pytorch_model.bin.index.json [INFO|configuration_utils.py:577] 2023-08-16 16:29:51,041 >> Generate config GenerationConfig { "_from_model_config": true, "eos_token_id": 2, "pad_token_id": 0, "transformers_version": "4.29.1" }

Loading checkpoint shards: 100%|██████████| 7/7 [00:16<00:00, 2.42s/it] [INFO|modeling_utils.py:3185] 2023-08-16 16:30:08,574 >> All model checkpoint weights were used when initializing ChatGLMForConditionalGeneration.

[WARNING modeling_utils.py:3187] 2023-08-16 16:30:08,574 >> Some weights of ChatGLMForConditionalGeneration were not initialized from the model checkpoint at chatglm2-6b and are newly initialized: ['transformer.prefix_encoder.embedding.weight'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. [INFO modeling_utils.py:2821] 2023-08-16 16:30:08,579 >> Generation config file not found, using a generation config created from the model config. Running tokenizer on train dataset: 0% 0/114599 [00:00<?, ? examples/s] +--------------------- Traceback (most recent call last) ---------------------+ C:\Users\1\IdeaProjects\sgdd-python\chatglm\sgdd-chat-glm-6b\ptuning\main.p y:430 in
427
428
429 if name == "main":
> 430 main()
431
C:\Users\1\IdeaProjects\sgdd-python\chatglm\sgdd-chat-glm-6b\ptuning\main.p
y:248 in main
245 max_train_samples = min(len(train_dataset), data_args.max
246 train_dataset = train_dataset.select(range(max_train_samp
247 with training_args.main_process_first(desc="train dataset map
> 248 train_dataset = train_dataset.map(
249 preprocess_function_train,
250 batched=True,
251 num_proc=data_args.preprocessing_num_workers,
C:\ProgramData\anaconda3\envs\ChatGLM\lib\site-packages\datasets\arrow_data
set.py:592 in wrapper
589 else:
590 self: "Dataset" = kwargs.pop("self")
591 # apply actual function
> 592 out: Union["Dataset", "DatasetDict"] = func(self, *args, **k
593 datasets: List["Dataset"] = list(out.values()) if isinstance
594 for dataset in datasets:
595 # Remove task templates if a column mapping of the templ
C:\ProgramData\anaconda3\envs\ChatGLM\lib\site-packages\datasets\arrow_data
set.py:557 in wrapper
554 "output_all_columns": self._output_all_columns,
555 }
556 # apply actual function
> 557 out: Union["Dataset", "DatasetDict"] = func(self, *args, **k
558 datasets: List["Dataset"] = list(out.values()) if isinstance
559 # re-apply format to the output
560 for dataset in datasets:
C:\ProgramData\anaconda3\envs\ChatGLM\lib\site-packages\datasets\arrow_data
set.py:3097 in map
3094 total=pbar_total,
3095 desc=desc or "Map",
3096 ) as pbar:
> 3097 for rank, done, content in Dataset._map_single(*
3098 if done:
3099 shards_done += 1
3100 logger.debug(f"Finished processing shard
C:\ProgramData\anaconda3\envs\ChatGLM\lib\site-packages\datasets\arrow_data
set.py:3474 in _map_single
3471 range(*(slice(i, i + batch_size).indices
3472 ) # Something simpler?
3473 try:
> 3474 batch = apply_function_on_filtered_input
3475 batch,
3476 indices,
3477 check_same_num_examples=len(shard.li
C:\ProgramData\anaconda3\envs\ChatGLM\lib\site-packages\datasets\arrow_data
set.py:3353 in apply_function_on_filtered_inputs
3350 additional_args += (effective_indices,)
3351 if with_rank:
3352 additional_args += (rank,)
> 3353 processed_inputs = function(fn_args, additional_args,
3354 if isinstance(processed_inputs, LazyDict):
3355 processed_inputs = {
3356 k: v for k, v in processed_inputs.data.items() i
C:\Users\1\IdeaProjects\sgdd-python\chatglm\sgdd-chat-glm-6b\ptuning\main.p
y:219 in preprocess_function_train
216
217 input_ids = tokenizer.build_inputs_with_special_token
218
> 219 context_length = input_ids.index(tokenizer.bostoken
220 mask_position = context_length - 1
221 labels = [-100] * context_length + input_ids[mask_pos
222

+-----------------------------------------------------------------------------+ ValueError: None is not in list

Environment

- OS:
- Python:
- Transformers:
- PyTorch:
- CUDA Support (`python -c "import torch; print(torch.cuda.is_available())"`) :

Anything else?

。

waterstars19 commented 1 year ago

same here

CreamyLong commented 1 year ago

https://github.com/THUDM/ChatGLM2-6B/issues/54 This may help you

Siqi-c commented 10 months ago

您好，请问您的问题解决了吗？我也遇到了这个问题。