Closed yitongshang2021 closed 1 month ago
Hello @yitongshang2021,
Thanks for using TabLLM! I only tested the environment on Linux, but I am pretty sure it should also run on Windows. It could maybe be helpful if you specify your error in a bit more detail. Does the entire program run without errors and only the mentioned files are not generated?
Another option might be to use a Colab notebook or other online instance that runs Linux.
Hope that helps!
Stefan
Hello @stefanhgm, Thanks for your message. In the tfew environment, due to the deepspeed==0.5.10 hard to install in the windows, i use the ==0.3.16, and installed successfully.
However, when I run the following code, the error is get, can you help me?
(tfew) C:\Users\78166\t-few>set HF_HOME=%USERPROFILE%.cache\huggingface
(tfew) C:\Users\78166\t-few>set CUDA_VISIBLE_DEVICES=0
(tfew) C:\Users\78166\t-few>python -m src.pl_train -c t03b.json+rte.json -k save_model=False exp_name=first_exp
Start experiment first_exp
{
"exp_dir": "exp_out\first_exp",
"exp_name": "first_exp",
"allow_skip_exp": true,
"seed": 42,
"model": "EncDec",
"max_seq_len": 256,
"origin_model": "bigscience/T0_3B",
"load_weight": "",
"dataset": "rte",
"few_shot": true,
"num_shot": 32,
"few_shot_random_seed": 100,
"train_template_idx": -1,
"eval_template_idx": -1,
"batch_size": 8,
"eval_batch_size": 16,
"num_workers": 8,
"change_hswag_templates": false,
"raft_cross_validation": true,
"raft_validation_start": 0,
"raft_labels_in_input_string": "comma",
"cleaned_answer_choices_b77": false,
"compute_precision": "bf16",
"compute_strategy": "none",
"num_steps": 300,
"eval_epoch_interval": 10000,
"eval_before_training": true,
"save_model": false,
"save_step_interval": 20000,
"mc_loss": 1,
"unlikely_loss": 1,
"length_norm": 1,
"grad_accum_factor": 1,
"split_option_at_inference": false,
"optimizer": "adafactor",
"lr": 0.0003,
"trainable_param_names": ".",
"scheduler": "linear_decay_with_warmup",
"warmup_ratio": 0.06,
"weight_decay": 0,
"scale_parameter": true,
"grad_clip_norm": 1,
"model_modifier": "",
"prompt_tuning_num_prefix_emb": 100,
"prompt_tuning_encoder": true,
"prompt_tuning_decoder": true,
"lora_rank": 4,
"lora_scaling_rank": 0,
"lora_init_scale": 0.01,
"lora_modules": "none",
"lora_layers": "none",
"bitfit_modules": ".",
"bitfitlayers": "q|k|v|o|wi[01]|w_o",
"adapter_type": "normal",
"adapter_non_linearity": "relu",
"adapter_reduction_factor": 4,
"normal_adapter_residual": true,
"lowrank_adapter_w_init": "glorot-uniform",
"lowrank_adapter_rank": 1,
"compacter_hypercomplex_division": 8,
"compacter_learn_phm": true,
"compacter_hypercomplex_nonlinearity": "glorot-uniform",
"compacter_shared_phm_rule": false,
"compacter_factorized_phm": false,
"compacter_shared_W_phm": false,
"compacter_factorized_phm_rule": false,
"compacter_phm_c_init": "normal",
"compacter_phm_rank": 1,
"compacter_phm_init_range": 0.01,
"compacter_kronecker_prod": false,
"compacter_add_compacter_in_self_attention": false,
"compacter_add_compacter_in_cross_attention": false,
"intrinsic_projection": "fastfood",
"intrinsic_said": true,
"intrinsic_dim": 2000,
"intrinsic_device": "cpu",
"fishmask_mode": null,
"fishmask_path": null,
"fishmask_keep_ratio": 0.05,
"prefix_tuning_num_input_tokens": 10,
"prefix_tuning_num_target_tokens": 10,
"prefix_tuning_init_path": null,
"prefix_tuning_init_text": null,
"prefix_tuning_parameterization": "mlp-512",
"train_pred_file": "exp_out\first_exp\train_pred.txt",
"dev_pred_file": "exp_out\first_exp\dev_pred.txt",
"dev_score_file": "exp_out\first_exp\dev_scores.json",
"test_pred_file": "exp_out\first_exp\test_pred.txt",
"test_score_file": "exp_out\first_exp\test_scores.json",
"finish_flag_file": "exp_out\first_exp\exp_completed.txt"
}
Mark experiment first_exp as claimed
Traceback (most recent call last):
File "C:\Users\78166\anaconda3\envs\tfew\lib\runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "C:\Users\78166\anaconda3\envs\tfew\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\78166\t-few\src\pl_train.py", line 86, in
Hi @yitongshang2021,
this looks as if the program already fails while downloading the huggingface model (see line tokenizer_config = get_tokenizer_config(pretrained_model_name_or_path, **kwargs)
).
Unfortunately, this could be due to a variety of reasons. You could try running a minimal huggingface examples first or searching for solutions for the error ValueError: check_hostname requires server_hostname
on Google. Maybe this helps: https://stackoverflow.com/questions/67297278/valueerror-check-hostname-requires-server-hostname ?
Hope that helps!
Hi @stefanhgm! Thanks again for your message. With the help of your information, I have sloved the above bug. However, a new bug has been appeared, as following, I would really appreciate it if you could give me some insight.
(tfew) C:\Users\78166\t-few-master>python -m src.pl_train -c t03b.json+rte.json -k save_model=False exp_name=first_exp
Start experiment first_exp
{
"exp_dir": "exp_out\first_exp",
"exp_name": "first_exp",
"allow_skip_exp": true,
"seed": 42,
"model": "EncDec",
"max_seq_len": 256,
"origin_model": "bigscience/T0_3B",
"load_weight": "",
"dataset": "rte",
"few_shot": true,
"num_shot": 32,
"few_shot_random_seed": 100,
"train_template_idx": -1,
"eval_template_idx": -1,
"batch_size": 8,
"eval_batch_size": 16,
"num_workers": 8,
"change_hswag_templates": false,
"raft_cross_validation": true,
"raft_validation_start": 0,
"raft_labels_in_input_string": "comma",
"cleaned_answer_choices_b77": false,
"compute_precision": "bf16",
"compute_strategy": "none",
"num_steps": 300,
"eval_epoch_interval": 10000,
"eval_before_training": true,
"save_model": false,
"save_step_interval": 20000,
"mc_loss": 1,
"unlikely_loss": 1,
"length_norm": 1,
"grad_accum_factor": 1,
"split_option_at_inference": false,
"optimizer": "adafactor",
"lr": 0.0003,
"trainable_param_names": ".",
"scheduler": "linear_decay_with_warmup",
"warmup_ratio": 0.06,
"weight_decay": 0,
"scale_parameter": true,
"grad_clip_norm": 1,
"model_modifier": "",
"prompt_tuning_num_prefix_emb": 100,
"prompt_tuning_encoder": true,
"prompt_tuning_decoder": true,
"lora_rank": 4,
"lora_scaling_rank": 0,
"lora_init_scale": 0.01,
"lora_modules": "none",
"lora_layers": "none",
"bitfit_modules": ".",
"bitfitlayers": "q|k|v|o|wi[01]|w_o",
"adapter_type": "normal",
"adapter_non_linearity": "relu",
"adapter_reduction_factor": 4,
"normal_adapter_residual": true,
"lowrank_adapter_w_init": "glorot-uniform",
"lowrank_adapter_rank": 1,
"compacter_hypercomplex_division": 8,
"compacter_learn_phm": true,
"compacter_hypercomplex_nonlinearity": "glorot-uniform",
"compacter_shared_phm_rule": false,
"compacter_factorized_phm": false,
"compacter_shared_W_phm": false,
"compacter_factorized_phm_rule": false,
"compacter_phm_c_init": "normal",
"compacter_phm_rank": 1,
"compacter_phm_init_range": 0.01,
"compacter_kronecker_prod": false,
"compacter_add_compacter_in_self_attention": false,
"compacter_add_compacter_in_cross_attention": false,
"intrinsic_projection": "fastfood",
"intrinsic_said": true,
"intrinsic_dim": 2000,
"intrinsic_device": "cpu",
"fishmask_mode": null,
"fishmask_path": null,
"fishmask_keep_ratio": 0.05,
"prefix_tuning_num_input_tokens": 10,
"prefix_tuning_num_target_tokens": 10,
"prefix_tuning_init_path": null,
"prefix_tuning_init_text": null,
"prefix_tuning_parameterization": "mlp-512",
"train_pred_file": "exp_out\first_exp\train_pred.txt",
"dev_pred_file": "exp_out\first_exp\dev_pred.txt",
"dev_score_file": "exp_out\first_exp\dev_scores.json",
"test_pred_file": "exp_out\first_exp\test_pred.txt",
"test_score_file": "exp_out\first_exp\test_scores.json",
"finish_flag_file": "exp_out\first_exp\exp_completed.txt"
}
Mark experiment first_exp as claimed
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
Using bfloat16 Automatic Mixed Precision (AMP)
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
Reusing dataset super_glue (C:\Users\78166.cache\huggingface\super_glue\rte\1.0.2\d040c658e2ddef6934fdd97deb45c777b6ff50c524781ea434e7219b56a428a7)
Train size 32
Eval size 277
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
C:\Users\78166\anaconda3\envs\tfew\lib\site-packages\pytorch_lightning\utilities\model_summary.py:127: RuntimeWarning: overflow encountered in long_scalars
return sum(np.prod(p.shape) if not _is_lazy_weight_tensor(p) else 0 for p in self._module.parameters())
Traceback (most recent call last):
File "C:\Users\78166\anaconda3\envs\tfew\lib\runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "C:\Users\78166\anaconda3\envs\tfew\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\78166\t-few-master\src\pl_train.py", line 89, in
Hi @stefanhgm, I have run the demo on the linux. It's a very helpful code! According to my experience, it is difficult to run successfully on Windows, especially the compatibility issues of some libraries. Thank you for your help.
Dear author, I can't get dev_scores.json and other files when I run tfew code. Can TabLLM only run on Linux? Thanks!