race score using run_race.py with albert base v2 is so much higher than score in table(66.8)

[seongwook-ham]

thanks for your race training evalution script update. when i tested race script score is so much higher than score in table(66.8) i got 70.49 best score eval_accuracy = 0.70493144 eval_loss = 2.0332112 global_step = 12000 loss = 2.160366 best = 0.7028852105140686 i add two line in race_utils.py to read separated race files cur_path_list = tf.gfile.Glob(cur_dir+"/*.txt") for cur_path in cur_path_list: with tf.gfile.Open(cur_path) as f: following is my hyper parameter due to albert paper, path information is omitted because it has personal informations --do_lower_case=True --max_seq_length=512 --max_qa_length=128 --do_train=True --do_eval=True --do_predict=False --train_batch_size=32 --predict_batch_size=8 --learning_rate=2e-5 --train_step=12000 --warmup_step=1000 --save_checkpoints_steps=1000 --iterations_per_loop=1000 --use_tpu=True in config file i use ALBERT DR=0 (in paper it is 0.1). could it affect result? Classifier DR remain 0.1(same as paper) did i do something different from yours or script has some incomplete part? with few more trial race result is between 70.0-71.0

[MaybeLL]

hello,i'm trying to run run_race.py ,but it failed. here is the error info: TypeError: sequence item 373: expected str instance, bytes found

actually I haven't changed the run_race.py. I just simply clone the repo to my machine. so i guess that i may have mistake with the parameter. here is my parameter:

--albert_config_file=/home/dy/Project/ALBERT/albert_base/albert_config.json --output_dir=/home/dy/Project/ALBERT/output_dir --data_dir=/home/dy/Project/ALBERT --init_checkpoint=/home/dy/Project/ALBERT/albert_base/model.ckpt-best --train_file=/home/dy/Project/ALBERT/train_file/train --eval_file=/home/dy/Project/ALBERT/eval_file/eval --spm_model_file=/home/dy/Project/ALBERT/albert_base/30k-clean.model --max_seq_length=512 --max_qa_length=128 --do_train --do_eval --train_batch_size=8 --eval_batch_size=8 --learning_rate=1e-5 --train_step=12000 --warmup_step=1000 --save_checkpoints_steps=100

[seongwook-ham]

@MaybeLL race folder should be in /home/dy/Project/ALBERT. is it right? and your race file should be merged or you should add following lines because basic race data is composed of many txt files cur_path_list = tf.gfile.Glob(cur_dir+"/*.txt") for cur_path in cur_path_list: with tf.gfile.Open(cur_path) as f: before "with" there should be tab but i don't know how to insert tab in github code

[MaybeLL]

@MaybeLL race folder should be in /home/dy/Project/ALBERT. is it right? and your race file should be merged or you should add following lines because basic race data is composed of many txt files cur_path_list = tf.gfile.Glob(cur_dir+"/*.txt") for cur_path in cur_path_list: with tf.gfile.Open(cur_path) as f: before "with" there should be tab but i don't know how to insert tab in github code

thank for you reply. i have try to follow your advise and it still occured the same error: TypeError: sequence item 373: expected str instance, bytes found But the good news is i may find the key of the problem:

--train_file=/home/dy/Project/ALBERT/train_file/train

as i simpy copy the official code without see what is train_file, and i don't really have a train file. so i want know what is the train file ? how can i get it ?how to used it ?

[0x0539]

@MaybeLL what Python version are you using?

[theword]

Using Python 3, no drop and 2e-5 learning rate. I was able to replicate the 70% for ALBERT base on RACE. Previously using 1e-5 learning rate for RACE returned 68.04%.

Results:

eval_accuracy = 0.7034991 eval_loss = 2.0648174 global_step = 12000 loss = 2.1602752 best = 0.7055453062057495

[light111lll]

@seongwook-ham hi,i only run race.m ,as you way.But the result is much lower than yours. Can you help me see why here is my parameter: python run_race.py --albert_config_file=prev_trained_model/base/config.json --output_dir=output_dir --data_dir=./data/data --train_file=train_file/train --eval_file=eval_file/eval --spm_model_file=prev_trained_model/base/30k-clean.model --max_seq_length=512 --max_qa_length=128 --do_train --do_eval --train_batch_size=2 --eval_batch_size=2 --learning_rate=1e-5 --train_step=12000 --warmup_step=1000 --save_checkpoints_steps=100 and here is my results: I0404 04:19:14.645840 140554931050240 error_handling.py:101] evaluation_loop marked as finished INFO:tensorflow: Eval results I0404 04:19:14.646119 140554931050240 run_race.py:382] Eval results INFO:tensorflow: eval_accuracy = 0.29735377 I0404 04:19:14.646275 140554931050240 run_race.py:384] eval_accuracy = 0.29735377 INFO:tensorflow: eval_loss = 1.3716685 I0404 04:19:14.646464 140554931050240 run_race.py:384] eval_loss = 1.3716685 INFO:tensorflow: global_step = 12000 I0404 04:19:14.646620 140554931050240 run_race.py:384] global_step = 12000 INFO:tensorflow: loss = 1.3716685 I0404 04:19:14.646771 140554931050240 run_race.py:384] loss = 1.3716685

[seongwook-ham]

@light111lll train batch size is not proper i think

[light111lll]

@seongwook-ham thanks but increase batchsize as 8 the bestresults only 0.34 . I think it must be something what have i done wrong

[vqiangv]

thanks for your race training evalution script update. when i tested race script score is so much higher than score in table(66.8) i got 70.49 best score eval_accuracy = 0.70493144 eval_loss = 2.0332112 global_step = 12000 loss = 2.160366 best = 0.7028852105140686 i add two line in race_utils.py to read separated race files cur_path_list = tf.gfile.Glob(cur_dir+"/*.txt") for cur_path in cur_path_list: with tf.gfile.Open(cur_path) as f: following is my hyper parameter due to albert paper, path information is omitted because it has personal informations --do_lower_case=True --max_seq_length=512 --max_qa_length=128 --do_train=True --do_eval=True --do_predict=False --train_batch_size=32 --predict_batch_size=8 --learning_rate=2e-5 --train_step=12000 --warmup_step=1000 --save_checkpoints_steps=1000 --iterations_per_loop=1000 --use_tpu=True in config file i use ALBERT DR=0 (in paper it is 0.1). could it affect result? Classifier DR remain 0.1(same as paper) did i do something different from yours or script has some incomplete part? with few more trial race result is between 70.0-71.0

hi，Have you solved the problem? My result is also very high(70.8). In some papers, I see that their results are similar to the data in the table. I don't know what's wrong with me.

here is my parameter:

--do_lower_case --do_train --do_eval --do_test --overwrite_output --eval_all_checkpoints --task_name race --per_gpu_eval_batch_size 8 --logging_steps 1 --max_seq_length 512 --model_type albert --model_name_or_path albert-base-v2 --data_dir RACE --learning_rate 1e-5 --num_train_epochs 2 --output_dir albert_base_race --per_gpu_train_batch_size 8 --gradient_accumulation_steps 1 --warmup_steps 1000 --save_steps 3000

[vqiangv]

I set epoch to 1, The result is: 0.6892987434130523