[Problem/ Squad V2] the result is too low compare with the F1 score in paper, Is something wrong?

[Gs-Zhang]

I0922 11:46:38.663871 140308634334976 run_squad_v2.py:505] Final Eval results INFO:tensorflow: exact = 50.09685841825992 I0922 11:46:38.663987 140308634334976 run_squad_v2.py:507] exact = 50.09685841825992 INFO:tensorflow: f1 = 50.11359538016659 I0922 11:46:38.664040 140308634334976 run_squad_v2.py:507] f1 = 50.11359538016659 INFO:tensorflow: null_score_diff_threshold = -1.230899453163147 I0922 11:46:38.664077 140308634334976 run_squad_v2.py:507] null_score_diff_threshold = -1.230899453163147 INFO:tensorflow: total = 11873 I0922 11:46:38.664113 140308634334976 run_squad_v2.py:507] total = 11873

[Gs-Zhang]

flags.DEFINE_string( "albert_config_file", 'albert_base/albert_config.json', "The config json file corresponding to the pre-trained ALBERT model. " "This specifies the model architecture.")

flags.DEFINE_string("vocab_file", 'albert_xlarge/30k-clean.vocab', "The vocabulary file that the ALBERT model was trained on.")

flags.DEFINE_string("spm_model_file", 'albert_xlarge/30k-clean.model', "The model file for sentence piece tokenization.")

flags.DEFINE_string( "output_dir", 'result', "The output directory where the model checkpoints will be written.")

Other parameters

flags.DEFINE_string("train_file", '/home/gszhang/code/NLP/albert/train-v2.0.json', "SQuAD json for training. E.g., train-v1.1.json")

flags.DEFINE_string( "predict_file", 'dev-v2.0.json', "SQuAD json for predictions. E.g., dev-v1.1.json or test-v1.1.json")

flags.DEFINE_string("train_feature_file", '/home/gszhang/code/NLP/albert/result/train_feature', "training feature file.")

flags.DEFINE_string( "predict_feature_file", '/home/gszhang/code/NLP/albert/result/predict_feature', "Location of predict features. If it doesn't exist, it will be written. " "If it does exist, it will be read.")

flags.DEFINE_string( "predict_feature_left_file", '/home/gszhang/code/NLP/albert/result/predict_left_feature', "Location of predict features not passed to TPU. If it doesn't exist, it " "will be written. If it does exist, it will be read.")

flags.DEFINE_string( "init_checkpoint", 'albert_xlarge/model.ckpt-best.index', "Initial checkpoint (usually from a pre-trained BERT model).")

flags.DEFINE_string( "albert_hub_module_handle", None, "If set, the ALBERT hub module to use.")

flags.DEFINE_bool( "do_lower_case", True, "Whether to lower case the input text. Should be True for uncased " "models and False for cased models.")

flags.DEFINE_integer( "max_seq_length", 384, "The maximum total input sequence length after WordPiece tokenization. " "Sequences longer than this will be truncated, and sequences shorter " "than this will be padded.")

flags.DEFINE_integer( "doc_stride", 128, "When splitting up a long document into chunks, how much stride to " "take between chunks.")

flags.DEFINE_integer( "max_query_length", 64, "The maximum number of tokens for the question. Questions longer than " "this will be truncated to this length.")

flags.DEFINE_bool("do_train", False, "Whether to run training.")

flags.DEFINE_bool("do_predict", True, "Whether to run eval on the dev set.")

flags.DEFINE_integer("train_batch_size", 8, "Total batch size for training.")

flags.DEFINE_integer("predict_batch_size", 8, "Total batch size for predictions.")

flags.DEFINE_float("learning_rate", 5e-5, "The initial learning rate for Adam.")

flags.DEFINE_float("num_train_epochs", 3.0, "Total number of training epochs to perform.")

flags.DEFINE_float( "warmup_proportion", 0.1, "Proportion of training to perform linear learning rate warmup for. " "E.g., 0.1 = 10% of training.")

flags.DEFINE_integer("save_checkpoints_steps", 1000, "How often to save the model checkpoint.")

flags.DEFINE_integer("iterations_per_loop", 1000, "How many steps to make in each estimator call.")

flags.DEFINE_integer( "n_best_size", 20, "The total number of n-best predictions to generate in the " "nbest_predictions.json output file.")

flags.DEFINE_integer( "max_answer_length", 30, "The maximum length of an answer that can be generated. This is needed " "because the start and end predictions are not conditioned on one another.")

flags.DEFINE_bool("use_tpu", False, "Whether to use TPU or GPU/CPU.")

tf.flags.DEFINE_string( "tpu_name", None, "The Cloud TPU to use for training. This should be either the name " "used when creating the Cloud TPU, or a grpc://ip.address.of.tpu:8470 " "url.")

tf.flags.DEFINE_string( "tpu_zone", None, "[Optional] GCE zone where the Cloud TPU is located in. If not " "specified, we will attempt to automatically detect the GCE project from " "metadata.")

tf.flags.DEFINE_string( "gcp_project", None, "[Optional] Project name for the Cloud TPU-enabled project. If not " "specified, we will attempt to automatically detect the GCE project from " "metadata.")

tf.flags.DEFINE_string("master", None, "[Optional] TensorFlow master URL.")

flags.DEFINE_integer( "num_tpu_cores", 8, "Only used if use_tpu is True. Total number of TPU cores to use.")

flags.DEFINE_integer("start_n_top", 5, "beam size for the start positions.")

flags.DEFINE_integer("end_n_top", 5, "beam size for the end positions.")

flags.DEFINE_float("dropout_prob", 0.1, "dropout probability.")

this is what I set in run_squad_v2, I can't find the problem, Thanks for your help!

[Gs-Zhang]

And the feature file is not exist, it is generated when I am running the .py.

[Huibin-Ge]

hi, I meet the same problem, since i use the gpu to run the code, and i change TPUEstimator to Estimator and change TPUEstimatorSpec to EstimatorSpec, and the problem was solved, and can get the f1 score as paper

[PremalMatalia]

@Huibin-Ge - Is it possible to provide your notebook file or code which you are using. I am facing issues in running fine-tuning of albert base using SQuAD 2.0 and training doesn't start and stopped abruptly without any error. Must be some parameter is wrong.

[marvel2120]

same problem

[kavin525zhang]

hi, I meet the same problem, the result is too low, can you tell me how to change TPUEstimator to Estimator and change TPUEstimatorSpec to EstimatorSpec?

[huibinGe]

Hi, I public my fixed code in https://github.com/huibinGe/albert_gpu_squad. TPUEstimator to Estimator mainly in run_squad_v2.py and TPUEstimatorSpec to EstimatorSpec mainly in squad_utils.py