Could not make predictions for own data

[Arjunsankarlal]

I am trying to make predictions on my own dataset with the dowloaded pretrained model. I have changed the params with the locations of the file in my local respectively. When I try to run run_msmacro.py with train=False and evaluating only with the dev which I have created on my own. In data_dir I have the tfrecord processed files as dataset_dev.tf, query_doc_ids_dev.txt. The params after modifications are,

Required parameters

flags.DEFINE_string( "data_dir", "/Users/arjun/DataSets/MARCO/test", "The input data dir. Should contain the .tfrecord files and the supporting " "query-docids mapping files.")

flags.DEFINE_string( "bert_config_file", "/Users/arjun/DataSets/uncased_L-24_H-1024_A-16/bert_config.json", "The config json file corresponding to the pre-trained BERT model. " "This specifies the model architecture.")

flags.DEFINE_string( "output_dir","/Users/arjun/DataSets/MARCO/BERT_Large_trained_on_MSMARCO", "The output directory where the model checkpoints will be written.")

flags.DEFINE_boolean( "msmarco_output", True, "Whether to write the predictions to a MS-MARCO-formatted file.")

flags.DEFINE_string( "init_checkpoint", "/Users/arjun/DataSets/MARCO/BERT_Large_trained_on_MSMARCO/model.ckpt-100000.index", "Initial checkpoint (usually from a pre-trained BERT model).")

flags.DEFINE_integer( "max_seq_length", 512, "The maximum total input sequence length after WordPiece tokenization. " "Sequences longer than this will be truncated, and sequences shorter " "than this will be padded.")

flags.DEFINE_bool("do_train", False, "Whether to run training.")

flags.DEFINE_bool("do_eval", True, "Whether to run eval on the dev set.")

flags.DEFINE_integer("train_batch_size", 32, "Total batch size for training.")

flags.DEFINE_integer("eval_batch_size", 32, "Total batch size for eval.")

flags.DEFINE_float("learning_rate", 1e-6, "The initial learning rate for Adam.")

flags.DEFINE_integer("num_train_steps", 400000, "Total number of training steps to perform.")

flags.DEFINE_integer("max_eval_examples", None, "Maximum number of examples to be evaluated.")

flags.DEFINE_integer("num_eval_docs", 10, "Number of docs per query in the dev and eval files.")

flags.DEFINE_integer( "num_warmup_steps", 40000, "Number of training steps to perform linear learning rate warmup.")

flags.DEFINE_integer("save_checkpoints_steps", 1000, "How often to save the model checkpoint.")

flags.DEFINE_integer("iterations_per_loop", 1000, "How many steps to make in each estimator call.")

flags.DEFINE_bool("use_tpu", False, "Whether to use TPU or GPU/CPU.")

So now when I run this, it completes without any error. At the end of the logs I see this,

INFO:tensorflow:Running local_init_op. INFO:tensorflow:Done running local_init_op. INFO:tensorflow:prediction_loop marked as finished INFO:tensorflow:prediction_loop marked as finished /Users/arjun/Projects/Github/dl4marco-bert/run_msmarco.py:445: RuntimeWarning: invalid value encountered in true_divide all_metrics /= example_idx INFO:tensorflow:Eval dev: INFO:tensorflow:MAP RPrec MRR NDCG MRR@10 INFO:tensorflow:[nan nan nan nan nan]

Also going through the logs detailed, I found this in it,

INFO:tensorflow: Running evaluation INFO:tensorflow: Batch size = 32 INFO:tensorflow:Computing metrics... INFO:tensorflow:Could not find trained model in model_dir: /Users/arjun/DataSets/MARCO/BERT_Large_trained_on_MSMARCO, running initialization to predict. INFO:tensorflow:Calling model_fn. INFO:tensorflow:Running infer on CPU

I initially had the the output_dir (model_dir assignment done inside the main()) a different one, after looking at this warning I changed the path to the current one. Yet it is throwing me the same error and it completes the execution without any errors. BERT_Large_trained_on_MSMARCO folder has the three files that were downloaded with the pretrained model given the link in readme.md

Need help on figuring this out. Kindly comment if any other details required.

[rodrigonogueira4]

The "nan" values you are seeing are mostly likely because the eval loop could not load any example from dataset_dev.tf. Do your .tf files have the following features: "query_ids", "doc_ids", "label"?

Just for a sanity check, if you use the .tf files provided in the README page, do you see the same error?

[Arjunsankarlal]

I tried the evaluation part alone for the actual dataset that has been given, I did it on a MacBook Pro, After some 12 hrs it just printed one line of evaluation, which had some numbers, so for that it ran well. Later I stopped that process since it was not necessary for me. With the convert_msmarco_to_tfrecord.py file I converted my dataset also I checked the changes that I made it that file for converting my data to tf format.

In the qrels file I have these data, 100 0 1 1 101 0 2 2

And in dev_dataset_path, I have set path to a file which has data in the following format, query_id \t doc_id \t query \t paragraph_related_to_that_query

Is this format correct? or should I replace the 'paragraph_related_to_that_query' with the whole document text?

Also I have duplicated the query multiple times with different paragraph.

[ghost]

Hi, Arjunsankarlal Did you see that error in the code block?

/Users/arjun/Projects/Github/dl4marco-bert/run_msmarco.py:445: RuntimeWarning: invalid value encountered in true_divide

In the code, we initial example_idx as 0. Then there is if statement.

if len(results) == FLAGS.num_eval_docs, then example_idx += 1. Else do nothing.

So if the results never equal num_eval_docs, then the example_idx will be 0. Then all_metrics /=example_idx will raise an error because 0 can't be divided.

Finally, the reason why results can't equal to num_eval_docs is that your total eval data is less than FLAGS.num_eval_docs.

So just add some eval datas and everything will be good.

All the detail can be found at line 363 and below.

[ghost]

There is a Readme that tell us how to create the original ms macro ranking data.

https://github.com/dfcf93/MSMARCO/blob/master/Ranking/README.md

Please read this and you will know how to create your own data.

[Arjunsankarlal]

Hey @frankabc, Thanks for reply. It worked, I added few docs and modified the params to the same count and generated along with few fake documents too.