search

nyu-dl / dl4marco-bert

BSD 3-Clause "New" or "Revised" License
476 stars 87 forks source link

Ranking more than 1000 documents #45

Closed Yastil closed 3 years ago

Yastil commented 3 years ago

Hi,

I am trying to rank approximately 17000 documents per query. To achieve this, I set NUM_EVAL_DOCS = 17000. However, when I run evaluation using the provided MS MARCO fine-tuned model on Colab I get the following output;

WARNING:tensorflow:[nan nan nan nan nan]

The input to the model was converted to tfrecord using the provided code. Notably, the above output changes to the following after reading 20000 examples;

WARNING:tensorflow:[0. 0. 0. 0. 0.]

Do you perhaps understand why this is happening?

Yastil commented 3 years ago

Update: The code works just fine when ranking 1000 documents only

rodrigonogueira4 commented 3 years ago

Did you change num_eval_docs to 10k?

Yastil commented 3 years ago

Thank you for the response. Under the parameter initialization, I set NUM_EVAL_DOCS to 17k . Is that the same as what you are referring to?

rodrigonogueira4 commented 3 years ago

Sorry for taking long to reply. When you created the TFRecord, did you set num_eval_docs=1000 in the convert_msmarco_to_tfrecord.py script? https://github.com/nyu-dl/dl4marco-bert/blob/f1d18d63271d9e53cd23fc27bd7cc911f89b89db/convert_msmarco_to_tfrecord.py#L63

Yastil commented 3 years ago

I think I may have found a solution to my problem. Specifically, I needed to re-rank 17k documents. Setting num_eval_docs to 17k did not work, but when I set it to 20k it worked. Not sure why, but it worked.