search

nyu-dl / dl4marco-bert

BSD 3-Clause "New" or "Revised" License
476 stars 87 forks source link

Evaluate the trained model on my own trec-car data doesn't work. #32

Closed mahsash closed 5 years ago

mahsash commented 5 years ago

I managed to run the code on my own data after reading all issues and resolving errors ( using colab). I uploaded your pre-trained model for trec-car (meta, index, and data files) to my own bucket. I converted my data to tf-record (by using your code) and uploaded it to my bucket too. When I change the "DO_TRAIN" flag to False and run the whole code and get the prediction file, the results are very bad (0.01 MAP!). I guess I'm not using the pre-trained model correctly. Where I should set the path to the trained model (meta, index, and data files uploaded to my bucket) in colab code?

rodrigonogueira4 commented 5 years ago

You should set INIT_CHECKPOINT to point to the checkpoint. For example:

INIT_CHECKPOINT = 'gs://path/to/trained/model.ckpt'

mahsash commented 5 years ago

Thanks for your quick reply. I've set the INIT_CHECKPOINT to point to ckpt-data file, but it says that it should be a .ckpt file (as you mentioned). So, what should I do? Because the model I downloaded from here consists of three files: index, data, and meta file. (So, there is no .ckpt file.)

mahsash commented 5 years ago

OK! I got it (from here) I understood that there is no actual .ckpt file and when we set it to point to model.ckpt it used the three separate meta, index, and data file. It's really strange. But, now it works correctly. Thanks again!

rodrigonogueira4 commented 5 years ago

Yeah, checkpoint loading is confusing in TensorFlow :) I'm glad it worked!