search

nyu-dl / dl4marco-bert

BSD 3-Clause "New" or "Revised" License
476 stars 87 forks source link

PyTorch Version #16

Closed Ricocotam closed 5 years ago

Ricocotam commented 5 years ago

Hi, May I suggest a PyTorch version of the weights ? I'm currently figuring out how to provide such a thing, are you interested in a PR ?

rodrigonogueira4 commented 5 years ago

Wouldn't be easier to have a pytorch code? That should be possible by modifiying some weight names in the original Hugging Face's implementation.

Then, the tensorflow checkpoints could be automatically converted to using this tool: https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/convert_tf_checkpoint_to_pytorch.py

Ricocotam commented 5 years ago

I digged into it and I could convert the weights you provided (only tested on Bert-base, i don't have TPU at work :'( ) using the converter you linked, I'm gonna test a prediction and see if I get the same MRR, i'll give feedback.

I think only a small gist could be useful and enough. Once it works i'll publish it and send it to you.

rodrigonogueira4 commented 5 years ago

Great! Thanks a lot for doing this!

Ricocotam commented 5 years ago

New news ! I reproduced the code and I can run the experiments but I have some troubles reading the top1000 files. Did you do any pre-processing on it ? Parsing it with pandas crashes because some passages have tabs in it you don't seem to have this problem in your code. But when I search in the top1000.dev.tsv file I only find between 0 and 12 occurences on the 10 query-ids I tested, did you observe the same thing ? Otherwise it's ready to be tested and published. Thanks for the help :)

rodrigonogueira4 commented 5 years ago

You are probably referring to this issue, right? https://github.com/dfcf93/MSMARCO/issues/31

If yes, one workaround until they fixed it is to ignore the problematic lines.

Ricocotam commented 5 years ago

I finished reproduing your work using your weights but I get catastrophic results, I'll just train again the weights and hope I get similar results.

rodrigonogueira4 commented 5 years ago

Closing this due to inactivity.

Ricocotam commented 5 years ago

Hi, I actually reproduced it but didn't manage having the same results. I'm like two times below the BM25 mark on all metrics. Any idea on what could go wrong ?