Can I use the XML-R model?

neulab / awesome-align

A neural word aligner based on multilingual BERT

https://arxiv.org/abs/2101.08231

BSD 3-Clause "New" or "Revised" License

321 stars 46 forks source link

Can I use the XML-R model? #23

Closed sadanyh closed 3 years ago

sadanyh commented 3 years ago

Hi Thank you awesome-align team for the nice tool and the demo. I am trying to use XML-R with your model instead of M-BERT. I get an error with this line: out_src = model(ids_src.unsqueeze(0), output_hidden_states=True)[2][align_layer][0, 1:-1]

My guess is that the hidden_states and model output for XML-R is different. I think the index for the word embeddings in XML-R is index [-1]. Could you please give your direction on how to solve this, if it is possible?

Thanks

zdou0830 commented 3 years ago

Hi, right now the XLM-R-related code is not included in this repo as we found that it underperformed mBERT as in the paper. I'll upload the XLM-R code to a different branch.

sujoysarkarcs commented 3 years ago

Hi, thanks for the awesome tool. Any update on using XLM-R Code? In which branch can I find it?

zdou0830 commented 3 years ago

Hi @sujoysarkarcs, sorry for the delay. I'll upload the code by 08/16.

zdou0830 commented 3 years ago

Uploaded the code to https://github.com/neulab/awesome-align/tree/xlmr.

sujoycs commented 3 years ago

Thank you.