search

facebookresearch / XLM

PyTorch original implementation of Cross-lingual Language Model Pretraining.
Other
2.88k stars 497 forks source link

Cross Lingual Text Classification (Sentiment Analysis) Using XLM #252

Open hammad26 opened 4 years ago

hammad26 commented 4 years ago

Hi, I wanted to use XLM for text classification task, basically for sentiment analysis for any language. I want to use English language data for training and then other languages data for prediction. In BERT I came to know that we can provide data with its labels. So, I can provide English language data with sentiment labels and it will generate a model which will serve the required purpose. Wanted to do same thing using XLM. Will use mlm_tlm_xnli15_1024.pth model. I am quite confuse on few things. (Also I am new to pytorch)

  1. Should I use XLM/generate-embeddings.ipynb notebook to fine tune model on my English language data? If yes then it means I will be providing just Text without labels? Also my data is muti-sentences (as there is very well chance that sentiment is coming in last sentence), so its okay to break down it in sentences as labels are not needed for fine tuning?

  2. After fine tuning on English language data, now I will like to make a proper model. (Yes will use tensor[0] layer after fine tuning). Now how should I use this fine tuning results to make a model as I have English language data with labels too. I wanted to add let say just add one more hidden layer and then output layer. Ideally we have to provide train_x, train_y but each train_x is aligned with train_y. In this case how will I do that?

Will be thankful to you.

chiragsanghvi10 commented 4 years ago

Hi @hammad26 were you able to predict the sentiments for other languages using XLM? If yes, can you please the same/ how did you achieve that?

hammad26 commented 4 years ago

Hi @chiragsanghvi10 Yes, I was able to do that by exploring simpletransformers repo. Great and simple work is done here.