Reading: Learning Cross-Lingual Sentence Representations via a Multi-task Dual-Encoder Model

[a1da4]

0. Paper

@inproceedings{chidambaram-etal-2019-learning, title = "Learning Cross-Lingual Sentence Representations via a Multi-task Dual-Encoder Model", author = "Chidambaram, Muthu and Yang, Yinfei and Cer, Daniel and Yuan, Steve and Sung, Yunhsuan and Strope, Brian and Kurzweil, Ray", booktitle = "Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019)", month = aug, year = "2019", address = "Florence, Italy", publisher = "Association for Computational Linguistics", url = "https://www.aclweb.org/anthology/W19-4330", doi = "10.18653/v1/W19-4330", pages = "250--259", }

Article is here

1. What is it?

In this paper, the authors proposed a novel approach for cross-lingual representation learning using Universal Sentence Encoder.

2. What is amazing compared to previous studies?

They construct a multitask training scheme using

• native source|target language tasks
• bridging translation task

3. Where is the key to technologies and techniques?

The key is Multi-Task Dual-Encoder Model.

Input sentence sIi and response sentence sRi, and seek to rank sRi over all other possible response sentences.
Maximize the log-likelihood, P(sRi|sIi) for each task.

However, P(sRi|sIi) is hard to calculated, so they used P'(sRi|sIi) as below:

They used 2 USE (Transformer encoder based) to embed each sentence. To calculate the sentence representation, they used the average of each position of words in a sentence.

4. How did validate it?

They evaluated their learned representation using monolingual and cross-lingual tasks. Their model achieved near-state-of-the-art or state-of-the-art performance on a variety of English tasks.

5. Is there a discussion?

6. Which paper should read next?

[thak123]

Did you get any reference code for this paper ?

[a1da4]

Did you get any reference code for this paper ?

Sorry, I didn’t. But you can use pre-trained model. [TensorFlow Hub]

[thak123]

No I wanted to train the model from scratch to reproduce the results