search

yaoxingcheng / TLM

ICML'2022: NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Framework
MIT License
257 stars 21 forks source link

I don't notice the Task-driven function anywhere. #10

Closed Huynh-Chinh closed 2 years ago

Huynh-Chinh commented 2 years ago

Hi yaoxingcheng,

Thanks for releasing the code and paper for your task-driven language modeling approach.

I tried running your code. But I don't notice the Task-driven functionality anywhere. Can you explain a little more the difference between having Task-driven and traditional LM training? thank you very much.

Best, ChinhH framework

yaoxingcheng commented 2 years ago

Hi ChinhH. In the released code, data selection is decomposed from LM training. You can read the scripts in src/data_selection.py to see how task data is used to retrieve similar data from general corpus (which is one of the key differences between TLM and PLM). Also, in line 113 of the code in src/model.py, you can see the loss of the model is an weighted average of task objective and LM objective. Such multi-task learning scheme is also one of the points that makes a difference.

Huynh-Chinh commented 2 years ago

thanks for your support. I have 1 question about data set to train Language Modeling. Is it generated from source.csv+selected.csv ? Screenshot from 2022-02-16 09-58-32

yaoxingcheng commented 2 years ago

We only use selected data (in your case selected_rake.csv) as the external data for language modeling

Huynh-Chinh commented 2 years ago

Thanks for your answer