search

patil-suraj / question_generation

Neural question generation using transformers
MIT License
1.11k stars 348 forks source link

Procedure for pre-training #69

Closed saichandrapandraju closed 3 years ago

saichandrapandraju commented 3 years ago

Hi, Can anyone explain how to pre-train a t5 for Question_Generation ? Like what are the features and labels for pre-training ?

patil-suraj commented 3 years ago

Hi @saichandrapandraju

If you want to pre-train T5 on question generation then it will be similar to the fine-tuning process. The context and answer (if you have it) will be the input text and the question will be the label.

You can decide the format for input text i.e whether you want to highlight the answer special tokens or just simply prepend it to the context.

I'm a bit curious, why do you want to pre-train T5 for question generation?

saichandrapandraju commented 3 years ago

Thanks @patil-suraj .

Also, you're using another model for extracting answers for each sentence in passage. I used Spacy to extract entities and generated question for each entity.

Your approach is mostly focused on creating 'one word' answers and respective questions. Any thoughts of creating generic questions which depend on entire passage to answer ?

I wanted to explore pre-training part and witness the evolution of the model for diff tasks. That's y I'm pre-training T5 for Question Generation :-)