Bert fine tuning for MCQ style reading comprehension tasks?

[vikigenius]

Hi all, the Bert paper discusses how to adopt this model for Question Answering/Reading Comprehension tasks like SQUAD where you just combine the passage and question into a single packed sequence and do a token classification for span prediction.

But what happens if you have a Reading Comprehension task like RACE: https://arxiv.org/pdf/1704.04683.pdf, where you have a long paragraph and then multiple MCQ style questions on the paragraph.

The simplest way to reformulate this task would be: (paragraph, question, option) -> {0, 1}

A binary classification task.

What is the most sane way of fine tuning Bert for such a task. Right now Bert can only represent two sequences simultaneously right?

If i do something like: [CLS] long paragraph [SEP] question [SEP] option [SEP] and train a classifier on top of this should it work? Both question and option will have B segmentation embedding (type 1 typeid) right?

[zpqiu]

The SWAG example of pytorch-pretrained-BERT may help you. The ULMFit also show the fine-tuning method for MCQ.

[hfaghihi15]

@vikigenius we are thinking about the same issue. Did you solve your problem?

[vikigenius]

@hfaghihi15, For my purpose, my initial idea of simply concatenating all of them sufficed. The segmentation embedding type doesn't seem to make that much of a difference