search

OSU-NLP-Group / TableLlama

[NAACL'24] Dataset, code and models for "TableLlama: Towards Open Large Generalist Models for Tables".
https://osu-nlp-group.github.io/TableLlama/
MIT License
102 stars 8 forks source link

[possible bug] The way using highlighted cells in evaluation of TableLlama #8

Open giangnguyen2412 opened 4 months ago

giangnguyen2412 commented 4 months ago

Hello authors,

I am concerning about the way you used the highlighted cells in test for TableLlama.

Taking FeTaQA as an example,

From the paper, it looks like the highlighted cells in test should be used to evaluate the model reasoning rather than be used as the part of the prompt.

Currently, the "question" field in the prompts to TableLlama is like this:

### Question:
The highlighted cells of the table are: [HIGHLIGHTED_BEGIN] [KSW 18: Unfinished Sympathy], [February 25, 2012], [Orlen Arena], [Poland Płock, Poland], [KSW 21: Ultimate Explanation], [Hala Torwar], [Poland Warsaw, Poland] [HIGHLIGHTED_END] Were any KSW events in 2012 not held in arenas?

This contains the highlighted cells. Could the authors help to clarify this/correct me?

Thanks a lot!

giangnguyen2412 commented 4 months ago

The main author confirms in an email that:

Hi Giang,

The initial point of incorporating FeTaQA is we want TableLlama has the ability to predict text based on highlighted cells, that’s why we format the input like this and we call it highlighted cells QA instead of just table QA. But you are correct that the original FeTaQA paper doesn’t directly inject the highlighted cells into the input for test.