facebookresearch / clutrr

Diagnostic benchmark suite to explicitly test logical relational reasoning on natural language
Other
90 stars 14 forks source link

--train_tasks 1.# always generates few stories from task 1.3 #5

Closed NicolasAG closed 3 years ago

NicolasAG commented 4 years ago

Hi,

I noticed that when running python main.py with --train_tasks 1.2,1.4,1.6 it also generated a few stories from task 1.3.

This is the detailed command I run to reproduce the bug: python main.py --train_tasks 1.2,1.4,1.6 --test_tasks 1.2,1.3,1.4,1.5,1.6,1.7,1.8,1.9,1.10 --train_rows 100000 --test_rows 10000 --equal --template_split which generated the following training lines:

A similar behavior is observed with --train_tasks 1.2,1.4,1.8.

I didn't try with other tasks than 1.#

Thanks :)

koustuvsinha commented 3 years ago

Should be fixed with GLC integration #9. @NicolasAG, can you retry this on the develop branch? Closing this for now, feel free to re-open if issue persists.