facebookarchive / bAbI-tasks

Task generation for testing text understanding and reasoning
Other
898 stars 183 forks source link

Inconsistent output in tasks 5 and 17 #9

Closed danieldjohnson closed 7 years ago

danieldjohnson commented 7 years ago

I am getting incorrect/inconsistent answers generated from tasks 5 and 17.

For task 5:

...
11 John journeyed to the kitchen.
12 Daniel gave John the milk.
13 Daniel moved to the bathroom.
14 Mary journeyed to the kitchen.
15 John gave Daniel the milk.
16 Who received the milk?   John    12
17 Sandra moved to the bedroom.
18 Sandra moved to the garden.
19 John moved to the hallway.
20 Sandra went to the office.
21 John gave Daniel the apple.
22 John journeyed to the bathroom.
23 Daniel gave John the apple.
24 Who received the apple?  John    23

Note that lines 12 and 15 both involve trading milk between Daniel and John. In this case the answer to "Who received the milk?" (16) is John. On lines 21 and 23, there is a trade of the apple, and in this case the answer to "Who received the apple?" (24) is also John. But the order of the trade is reversed between the two cases, so it seems like at least one of these answers is incorrect. (In particular, if the question is supposed to correspond to the most recent transfer, then the first answer, on line 16, is incorrect.)

For task 17:

1 The pink triangle is above of the square.
2 The red square is to the right of of the pink triangle.
...
10 Is the red square to the left of the pink triangle?  yes 1 2

This is wrong according to the facts given. (Also, I think there is an extra "of" in the generated facts.)

Since these are randomly generated, these exact examples might not be directly reproducible, but I was able to generate similar inconsistencies by repeatedly running the generation code for these two tasks.

alexholdenmiller commented 7 years ago

Thanks for flagging, I'll check it out!

alexholdenmiller commented 7 years ago

Fixed in commit c64c49f75845bc399f5116c181d2945bc3770c1c.

Thanks for reporting!

danieldjohnson commented 7 years ago

Thanks for the fast update!

As far as I can tell, task 5 is fixed now. But I am still getting weird behavior for task 17:

1 The blue rectangle is to the right of the red square.
2 The pink rectangle is to the left of the red square.
3 Is the blue rectangle to the right of the pink rectangle? no  1 2
4 Is the red square to the right of the blue rectangle? yes 1 2
5 Is the red square to the left of the blue rectangle?  no  1 2
6 Is the blue rectangle to the right of the pink rectangle? no  1 2
7 Is the pink rectangle to the left of the red square?  no  1 2
8 Is the red square to the right of the pink rectangle? no  1 2
9 Is the red square to the right of the blue rectangle? yes 1 2
10 Is the red square to the left of the blue rectangle? no  1 2

A lot of these answers seem incorrect now. Based on the facts, we should have

pink rectangle, red square, blue rectangle

But then the answer to 3 should be yes, the answer to 4 should be no, the answer to 5 should be yes, etc.

alexholdenmiller commented 7 years ago

Ah you're right, I think my fix didn't cover it like I thought. Here we go: ccd8fd6

Thanks for the followup!