fchollet / ARC-AGI

The Abstraction and Reasoning Corpus
Apache License 2.0
3.31k stars 548 forks source link

Ambiguous rule for 070dd51e + some questions #134

Open maxficco opened 1 month ago

maxficco commented 1 month ago

070dd51e

Example 1 Example 2

Rule seems pretty straightforward:

But when I saw the puzzle, instead of thinking in terms of vertical/horizontal lines, I thought the rule was:

Looking at the examples, this rule would be supported and produces the incorrect result:

Incorrect Solution

This ambiguity could be avoided by providing an example where two lines of equal length intersect (and the vertical line overlaps the horizontal line). For example: shorten the distance between the maroon (color 9) dots in Example 2 by so that the light blue (8) and maroon (9) lines both have a length of 6.


This brings up other concerns: How is it made clear by these two examples that color is not an important factor in determining overlap? Could I come to the belief that green must always be drawn beneath other lines? What about line lengths? Can I say that all lines of length six must be overlapped when in intersection? What about interactions between two different colored lines?

Surely I could keep going with an infinite list of increasingly complex rules. So, should an agent solving the puzzle be required to ascertain the most "simple" solution, or should any rule outside of the correct solution be necessarily contradicted by the examples? Is the latter even possible? How is this simplicity defined? Is there a method by which these puzzles are checked/proven to be solvable (with only one solution)?

I am new to the arc challenge and have not read all of François Chollet's paper, so my questions are not a critique but simply an attempt at understanding. Please let me know if you have any thoughts! I'd be happy to submit a PR if other people believe it to be necessary. Thanks, Max

GaussianMongrel commented 2 weeks ago

Hi. You pose some tough questions!

"How is it made clear by these two examples that color is not an important factor in determining overlap?" Who said color order rules aren't expected to be part of the information acquisition need of an AGI?

"So, should an agent solving the puzzle be required to ascertain the most "simple" solution, or should any rule outside of the correct solution be necessarily contradicted by the examples?" I would expect that some kind of optimization is occurring in any solution. Simplicity for your approach might be depth of rules actioned, for example.

"Is there a method by which these puzzles are checked/proven to be solvable (with only one solution)?" Yes. If your output equals the answer. That's the verification method.

"I am new to the arc challenge and have not read all of François Chollet's paper" A great place to start is to read the entire paper, a few times.

Hope this has helped. I think the issue can be closed. Cheers GM

maxficco commented 2 weeks ago

Who said color order rules aren't expected to be part of the information acquisition need of an AGI?

Unfortunately I don't think you answered any of my questions. I understand the premise of how these models should be solved, but I was curious how to differentiate between solutions when each solution could be valid given the implied rule set. To your first response, I obviously understand color is important to the arc challenge, but I was curious for this specific example how we should know for certain that color actually wasn't important in the solution.

Feels a bit like you fed my post into an LLM and copied it here. I don't believe the issue has been solved or should be closed.

Thanks, Max