ARC-AGI Solver by inducing tactics and curriculum learning as in Peano

[nileshtrivedi]

ARC-AGI is a problem-solving benchmark which is claimed to be too difficult for today's machine learning and AI systems. The current best solution scores only about 39%. And there's an ongoing prize for the first team to achieve 85% success on this test.

This is what the tasks for AI to solve look like:

Apparently, humans, can solve these kind of puzzles easily. But because every puzzle requires different transformation rules which are to be learned from very few examples, this is a really hard task for AI. Making good progress on this has a strong chance of being key to achieve AGI.

Now, there are many angles to attack this problem. The approach that I want to mentor people for is based on the one taken in the Peano paper which attempts to train an agent on formal math reasoning by inducing reusable abstractions ("tactics") from its own solutions and then building a curriculum to train the second-generation of agents faster.

If you want to showcase your ability to do both deep thinking AND practical engineering, I think this would make a great showcase project.

Here is what I suggest for exploring this project:

• Read the paper.
• You will encounter many new concepts from mathematics / machine-learning / software-engineering which you will need to study.
• This paper comes with code (Python + Rust). Clone the repo, compile and test it on your own machine, and read the code to understand how it is put together. Keep in mind that some config files and docs are missing.
• Replicate the findings of the paper as close as possible.
• Study and experiment whatever is necessary to come up with a real plan for trying this for ARC-AGI problems instead of KhanAcademy math problems. This means you should have concrete ideas for:
  • Building the problem generator and solution evaluator for ARC-AGI puzzles
  • Examples of tactics that you expect the agent to induce from examples
  • The training mechanism

If you can reach this stage and want to proceed with actual implementation, I will spend up to 2 hours per week to guide and mentor you.

---

This repo was inspired by this tweet which bemoaned that most beginning engineers are building cookie-cutter / tutorial-style projects for their CVs, instead of hard and novel projects that demonstrate their ability to do real engineering and also signal their genuine curiosities.

I invite other experienced engineers to offer similar mentorship for world-class project ideas. Good ideas are those which you are currently exploring yourself. You simply take a few more people along with you. You can create an issue in this repo with a detailed description of the project and add appropriate labels.

[nileshtrivedi]

I have proposed a talk for the upcoming FunctionalConf (Nov 2024, Bengaluru) to share my understanding and learning from trying to reproduce the original Peano paper and the latest one about learning math via intrinsic motivation.