search

alxndrTL / ARC_LLMs

Evaluating majors LLMs on the Abstraction and Reasoning Corpus
14 stars 1 forks source link

Prompts #1

Open philipkd opened 2 months ago

philipkd commented 2 months ago

Thanks for compiling this. What prompts did you use?

alxndrTL commented 2 months ago

For the prompts, its just the grids laid out rows by rows. It goes like this :

input:
1,1,0,5,6,7
1,4,5,6,7,8
4,5,8,1,3,5
output:
1,1,1,1
---
input:
1,1,0,5,6,7
1,4,5,6,7,8
4,5,8,1,3,5
output:\n

I also tried replacing each "color" with a random token of the vocab, that's a good test to see if it's just memorization or not. But no fancy prompt techniques.