search

michaelhodel / re-arc

Reverse Engineering the Abstraction and Reasoning Corpus
MIT License
194 stars 33 forks source link

Some original tasks fail #8

Closed rteehas closed 2 months ago

rteehas commented 2 months ago

Hi,

I noticed that verification on the original ARC tasks does not give 100% success. It seems like it fails on 6 tasks. Specifically, I get the following output:

verification programs work for all examples for 394/400 tasks
verification fails (on one example) for tasks {'a64e4611', '7e0986d6', '53b68214', '29ec7d0e', '4290ef0e', '97a05b5b'}
michaelhodel commented 2 months ago

That is true, but not an issue, as I don't claim otherwise and 100% coverage of the original examples isn't the purpose of the verifiers in the first place.

rteehas commented 2 months ago

Ah, no worries then. My mistake!