Herb-AI / HerbBenchmarks.jl

Benchmarks and problems for Herb.jl
https://herb-ai.github.io/
MIT License
1 stars 0 forks source link

Fix Brute benchmarks #39

Closed THinnerichs closed 3 months ago

THinnerichs commented 5 months ago

2 comments:

  1. I really like your idea of directly matching and evaluating the AST. However, the hard-coded matching of rulenode indices with their respective funciton does not allow for 1. shuffle/alteration of rule order, and 2. compositions of rules, as we need it for multiple projects of ours.
  2. I think we formulated that we stick with the notation of _arg_1, _arg_2, ... as names for input symbols.
sebdumancic commented 5 months ago
  1. not to difficult to rewrite so that the interpreter matches the symbols in the referenced rules.
  2. easy fix
sebdumancic commented 5 months ago

I've reworked the interpreter so that it doesn't depend on the order of the rules anymore.

I think this is a general principle we can follow for any interpreter. The main idea is to tag every derivation rule with the symbol of the operation:

function get_relevant_tags(grammar::ContextSensitiveGrammar)
    tags = Dict{Int,Symbol}()
    for (ind, r) in pairs(grammar.rules)
        tags[ind] = if typeof(r) == Symbol
                        r
                    else
                        @match r.head begin
                            :block =>  :OpSeq
                            :call =>  r.args[1]             
                            end
                        end
                    end
    return tags
end

The interpreter can then be written by matching the tags:

function interpret(prog::AbstractRuleNode, grammartags::Dict{Int,Symbol}, state::RobotState)
    rule_node =  get_rule(prog)

    @match grammartags[rule_node] begin
        :OpSeq => interpret(prog.children[2], grammartags, interpret(prog.children[1], grammar, state)) # (Operation ; Sequence)
        :moveRight => !(state.robot_x == state.size) ? RobotState(state.holds_ball, state.robot_x+1, state.robot_y, state.ball_x, state.ball_y, state.size) : state       #moveright
        :moveDown => !(state.robot_y == state.size) ? RobotState(state.holds_ball, state.robot_x, state.robot_y+1, state.ball_x, state.ball_y, state.size) : state      #moveDown
        :moveLeft => !(state.robot_x == 1) ? RobotState(state.holds_ball, state.robot_x-1, state.robot_y, state.ball_x, state.ball_y, state.size) : state        #moveLeft
        :moveUp => !(state.robot_y == 1) ? RobotState(state.holds_ball, state.robot_x, state.robot_y-1, state.ball_x, state.ball_y, state.size) : state         #moveUp
        :drop => state.holds_ball == 1 ? RobotState(0, state.robot_x, state.robot_y, state.robot_x, state.robot_y, state.size) : state                 #drop
        :grab => can_pickup(state) ? RobotState(1, state.robot_x, state.robot_y, state.ball_x, state.ball_y, state.size) : state                     # grab
        :IF => interpret(prog.children[1], grammartags, state) ? interpret(prog.children[2], grammartags, state) : interpret(prog.children[3], grammartags, state)                      #If statement 
        :WHILE => command_while(prog.children[1], prog.children[2], grammartags, state)              # while loop
        :atTop => state.robot_y == 1            #atTop 
        :atBottom => state.robot_y == state.size   #atBottom 
        :atLeft => state.robot_x == 1            #atLeft 
        :atRight => state.robot_x == state.size   #atRight
        :notAtTop => !(state.robot_y == 1)         #notAtTop
        :notAtBottom => !(state.robot_y == state.size)    # notAtBottom
        :notAtLeft => !(state.robot_x == 1)             #notAtLeft
        :notAtRight => !(state.robot_x == state.size)    # notAtRight
        _ => interpret(prog.children[1], grammartags, state) # Start operation Transformation ControlStatement
    end
end
pwochner commented 4 months ago

@sebdumancic If you could review one of the benchmarks, that would be great.

THinnerichs commented 4 months ago

At the moment, StringStates are not equal when their respective strs are equal. We could either do

Base.==(a::StringState, b::StringState) = a.str == b.str

or check for one of the pointers being nothing with

Base.==(a::StringState, b::StringState) = a.str == b.str && (a.pointer === nothing || b.pointer === nothing)
THinnerichs commented 4 months ago

During search we get a lot of solutions like

Solution:    problem_b156    61      WHILE(isNotUppercase(), drop())

which is basically an if then statement, but is shorter as only two holes have to be filled.

sebdumancic commented 3 months ago

I really like the tests! I think this is already a really great standard. I don't think we need more than this.