search

iggyray / llms-planning

A benchmark for evaluating large language models in planning
0 stars 0 forks source link

get_llm_action_description method is flaky #13

Open iggyray opened 1 month ago

iggyray commented 1 month ago

Issue

    def get_llm_action_description(self, llm_raw_response):
        pddl_action, _ = text_to_plan(llm_raw_response, self.problem.actions, "llm_plan", self.config)
        return get_action_description(self.config, pddl_action)

get_llm_action_description() is flaky because llm_raw_response might not be formatted correctly

Possible Solution

Implement simplify_llm_response() method which passes the llm_raw_response to the LLM and makes it simplify the output in the desired format for get_llm_action_description