"is_solved": false on all results

noahshinn / reflexion

[NeurIPS 2023] Reflexion: Language Agents with Verbal Reinforcement Learning

MIT License

2.46k stars 240 forks source link

"is_solved": false on all results #2

Closed drammen94 closed 1 year ago

drammen94 commented 1 year ago

Do we need to adjust some parameters or anything? i tried running both run_reflexion.sh and run_simple.sh but all of the tests fail.

noahshinn commented 1 year ago

Do we need to adjust some parameters or anything? i tried running both run_reflexion.sh and run_simple.sh but all of the tests fail.

Thanks for the note. Can you specify the run parameters that you used? Also, check the latest rewrite. It may make it easier to interpret results now.

drammen94 commented 1 year ago

Hi Noah! Im running the default parameter except im using 3.5 turbo. It appears to fault under execution.

I tried running on jupyter lab to debug further

It says attribute error. Although the code passes when i paste it in jupyter lab.