search

paralleldrive / sudolang-llm-support

SudoLang LLM Support for VSCode
MIT License
1.07k stars 71 forks source link

Hallucinations in the teach() program. #2

Closed jdriscoll98 closed 1 year ago

jdriscoll98 commented 1 year ago

I started using the teach() command and it did fine for lessons that were in the specification. However, after I finished "level 2" , it got confused and thought I was on "level 1". I tried to get it back on track but it started teaching me javascript but calling it SudoLang.

Maybe the teach program should be a bit more restrained, and once it reaches beyond what's in the specification, have it exit the teach() program and enter some sort of "playground" where the user can learn by doing.

Edit: Amazing repo btw

jdriscoll98 commented 1 year ago

Side node: anyone building anything with langchain + sudolang?

ericelliott commented 1 year ago

Thanks for the report. Which language model were you using?

Simple SudoLang programs sometimes kinda work in GPT-3.5, but GPT-4 works much better.

Also, sometimes LLMs just get confused and you need to start a new session. This is technology in its infancy, and because of that, all SudoLang programs are inherently unpredictable and will make occasional mistakes, even with GPT-4.

Yes, people are building things with Langchain and SudoLang.