Fix Llama3 backtick hallucination in code blocks - Githubissues

OpenInterpreter / open-interpreter

A natural language interface for computers

http://openinterpreter.com/

GNU Affero General Public License v3.0

50.66k stars 4.42k forks source link

Fix Llama3 backtick hallucination in code blocks #1217

Closed CyanideByte closed 2 months ago

CyanideByte commented 2 months ago

Describe the changes you have made:

Fixed a common hallucination of Llama3 inserting a backtick at the start of a code block.

Reference any relevant issues (e.g. "Fixes #000"):

Fixes https://github.com/OpenInterpreter/open-interpreter/issues/1228

  `
  print("Hello, World!")

Cell In[6], line 1 ` ^ SyntaxError: invalid syntax

Pre-Submission Checklist (optional but appreciated):

[x] I have included relevant documentation updates (stored in /docs)
[x] I have read docs/CONTRIBUTING.md
[x] I have read docs/ROADMAP.md

OS Tests (optional but appreciated):

[x] Tested on Windows
[ ] Tested on MacOS
[ ] Tested on Linux

KillianLucas commented 2 months ago

I think this is smart to do, unfortunately I think that it's a parsing issue! I tried printing what Llama was actually generating, and I think it's doing okay, but generating "`" and then "" whereas most LLMs generate all three backticks "```" all at once. Great catch on this problem @CyanideByte, llama 3 should work with OI! Merging.