search

entropy-research / Devon

Devon: An open-source pair programmer
GNU Affero General Public License v3.0
2.36k stars 172 forks source link

Local ollama issues: loops forever on random problems: problems with saving/ovewriting, no closing fence #55

Closed w84death closed 1 month ago

w84death commented 1 month ago

Running local ollama run deepseek-coder:6.7b Kubuntu 24.04, RTX 2060 6GB VRAM

Fail to generate files It loops forever trying to run CreateFileTool.function() but got too many arguments. 

+------------------------------------------------------------------------------------------+
|   Devon is thinking:                                                                     |
|   I understand that we are creating a website about some of my indie games which were    |
|   created using Assembly language for the x86 boot sector. I need to create the          |
|   necessary files and directories for this project.                                      |
|                                                                                          |
+------------------------------------------------------------------------------------------+
+--------------------------------------------------------------------------------------------+
|   Devon ran: Running command:                                                              |
|   create_file index.html                                                                   |
|   ```                                                                                      |
|   <DOCTYPE html>                                                                     |
|   <html>                                                                             |
|   <head>                                                                             |
|        <title>My Indie Games</title>                                           |
|   </head>                                                                            |
|   <body>                                                                             |
|        <h1>Welcome to my indie games website</h1>                              |
|        <p>Here you can find information about some of my favorite indie games, all   |
|    made using Assembly language for the x86 boot sector.</p>                         |
|   </body>                                                                            |
|   </html>                                                                            |
|   ```                                                                                      |
|                                                                                            |
|   > CreateFileTool.function() takes from 3 to 4 positional arguments but 44 were given     |
+--------------------------------------------------------------------------------------------+

Generate first file but fails with updating it After restarting and given same prompt it now creates index.html but then loops forever trying to update the file.

+-------------------------------------------------------------------------------------------+
|   Devon is thinking:                                                                      |
|   I understand the user's request to create a website about his indie games, which are    |
|   written in Assembly language and targeting an x86 boot sector. I will start by          |
|   creating a basic HTML structure for the site.                                           |
|                                                                                           |
+-------------------------------------------------------------------------------------------+
+--------------------------------------------------------------------------------------+
|   Devon ran: Running command:                                                        |
|   create_file index.html                                                             |
|   <<<                                                                                |
|   &lt;DOCTYPE html&gt;                                                               |
|   &lt;html lang="en"&gt;                                                             |
|   &lt;head&gt;                                                                       |
|        &lt;meta charset="UTF-8"&gt;                                                  |
|        &lt;title&gt;My Indie Games&lt;/title&gt;                                     |
|   &lt;/head&gt;                                                                      |
|   &lt;body&gt;                                                                       |
|        &lt;h1&gt;Welcome to my indie games website&lt;/h1&gt;                        |
|   &lt;/body&gt;                                                                      |
|   &lt;/html&gt;                                                                      |
|   >>>                                                                                |
|                                                                                      |
|   > Failed to create file: index.html. Error: Could not create file, file already    |
|   exists: /home/kj/Code/devon/games-web/index.html                                   |
+--------------------------------------------------------------------------------------+

Not valid HTML files The file itself lacks proper TAGs:

&lt;!DOCTYPE html&gt;
&lt;html lang="en"&gt;
&lt;head&gt;
    &lt;meta charset="UTF-8"&gt;
    &lt;title&gt;My Indie Games&lt;/title&gt;
&lt;/head&gt;
&lt;body&gt;
    &lt;h1&gt;Welcome to my indie games website!&lt;/h1&gt;
&lt;/body&gt;
&lt;/html&gt;

This only happens once.

Not closing fence

Third restart (clean folder) it generates valid index.html:

+--------------------------------------------------------------------------------------------+
|   Devon is thinking:                                                                       |
|   I understand the requirement of creating a website for my indie games. I will start by   |
|    creating an HTML file that serves as the main page of our website. This will include    |
|   information about the game catalog, introduction and instructions on how to run them     |
|   (in emulator or from floppy).                                                            |
|                                                                                            |
+--------------------------------------------------------------------------------------------+
+--------------------------------------------------------------------------------------------+
|   Devon ran: Running command:                                                              |
|   create_file index.html                                                                   |
|   <<<                                                                                      |
|   <!DOCTYPE html>                                                                          |
|   <html>                                                                                   |
|       <head>                                                                               |
|           <title>Indie Games Catalog</title>                                               |
|       </head>                                                                              |
|       <body>                                                                               |
|           <h1>Welcome to our Indie Games Catalog</h1>                                      |
|           <p>Here you can find a collection of indie games developed in assembly for x86   |
|    bootsector.                                                                             |
|           Each game comes with an introduction and instructions on how to run them (in     |
|   emulator or from floppy).</<｜begin▁of▁sentence｜>                                       |
|       </body>                                                                              |
|   </html>                                                                                  |
|   >>>                                                                                      |
|                                                                                            |
|   > Successfully created file /home/kj/Code/devon/games-web/index.html                     |
+--------------------------------------------------------------------------------------------+

..but then loops forever on No closing fence (>>>) :

+--------------------------------------------------------------------------------------------+
|   Devon is thinking:                                                                       |
|   I understand the requirement of creating a website for my indie games. I will start by   |
|    creating an HTML file that serves as the main page of our website. This will include    |
|   information about the game catalog, introduction and instructions on how to run them     |
|   (in emulator or from floppy).                                                            |
|                                                                                            |
+--------------------------------------------------------------------------------------------+
+------------------------------------------------------------+
|   Devon ran:                                               |
|   > No closing fence (>>>) found for multiline argument.   |
+------------------------------------------------------------+

Clean the folder (because it can not overwrite files) and it's the same thing with good html and then No closing fence loop.

emlazzarin commented 1 month ago

I have the same issue consistently with gpt-4o, so I don't think this is specific to local ollama.

akiradev0x commented 1 month ago

@emlazzarin @w84death Hey guys! thanks for this, I am going to try to find a fix for it, seems like a bug in the parsing code. Will fix today :)

akiradev0x commented 1 month ago

Just solved this with an update to the parser!

Also this seems to sometimes be a model hallucination problem. At least for deepseek. I'll be adding support to enforce output structure soon. Opened an issue for this: #62

closing!