[Bug]: ERROR: `to_replace` appears more than once, please include enough lines to make code in `to_replace` unique.

[imeDevelopers]

Is there an existing issue for the same bug?

• [X] I have checked the troubleshooting document at https://docs.all-hands.dev/modules/usage/troubleshooting
• [X] I have checked the existing issues.

Describe the bug

File editing background operations, the terminal prints "ERROR: to_replace appears more than once, please include enough lines to make code in to_replace unique."

My logical question: Why do you f* telling the LLM to modify the files (executing edit_file_by_replace and other functions) only one by one?!

just execute the functions the model asking to be executed "as needed". don't add dumb limits ..

the LLM logically want to execute function calls as a list, but you telling him every f* time to only execute one function! wtf!

the model "always" stuck in this case (it don't know what to do!) ..

open_file('/workspace/app/Http/Controllers/MessagesController.php')
scroll_down()
scroll_down()
edit_file_by_replace(
  '/workspace/app/Http/Controllers/MessagesController.php',
  to_replace='        $this->firestore->collection(\'Messages\')->document($id)->delete();\n        return redirect()->route(\'messages.index\');\n    }',
  new_content='    }',
)

ERROR: to_replace appears more than once, please include enough lines to make code in to_replace unique.

just do the f* replace operation!

model: Gemini 1.5 Flash latest.

here's the terminal logs, take a look at how many it stuck: (I can't include it because of huge files contents), but it's the operation number 50!

CodeActAgent LEVEL 0 LOCAL STEP 51 GLOBAL STEP 51

04:07:04 - openhands:INFO: llm.py:529 - Cost: 0.01 USD | Accumulated Cost: 0.49 USD
Input tokens: 124716
Output tokens: 702

04:07:04 - ACTION
**IPythonRunCellAction**
CODE:
open_file('/workspace/app/Http/Controllers/MessagesController.php')
scroll_down()
scroll_down()
edit_file_by_replace(
  '/workspace/app/Http/Controllers/MessagesController.php',
  to_replace='        $collection = $this->firestore->collection(\'Messages\');\n        $documents = $collection->limit(10)->documents();\n        $messages = [];\n        foreach ($documents as $document) {\n            if ($document->exists()) {\n                $messages[] = $document->data();\n            }\n        }\n        return view(\'messages.index\', [\'messages\' => $messages]);\n    }\n    {',
  new_content='    {',
)
edit_file_by_replace(
  '/workspace/app/Http/Controllers/MessagesController.php',
  to_replace='        $document = $this->firestore->collection(\'Messages\')->document($id)->snapshot();\n        if ($document->exists()) {\n            return view(\'messages.show\', [\'message\' => $document->data()]);\n        }\n        return redirect()->route(\'messages.index\');\n    }\n    {',
  new_content='    {',
)
edit_file_by_replace(
  '/workspace/app/Http/Controllers/MessagesController.php',
  to_replace='        $data = $request->validate([\n            // Add your validation rules here\n        ]);\n        $this->firestore->collection(\'Messages\')->add($data);\n        return redirect()->route(\'messages.index\');\n    }\n    {',
  new_content='    {',
)
edit_file_by_replace(
  '/workspace/app/Http/Controllers/MessagesController.php',
  to_replace='        $document = $this->firestore->collection(\'Messages\')->document($id)->snapshot();\n        if ($document->exists()) {\n            return view(\'messages.edit\', [\'message\' => $document->data()]);\n        }\n        return redirect()->route(\'messages.index\');\n    }\n    {',
  new_content='    {',
)
edit_file_by_replace(
  '/workspace/app/Http/Controllers/MessagesController.php',
  to_replace='        $data = $request->validate([\n            // Add your validation rules here\n        ]);\n        $this->firestore->collection(\'Messages\')->document($id)->set($data);\n        return redirect()->route(\'messages.index\');\n    }\n    {',
  new_content='    {',
)
edit_file_by_replace(
  '/workspace/app/Http/Controllers/MessagesController.php',
  to_replace='        $this->firestore->collection(\'Messages\')->document($id)->delete();\n        return redirect()->route(\'messages.index\');\n    }',
  new_content='    }',
)

ERROR: to_replace appears more than once, please include enough lines to make code in to_replace unique.

Current OpenHands version

0.9.2

Installation and Configuration

.

Model and Agent

No response

Operating System

No response

Reproduction Steps

No response

Logs, Errors, Screenshots, and Additional Context

No response

[imeDevelopers]

I am not sure what is the problem, but this really made me mad, because of the stupidness executing the same command 50 times! (in the background, while the user waiting and see nothing the browser prompt/conversation window!).

[enyst]

Just to clarify, this text: 'ERROR: to_replace appears more than once, please include enough lines to make code in to_replace unique.' appears when the LLM wants one replacement that is something like replace "I want to replace this", with "text1", in a file that looks like this:

long file
with things
I want to replace this thing
and more stuff
and maybe I want to replace this
or not

So the backend rejects this, because there are 2 or more possible replacements it can do, at line 3 or line 5. It's not clear which one the LLM intended, the first or the second.

So we ask it to give a replacement text like "I want to replace this thing", which is unique in the file.

[enyst]

On the second issue: can you please confirm that the log you show, from STEP 51, is actually as Gemini gives it? It includes multiple scroll down() and edits in the SAME code block of one single action?

[imeDevelopers]

Just to clarify, this text: 'ERROR: to_replace appears more than once, please include enough lines to make code in to_replace unique.' appears when the LLM wants one replacement that is something like replace "I want to replace this", with "text1", in a file that looks like this:

long file
with things
I want to replace this thing
and more stuff
and maybe I want to replace this
or not

So the backend rejects this, because there are 2 or more possible replacements it can do, at line 3 or line 5. It's not clear which one the LLM intended, the first or the second.

So we ask it to give a replacement text like "I want to replace this thing", which is unique in the file.

Logically: If you don't provide a function like "edit_file_by_replace_all" to replace "all" matched strings in the text. then (logically) you SHOULD replace first MATCH (everytime), this way the model will be able to remove all "duplicates" in the project code files by "iterating" over the string ... (replacing then checking then replacing again and so on .. LOGICALLY if you don't provide a way to replace all!)

When you "ask" .. the model don't understand what are you asking for! or what are you want from it! so it keep trying as your error message is not logically expected at all .. and don't provide any instructions or clarifications to the model .. it's just you're wrong! (the model: Okay, should I try again!!, and here stuck in a loop.. while its following your instructions!!)

[imeDevelopers]

@enyst

On the second issue: can you please confirm that the log you show, from STEP 51, is actually as Gemini gives it? It includes multiple scroll down() and edits in the SAME code block of one single action?

Yes, 100% sure!

[enyst]

@enyst

On the second issue: can you please confirm that the log you show, from STEP 51, is actually as Gemini gives it? It includes multiple scroll down() and edits in the SAME code block of one single action?

Yes, 100% sure!

Strange, I didn't replicate this. I just tried with Gemini-1.5-Flash-latest and it didn't even try to add a second action. It did edits one by one, then tried to run stuff, action by action. I'll try some more complex issues.

[enyst]

Logically: If you don't provide a function like "edit_file_by_replace_all" to replace "all" matched strings in the text. then (logically) you SHOULD replace first MATCH (everytime), this way the model will be able to remove all "duplicates" in the project code files by "iterating" over the string ... (replacing then checking then replacing again and so on .. LOGICALLY if you don't provide a way to replace all!)

When you "ask" .. the model don't understand what are you asking for! or what are you want from it! so it keep trying as your error message is not logically expected at all .. and don't provide any instructions or clarifications to the model .. it's just you're wrong! (the model: Okay, should I try again!!, and here stuck in a loop.. while its following your instructions!!)

You might be right that a "replace all" would help. I'm not sure, there is also a possibility that the models do better with fewer actions than more actions, and specially similar actions can be confusing rather than helpful. FWIW I've seen them forget parameters for one or another when they had something similar without parameters. CC @tobitege @xingyaoww

Edited to add: actually, a "replace all" would also have another problem: the LLM doesn't necessarily know the full file, when it asks for replace. It sees currently 100 lines iirc, then scrolls down, as your log shows. So it wouldn't necessarily know all replacements to be made with a hypothetical "replace all".

Same goes for the first match. I don't know, it seems to require a number of assumptions to be made by us.

When you "ask" .. the model don't understand what are you asking for!

I've seen this message work FWIW, the LLM can choose the next time more text to include. It depends on the LLM, a lot. Sadly it seems Gemini doesn't work well with it.

There's another aspect here. According to your report, it kept trying the same thing, stuck in the same edits. Did you by any chance send feedback to us? You can use "thumbs down" (or "thumbs up") in the UI to send the trajectory of the current task. Please note that it does send it all, the history of actions and observations. It would be useful to dig into, to see what happens. If not, maybe you can attach a log?

[imeDevelopers]

@enyst I already did it, I sent a report using thumb down button.

[enyst]

Do you have the link to the feedback you sent?

[imeDevelopers]

@enyst Sorry, so sorry I posted the wrong link, here's the updated one: https://www.all-hands.dev/share?share_id=11087ec6a2a91805f672927839fd6c7d025f58911cce8acb37f2f7d970f7f4c2

[enyst]

@imeDevelopers In history, I see a lot of:

Permission error during file operation: [Errno 13] Permission denied: '/workspace/...'

That looks like openhands did not succeed in some operations that need temporary files in its workspace. I'd suggest to check if the openhands user has permission to create files in its workspace.