Error with alternating messages structure in instruct mode

[arthursn]

Issue

I encounter the following error when using models in instruct mode (served in Databricks):

{"error_code":"BAD_REQUEST","message":"Bad request: Chat message input roles must alternate (user -> assistant -> u -> a -> ...) with an optional system at the start. Tool messages are optional and must follow a preceding assistant message containing tool calls.\\n"}

It appears that in instruct mode, the LLM requires an alternating message structure. In other applications, I implemented a placeholder solution that inserts filler messages to ensure the alternation between assistant and user messages. Perhaps a similar solution could be implemented in Aider.

Additionally, litellm has a user_continue_message feature that seems to partially address this issue: litellm documentation.

Version and model info

Aider v0.58.1 Model: Meta Llama 3.1 70B Instruct in Databricks

[fry69]

Thank you for filing this issue.

I recently experimented with this and found out that Anthropic's API is also very sensitive to the order of roles in the messages array. Since Anthropic's API does not complain, I assume that aider mostly conforms to the requested order.

Can you please check the actual messages that get sent, either with starting aider with --verbose or logging the conversations into a file with --llm-history-file <filename> and then see when this error occurs what got sent?

[arthursn]

Thanks for the reply. I've tested it with a few different instruct-mode LLM deployments and I've always managed to eventually trigger the issue. Attached is one of the log files that I have generated. In lines 478 and 483 you can see that there are two consecutive user messages that are sent to the LLM. I believe this is triggering the error.

.aider_history_20241006145022.log

[arthursn]

Raised issue in litellm: https://github.com/BerriAI/litellm/issues/6257

[krrishdholakia]

the LLM requires an alternating message structure

We do this manually for some providers. I believe we can expose a flag to always follow this structure. Would that solve the issue? @arthursn @paul-gauthier

open to alternatives

[arthursn]

Sounds good to me.

[arthursn]

@paul-gauthier litellm==1.49.7 has been released with the ensure_alternating_roles param. Issue is gone if I use this version. However, if I manually set ensure_alternating_roles: true in .aider.model.settings.yml, ModelSettings raises an error. Solution should be as simple as adding the fields ensure_alternating_roles, user_continue_message and assistant_continue_message to it.