search

Aider-AI / aider

aider is AI pair programming in your terminal
https://aider.chat/
Apache License 2.0
22.11k stars 2.06k forks source link

Gemini Pro Exp edit mode Vertex AI / Gemini and Gemini Pro 1.5 Pro 002 edit mode #2182

Open Jasmin68k opened 2 weeks ago

Jasmin68k commented 2 weeks ago

Issue

Hello!

Just two observations with v0.60.1:

  1. aider uses whole edit mode for vertex_ai/gemini-pro-experimental (probably because it doesn't know it), but the model is the same as gemini/gemini-1.5-pro-exp-0827, see https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/gemini-experimental, so it should probably use diff-fenced, too.

  2. aider uses diff edit mode for gemini/gemini-1.5-pro-002, while all other Gemini Pro models use diff-fenced. On https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/gemini-experimental https://aider.chat/docs/leaderboards/, it says diff-fenced also for gemini/gemini-1.5-pro-002. I don't know, which mode is better for the model (maybe the newer Gemini Pro really works better with diff?), but either aider's model config or the benchmark page is wrong.

Hope it helps.

Version and model info

Aider v0.60.1 Model: gemini/gemini-1.5-pro-exp-0827 // vertex_ai/gemini-pro-experimental // gemini/gemini-1.5-pro-002

paul-gauthier commented 5 days ago

Thanks for trying aider and filing this issue.

You can set the edit mode with --edit-format.

Jasmin68k commented 4 days ago

Thank you!

The reason I opened the issue is more about

  1. Suggesting to add vertex_ai/gemini-pro-experimental to the known models with the same edit mode as gemini/gemini-1.5-pro-exp-0827, since it's the same model, assuming that the reason for Aider to chose whole edit mode for this model is that it does not know of the model.

  2. Wondering what actually is the reason for Aider chosing diff as opposed to diff-fenced for gemini/gemini-1.5-pro-002, while diff-fenced is being described to be the proper edit mode in https://aider.chat/docs/leaderboards/ (sorry, had originally added wrong link in op for this) and making sure the proper/best edit mode is automatically chosen and the information in the leaderboard is correct.

paul-gauthier commented 4 days ago

Done.

The change is available in the main branch. You can get it by installing the latest version from github:

aider --install-main-branch

# or...

python -m pip install --upgrade --upgrade-strategy only-if-needed git+https://github.com/Aider-AI/aider.git

If you have a chance to try it, let me know if it works better for you.

Jasmin68k commented 2 days ago

Thanks!

I tested release v0.63.0:

  1. It uses diff-fenced for vertex_ai/gemini-pro-experimental now, so that is fixed.
  2. Same as in op. Looks like this should be fixed, probably by changing gemini-1.5-pro-002 to use diff-fenced, unless there's a specific reason you chose diff for this version of Gemini Pro, then the leaderboard needs updating.