search

princeton-nlp / SWE-agent

SWE-agent takes a GitHub issue and tries to automatically fix it, using GPT-4, or your LM of choice. It can also be employed for offensive cybersecurity or competitive coding challenges.
https://princeton-nlp.github.io/SWE-agent/
MIT License
13.4k stars 1.34k forks source link

Ollama vision models? #77

Closed emmettluzar closed 6 months ago

emmettluzar commented 6 months ago

Can I use an Ollama vision model for this? Is there any use or benefit to that? I'm not super familiar with this project yet but I'd like to try to use it for a MATLAB project, which requires a payed account so I was hoping it could use the MATLAB app on my computer or something and also see graphical results with an Ollama vision model. Is any of this possible to any degree? Thanks for the help.

erkinalp commented 6 months ago

matlab source code is text, and open source reimplementation, octave could be used

emmettluzar commented 6 months ago

Or maybe with a Matlab proxy? I heard about those but I'm not sure exactly how they work or how I'd integrate it with this project, and if I could, if swe could see the graphics

erkinalp commented 6 months ago

again, matlab source code is text, and you could use octave, the flagship open source matlab implementation, it can produce graphics, but lacks GUI ought to be easy to attach a debugger to octave and feed the debugger output back to the coding agent

klieret commented 6 months ago

We do support Ollama (see #51 , #5 ), but I don't understand what Ollama vision would be used for. The code is also focusing on python-based tasks, though other languages might be explored eventually.

emmettluzar commented 6 months ago

Okay thank you. Where could I figure out how to do what you just said? And sorry but I'm still unclear on whether using a vision model would have any benefit in the process you're describing. And if I would be "attaching the agent to a debugger" how would that be different from doing that in my actual matlab app rather than the octave thing? Sorry if these are stupid questions I'm still trying to figure all this kind of stuff out.