Suggestions: Terminal support and OpenDevin intergration, further training

[James4Ever0]

OpenDevin has a problem about interacting with terminal. It would be nice to let this RL agent to operate in terminal and use Vim, even play VimGolf.

I haven't seen a single agent/model capable of doing that, but this one has the potential.

---

Further training involves automatic task generation and evaluation.

I can name a few for you:

• Generate random trajectories with random clicks and keystrokes use agent to describe them as text, then train the RL agent to follow the description, finally verify the result by screenshots or keywords
• Mangle Python libraries, collect code running results before, train the agent to fix the code and verify by running the code afterwards
• Develop consensus mechanism between RL agents and human, complete human given tasks, earn cryptos for survival

Also you can check my project for more information.

[BiEchi]

Thanks for your interest in our work. We are currently developing better RL algorithms and integrating larger models for device control. Most of the ideas you mentioned are in scope of our near research - please keep an eye on our research!

[BiEchi]

Closing due to inactivity.

[James4Ever0]

@BiEchi Developed a terminal interaction environment for agents, capable of converting all info from terminal into meaningful text, including cursor and styling information.

Terminal environment can be captured as image with cursor denoted in red: