Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks

Zaijing Li^{1 2}, Yuquan Xie¹, Rui Shao^1✉, Gongwei Chen¹,
Dongmei Jiang², Liqiang Nie^1✉

¹Harbin Institute of Technology,Shenzhen    ²Peng Cheng Laboratory, Shenzhen
^✉ Corresponding author  

:fire: code will release soon

:new: Updates

• [08/2024] Arxiv paper released.

:balloon: Optimus-1 Framework

We divide the structure of Optimus-1 into Knowledge-Guided Planner, Experience-Driven Reflector, and Action Controller. In a given game environment with a long-horizon task, the Knowledge-Guided Planner senses the environment, retrieves knowledge from HDKG, and decomposes the task into executable sub-goals. The action controller then sequentially executes these sub-goals. During execution, the Experience-Driven Reflector is activated periodically, leveraging historical experience from AMEP to assess whether Optimus-1 can complete the current sub-goal. If not, it instructs the Knowledge-Guided Planner to revise its plan. Through iterative interaction with the environment,Optimus-1 ultimately completes the task.

:smile_cat: Evaluation results

We report the average success rate (SR), average number of steps (AS), and average time (AT) on each task group, the results of each task can be found in the Appendix experiment. Lower AS and AT metrics mean that the agent is more efficient at completing the task, while $∞$ indicates that the agent is unable to complete the task. Overall represents the average result on the five groups of Iron, Gold, Diamond, Redstone, and Armor.

:hugs: Citation

If you find this work useful for your research, please kindly cite our paper:

@misc{li2024optimus1hybridmultimodalmemory,
      title={Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks}, 
      author={Zaijing Li and Yuquan Xie and Rui Shao and Gongwei Chen and Dongmei Jiang and Liqiang Nie},
      year={2024},
      eprint={2408.03615},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2408.03615}, 
}