mult-modal / tool learning

[shm007g]

Open Multi-Modal LLMs

• https://github.com/THUDM/VisualGLM-6B
• https://github.com/DAMO-NLP-SG/Video-LLaMA
• https://github.com/melodysdreamj/WizardVicunaLM

[shm007g]

https://lilianweng.github.io/posts/2023-06-23-agent/

Tool Learning

Plugins

• Toolformer: [Toolformer: Language Models Can Teach Themselves to Use Tools, 2023/02, Meta AI]
  • https://github.com/conceptofmind/toolformer
  • https://github.com/TeamDman/toolformer
• GPT4 3rd Party Plugins: https://openai.com/blog/chatgpt-plugins
• Moss: https://github.com/OpenLMLab/MOSS
• microsoft/HuggingGPT: [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face, MSFT, 2023/03] https://github.com/microsoft/JARVIS
• microsoft/TaskMatrix
  • [TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs, 2023/03] https://github.com/microsoft/TaskMatrix/tree/main/TaskMatrix.AI
  • [Low-code LLM: Visual Programming over LLMs, 2023/04] https://github.com/microsoft/TaskMatrix/tree/main/LowCodeLLM
  • [Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models, 2023/05]
• Gorilla: [Gorilla: Large Language Model Connected with Massive APIs, 2023/05, UC Berkeley, Microsoft Research] https://gorilla.cs.berkeley.edu/
• ToolAlpaca: [ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases, 2023/06, iscas] https://github.com/tangqiaoyu/ToolAlpaca
• GPT4Tools: [GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction, 2023/05, Tsinghua] https://github.com/StevenGrove/GPT4Tools
• OpenBMB: [Tool Learning with Foundation Models, 2023/04, Tsinghua]
  • BMTools: https://github.com/OpenBMB/BMTools
  • ToolBench: https://github.com/OpenBMB/ToolBench
  • https://drive.google.com/drive/folders/1OaB-hM7eRiWi3TeqHij24VT9MAqgvC0H
• API-Bank: [API-Bank: A Benchmark for Tool-Augmented LLMs, 2023/04, Alibaba DAMO Academy] https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/api-bank
• https://github.com/public-apis/public-apis
• https://github.com/thunlp/ToolLearningPapers
• https://github.com/csunny/DB-GPT

Web Browsing

• [WebGPT: Improving the factual accuracy of language models through web browsing, 2021/12] https://openai.com/research/webgpt -> GPT4 Browsing Plugins
• BingChat
• [WebCPM: Interactive Web Search for Chinese Long-form Question Answering, ACL2023 清华] https://github.com/thunlp/WebCPM
• [WebGLM: Towards An Efficient Web-enhanced Question Answering System with Human Preferences, KDD2023清华] https://github.com/THUDM/WebGLM
• [Mind2Web: Towards a Generalist Agent for the Web, Ohio州立] https://osu-nlp-group.github.io/Mind2Web/

Planning

• Auto-GPT: https://github.com/Significant-Gravitas/Auto-GPT
• AgentGPT: https://github.com/reworkd/AgentGPT
• BabyAGI: https://github.com/yoheinakajima/babyagi
• ToT: https://github.com/princeton-nlp/tree-of-thought-llm

Embodied AI

• Ghost in the Minecraft: https://github.com/OpenGVLab/GITM
• Voyager, An Open-Ended Embodied Agent in Minecraft with LLMs: https://github.com/MineDojo/Voyager
• Training Socially Aligned Language Models in Simulated Human Society: https://github.com/agi-templar/Stable-Alignment
• https://www.deepmind.com/blog/robocat-a-self-improving-robotic-agent

Agent

• https://llmbench.ai/ Evaluating LLM as Agents

[shm007g]

Synthetic Data Generation

• LLM ICL Distillation: https://www.amazon.science/blog/using-large-language-models-llms-to-synthesize-training-data | [Does Synthetic Data Generation of LLMs Help Clinical Text Mining?] https://arxiv.org/abs/2303.04360
• [SELF-INSTRUCT: Aligning Language Models with Self-Generated Instructions , 2022/12, AllenAI ] https://github.com/yizhongw/self-instruct
• Dialogue Environment

Context Length

• [Extending Context Window of Large Language Models via Positional Interpolation, 2023/06, Meta Platforms] https://arxiv.org/abs/2306.15595