Real-time Animation Generation and Control on Rigged Models via Large Language Models

This repository serves as a showroom for various animations generated by large language models (LLM) with the approach described in

Huang, Han, et al. "Real-time Animation Generation and Control on Rigged Models via Large Language Models." NeurIPS Workshop on ML for Creativity and Design 2023 (Spotlight). [arXiv]

This work is part of a larger initiative that utilizes LLMs to create virtual experiences. For more details, please see

De La Torre, Fernanda, et al. "LLMR: Real-time Prompting of Interactive Worlds using Large Language Models." arXiv preprint arXiv:2309.12276 (2023). [arXiv]

Our method takes a rigged 3D model and produces novel animations with natural language descriptions in a matter of seconds. Currently, there are multiple limitations in robustness and excess token usage for complex joint hierarchies. However, it is fascinating that LLMs can manage this at all!

Animation Generation

The quoted texts are the prompts used to generate the animation.

Few-shot

"Tilting its head" for a whale	"Swimming aggressively" for a whale

"Flapping its tail" for a whale	"Attempting to fly like a bird" for a whale

"Eating" for a pig	"Jumping" for a pig

"Showing disapproval" for a raccoon	"Bowing" for a raccoon

Zero-shot

"Flying high and low" for a bird	"Cheering" for a stickman	"Jumping" for a person

Animation Control

https://github.com/Whalefishin/LLM_animation/assets/21124870/6523ca9e-3f26-4b34-beeb-9a1c61f60eb5