Progress in video generation may soon make it possible to evaluate robot policies in a completely learned world model. An end-to-end learned simulator of millions of robot environments would greatly accelerate progress in general-purpose robotics and provide a useful signal for scaling data and compute.
To accelerate progress in learned simulators for robots, we're announcing the 1X World Model Challenge, where the task is to predict future first-person observations of the EVE Android. We provide over 100 hours of vector-quantized image tokens and raw actions collected from operating EVE at 1X offices, baseline world model (GENIE-style), and a frame-level MAGVIT2 autoencoder that compresses images into 16x16 tokens and decodes them back into images.
We hope that this dataset will be helpful to roboticists who want to experiment with a diverse set of general-purpose robotics data in human environments. A sufficiently powerful world model will allow anyone to access a "neurally-simulated EVE". The evaluation challenge is the ultimate goal, and we have cash prizes for intermediate goals like fitting the data well (compression challenge) and sampling plausible videos (sampling challenge).
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
Each example is a sequence of 16 first-person images from the robot at 2Hz (so 8 seconds total), and your goal is to predict the next image given the previous ones.
These challenges are largely inspired by the commavq compression challenge. Please read the Additional Challenge Details
We require Python 3.10
or later. This code was tested with Python 3.10.12
.
# Install dependencies and download data
./build.sh
# Source the Python environment
source venv/bin/activate
This repo provides an implementation of the spatio-temporal transformer and MaskGIT sampler as described in Genie: Generative Interactive Environments. Note that this implementation only trains on video sequences, not actions (though it is trivial to add this via an additive embedding).
# Train the GENIE model
python train.py --genie_config genie/configs/magvit_n32_h8_d256.json --output_dir data/genie_model --max_eval_steps 10
# Generate frames from trained model
python genie/generate.py --checkpoint_dir data/genie_model/final_checkpt
# Visualize generated frames
python visualize.py --token_dir data/genie_generated
# Evaluate the trained model
python genie/evaluate.py --checkpoint_dir data/genie_model/final_checkpt
We provide two pre-trained GENIE models, linked in the leaderboard.
# Generate and visualize
output_dir='data/genie_baseline_generated'
for i in {0..240..10}; do
python genie/generate.py --checkpoint_dir 1x-technologies/GENIE_138M \
--output_dir $output_dir --example_ind $i --maskgit_steps 2 --temperature 0
python visualize.py --token_dir $output_dir
mv $output_dir/generated_offset0.gif $output_dir/example_$i.gif
mv $output_dir/generated_comic_offset0.png $output_dir/example_$i.png
done
# Evaluate
python genie/evaluate.py --checkpoint_dir 1x-technologies/GENIE_138M --maskgit_steps 2
See the Dataset Card on Huggingface.
The training dataset is stored in the data/train_v1.1
directory.
Please read the Additional Challenge Details first for clarification on rules.
Email source code + build script + some info about your approach to challenge@1x.tech. We will evaluate your submission on our held-out dataset and email you back with the results.
Please send us the following:
After manually reviewing your code, we run evals in a 22.04 + CUDA 12.3 sandboxed environment like so:
./build.sh # installs any dependencies + model weights you need
./evaluate.py --val_data_dir <PATH-TO-HELD-OUT-DATA> # runs your model on held-out data
magvit2.ckpt
in the dataset download, which are the weights for a MAGVIT2 encoder/decoder. The encoder allows you to tokenize external data to try to improve the metric.(B, 2^18, T, 16, 16)
. Therefore, the compression challenge considers families of models with a factorized pmfs of the form p(x1, x2) = p(x1)p(x2). For sampling and evaluation challenge, a factorized pmf is a necessary criteria.There are different scenarios for evaluation, which vary in the degree of ground truth context the model receives. In decreasing order of context, these scenarios are:
As an example, consider predicting the final token of a video, corresponding to the lower right patch of frame 15. The context the model receives in each scenario is:
The compression challenge uses the "temporally teacher-forced" scenario.
These are evaluation results on data/val_v1.1
.
User | Temporally Teacher-forced CE Loss |
Temporally Teacher-forced Token Accuracy |
Temporally Teacher-forced LPIPS |
Generation Time* (secs/frame) |
---|---|---|---|---|
1x-technologies/GENIE_138M ( --maskgit_steps 2 ) |
8.79 | 0.0320 | 0.207 | 0.075 |
1x-technologies/GENIE_35M ( --maskgit_steps 2 ) |
8.99 | 0.0301 | 0.217 | 0.030 |
*Generation time is the time to generate latents on a RTX 4090 GPU, and excludes the time to decode latents to images.
Beyond the World Model Challenge, we also want to make the challenges and datasets more useful for your research questions. Want more data interacting with humans? More safety-critical tasks like carrying cups of hot coffee without spilling? More dextrous tool use? Robots working with other robots? Robots dressing themselves in the mirror? Think of 1X as the operations team for getting you high quality humanoid data in extremely diverse scenarios.
Email challenge@1x.tech with your requests (and why you think the data is important) and we will try to include it in a future data release. You can also discuss your data questions with the community on Discord.
We also welcome donors to help us increase the bounty.
If you use this software or dataset in your work, please cite it using the "Cite this repository" button on Github.
The following table is necessary for this dataset to be indexed by search engines such as Google Dataset Search.
property | value | ||||
---|---|---|---|---|---|
name | 1X World Model Challenge |
||||
url | https://github.com/1x-technologies/1xgpt |
||||
description | A dataset of over 100 hours of compressed image tokens + raw actions across a fleet of EVE robots. |
||||
provider |
|
||||
license |
|