This repository contains the code for training and fine-tuning Sparse MoE models for vision (V-MoE) on ImageNet-21k, reproducing the results presented in the paper:
We will soon provide a colab analysing one of the models that we have released, as well as "config" files to train from scratch and fine-tune checkpoints. Stay tuned.
We also provide checkpoints, a notebook, and a config for Efficient Ensemble of Experts (E3), presented in the paper:
Simply clone this repository.
The file requirements.txt
contains the requirements that can be installed
via PyPi. However, we recommend installing jax
, flax
and optax
directly from GitHub, since we use some of the latest features that are not part
of any release yet.
In addition, you also have to clone the Vision Transformer repository, since we use some parts of it.
If you want to use RandAugment to train models (which we recommend if you train
on ImageNet-21k or ILSVRC2012 from scratch), you must also clone the
Cloud TPU repository, and name it
cloud_tpu
.
We release the checkpoints containing the weights of some models that we trained
on ImageNet (either ILSVRC2012 or ImageNet-21k). All checkpoints contain an
index file (with .index
extension) and one or multiple data files (
with extension .data-nnnnn-of-NNNNN
, called shards). In the following
list, we indicate only the prefix of each checkpoint.
We recommend using gsutil to
obtain the full list of files, download them, etc.
gs://vmoe_checkpoints/vmoe_s32_last2_ilsvrc2012_randaug_light1
.
gs://vmoe_checkpoints/vmoe_s32_last2_ilsvrc2012_randaug_light1_ft_ilsvrc2012
gs://vmoe_checkpoints/vmoe_s32_last2_ilsvrc2012_randaug_medium
.gs://vmoe_checkpoints/vmoe_b16_imagenet21k_randaug_strong
.
gs://vmoe_checkpoints/vmoe_b16_imagenet21k_randaug_strong_ft_ilsvrc2012
gs://vmoe_checkpoints/eee_s32_last2_ilsvrc2012
gs://vmoe_checkpoints/eee_s32_last2_ilsvrc2012_ft_cifar100
This is not an officially supported Google product.