search

libffcv / ffcv

FFCV: Fast Forward Computer Vision (and other ML workloads!)
https://ffcv.io
Apache License 2.0
2.87k stars 179 forks source link
data-science machine-learning pytorch

readme

Fast Forward Computer Vision: train models at a fraction of the cost with accelerated data loading!

[install] [quickstart] [features] [docs] [support slack] [homepage] [paper]
Maintainers: Guillaume Leclerc, Andrew Ilyas and Logan Engstrom

ffcv is a drop-in data loading system that dramatically increases data throughput in model training:

Keep your training algorithm the same, just replace the data loader! Look at these speedups:

ffcv also comes prepacked with fast, simple code for standard vision benchmarks:

Installation

Linux

conda create -y -n ffcv python=3.9 cupy pkg-config libjpeg-turbo opencv pytorch torchvision cudatoolkit=11.3 numba -c pytorch -c conda-forge
conda activate ffcv
pip install ffcv

Troubleshooting note 1: if the above commands result in a package conflict error, try running conda config --env --set channel_priority flexible in the environment and rerunning the installation command.

Troubleshooting note 2: on some systems (but rarely), you'll need to add the compilers package to the first command above.

Troubleshooting note 3: courtesy of @kschuerholt, here is a Dockerfile that may help with conda-free installation

Windows

Citation

If you use FFCV, please cite it as:

@inproceedings{leclerc2023ffcv,
    author = {Guillaume Leclerc and Andrew Ilyas and Logan Engstrom and Sung Min Park and Hadi Salman and Aleksander Madry},
    title = {{FFCV}: Accelerating Training by Removing Data Bottlenecks},
    year = {2023},
    booktitle = {Computer Vision and Pattern Recognition (CVPR)},
    note = {\url{https://github.com/libffcv/ffcv/}. commit xxxxxxx}
}

(Make sure to replace xxxxxxx above with the hash of the commit used!)

Quickstart

Accelerate any learning system with ffcv. First, convert your dataset into ffcv format (ffcv converts both indexed PyTorch datasets and WebDatasets):

from ffcv.writer import DatasetWriter
from ffcv.fields import RGBImageField, IntField

# Your dataset (`torch.utils.data.Dataset`) of (image, label) pairs
my_dataset = make_my_dataset()
write_path = '/output/path/for/converted/ds.beton'

# Pass a type for each data field
writer = DatasetWriter(write_path, {
    # Tune options to optimize dataset size, throughput at train-time
    'image': RGBImageField(max_resolution=256),
    'label': IntField()
})

# Write dataset
writer.from_indexed_dataset(my_dataset)

Then replace your old loader with the ffcv loader at train time (in PyTorch, no other changes required!):

from ffcv.loader import Loader, OrderOption
from ffcv.transforms import ToTensor, ToDevice, ToTorchImage, Cutout
from ffcv.fields.decoders import IntDecoder, RandomResizedCropRGBImageDecoder

# Random resized crop
decoder = RandomResizedCropRGBImageDecoder((224, 224))

# Data decoding and augmentation
image_pipeline = [decoder, Cutout(), ToTensor(), ToTorchImage(), ToDevice(0)]
label_pipeline = [IntDecoder(), ToTensor(), ToDevice(0)]

# Pipeline for each data field
pipelines = {
    'image': image_pipeline,
    'label': label_pipeline
}

# Replaces PyTorch data loader (`torch.utils.data.Dataloader`)
loader = Loader(write_path, batch_size=bs, num_workers=num_workers,
                order=OrderOption.RANDOM, pipelines=pipelines)

# rest of training / validation proceeds identically
for epoch in range(epochs):
    ...

See here for a more detailed guide to deploying ffcv for your dataset.

Prepackaged Computer Vision Benchmarks

From gridding to benchmarking to fast research iteration, there are many reasons to want faster model training. Below we present premade codebases for training on ImageNet and CIFAR, including both (a) extensible codebases and (b) numerous premade training configurations.

ImageNet

We provide a self-contained script for training ImageNet fast. Above we plot the training time versus accuracy frontier, and the dataloading speeds, for 1-GPU ResNet-18 and 8-GPU ResNet-50 alongside a few baselines.

Link to Config top_1 top_5 # Epochs Time (mins) Architecture Setup
Link 0.784 0.941 88 77.2 ResNet-50 8 x A100
Link 0.780 0.937 56 49.4 ResNet-50 8 x A100
Link 0.772 0.932 40 35.6 ResNet-50 8 x A100
Link 0.766 0.927 32 28.7 ResNet-50 8 x A100
Link 0.756 0.921 24 21.7 ResNet-50 8 x A100
Link 0.738 0.908 16 14.9 ResNet-50 8 x A100
Link 0.724 0.903 88 187.3 ResNet-18 1 x A100
Link 0.713 0.899 56 119.4 ResNet-18 1 x A100
Link 0.706 0.894 40 85.5 ResNet-18 1 x A100
Link 0.700 0.889 32 68.9 ResNet-18 1 x A100
Link 0.688 0.881 24 51.6 ResNet-18 1 x A100
Link 0.669 0.868 16 35.0 ResNet-18 1 x A100

Train your own ImageNet models! You can use our training script and premade configurations to train any model seen on the above graphs.

CIFAR-10

We also include premade code for efficient training on CIFAR-10 in the examples/ directory, obtaining 93\% top1 accuracy in 36 seconds on a single A100 GPU (without optimizations such as MixUp, Ghost BatchNorm, etc. which have the potential to raise the accuracy even further). You can find the training script here.

Features

Computer vision or not, FFCV can help make training faster in a variety of resource-constrained settings! Our performance guide has a more detailed account of the ways in which FFCV can adapt to different performance bottlenecks.

Contributors