Synthetic data has slower throughput than imagenet

[growlix]

Environment

• Hardware (GPU, or instance type): 8x COTA 3080

** To reproduce

Steps to reproduce the behavior:

Compare running with composer/yamls/models/resnet50_synthetic.yaml to composer/yamls/models/resnet50.yaml. See W&B runs here.

Expected behavior

Using synthetic data should be at least as fast, if not faster than using actual data. Issue is slightly improved by increasing train_dataset.synthetic.total_dataset_size: 12288 to something larger (e.g. 100000).

[ravi-mosaicml]

I re-ran an experiment using a newer docker image and the latest dev branch of composer. Specifically:

• Base Dockerfile: https://github.com/mosaicml/composer/blob/938ac0028c8622d5c59d8a3c470c85a326b8b6ee/docker/pytorch/Dockerfile
• Docker image: https://hub.docker.com/layers/mosaicml/pytorch/1.9.1_cu111-python3.8-ubuntu20.04/images/sha256-d0f3c9a101f38d01b4b468e77b42d95c4490364302b750e5ee3f7e7ccb365c86?context=repo
• WandB proejct: https://wandb.ai/mosaic-ml/ravi_resnet50_matt_regression?workspace=user-ravimosaicml

Both Synthetic Data and Real data got around 4900 img / second.

I suppose this regression was caused by:

• Some dockerfile misconfiguration (similar to the pillow simd dependnecy)
• Some bad synthetic dataset configuration (which has since been changed between what is on main and the latest which is on dev)

Closing this issue as it will go away with the next release of composer.