The Dataflux Accelerated Dataloader for PyTorch with GCS is an effort to improve ML-training efficiency when using data stored in GCS for training datasets. Using the Dataflux Accelerated Dataloader for training is up to 3X faster when the dataset consists of many small files (e.g., 100 - 500 KB).
Apache License 2.0
26
stars
4
forks
source link
Use Transfer Manager for reading and writing PyTorch Lightning checkpoints #97
This speeds up checkpointing so that our implementation is much faster than gcsfs & the previous implementation since we're using Transfer Manager to parallelize uploads and downloads.
I've updated the benchmark numbers, as well as updating the benchmark to also record checkpoint load times.
[x] Tests pass
[x] Appropriate changes to documentation are included in the PR
This speeds up checkpointing so that our implementation is much faster than gcsfs & the previous implementation since we're using Transfer Manager to parallelize uploads and downloads.
I've updated the benchmark numbers, as well as updating the benchmark to also record checkpoint load times.