giotto-ai / giotto-deep

Deep learning made topological.
Other
79 stars 11 forks source link

Dataloader optimization #43

Closed raphaelreinauer closed 3 years ago

raphaelreinauer commented 3 years ago

To achieve high performance, the data loader should have pin_memory=True since, in this way, one can avoid the transfer between pageable and pinned host arrays, see: https://developer.nvidia.com/blog/how-optimize-data-transfers-cuda-cc/ This can result in a 2x increase in speed.

Here is a reference to the code: https://github.com/giotto-ai/giotto-deep/blob/2965f49a5653bc51790bc76c66e21f2e5ca3e65d/gdeep/pipeline/pipeline.py#L204

matteocao commented 3 years ago

done on all the data loaders.