pesser / edflow

Reduce boilerplate code for your ML projects. TensorFlow and PyTorch.
https://edflow.readthedocs.io/
MIT License
24 stars 13 forks source link

Benchmark data-loading #242

Open theRealSuperMario opened 4 years ago

theRealSuperMario commented 4 years ago

It might be good to have an easy way to see if the data loading is a bottleneck of the GPU is a bottleneck.

In a second stage, it might be useful to be able to automatically generate a binary file, such as tfrecords, from a specified config file and then load the data from the tf records.

That allows the user to be able to work normally and if the data I/O is the bottleneck, it is automatically binarized to improve performance.