frankkramer-lab / aucmedi

a framework for Automated Classification of Medical Images
https://frankkramer-lab.github.io/aucmedi/
GNU General Public License v3.0
38 stars 12 forks source link

TF dataset: Improve CPU performance #191

Closed muellerdo closed 1 year ago

muellerdo commented 1 year ago

Add prefetch to the NeuralNetwork model based on max_queue_size.

Should we rename it to be consistent to TF?

Also workers /multiprocessing` parameter can be removed as no generators/sequence from keras is used anymore in the rework.datagenerator branch.

https://stackoverflow.com/questions/63796936/what-is-the-proper-use-of-tensorflow-dataset-prefetch-and-cache-options https://cs230.stanford.edu/blog/datapipeline/#best-practices https://www.tensorflow.org/datasets/performances

muellerdo commented 1 year ago

Implemented via 2cdb8d363d391e25056ae28873c2fefcf91866e5.

muellerdo commented 1 year ago

https://stackoverflow.com/questions/61370108/tf-data-parallelize-loading-step

https://stackoverflow.com/questions/47086599/parallelising-tf-data-dataset-from-generator

muellerdo commented 1 year ago

Withdraw from tf.datasets. See issue: #194