Closed nazmi closed 2 years ago
Finished reading 05_finetuning
and just realised you used it from there. I guess I got curious too early.
@nazmi thank you for the suggestion :)
I'm glad you found the updated data loading in the later notebook, I was going to comment something similar.
It's good to be curious!
This will greatly save time for some students instead of waiting for the kernel to finish the run. As a bonus, to teach students how to pipeline the input better. Thanks to your "documentation reading" skills, I have learned to improve things because I hate to wait.
Current behaviour
Your code in
03_convolutional_neural_networks_in_tensorflow
usesImageDataGenerator
. As you said, this method causes more delay in every epoch because it augments the data when it loaded the model.Suggestion
Since
tf.keras.preprocessing
is marked as deprecated (but still usable), maybe we can usetf.keras.utils.image_dataset_from_directory
as to utilizetf.data.Dataset
.Benchmark
Tested on multiclass classification with 20 classes without sampling. 15000 training files and 5000 validation files. Although this dataset is not similar to the notebook example, it shows how it can benefit smaller datasets as well.
ImageDataGenerator
takes 11 minutes and 19.4 seconds.image_dataset_from_directory
takes 1 minute and 37.5 seconds. Improvement speed 6.97 times over normal run.System information