If we load a sufficiently big dataset (using tf.data.dataset ==> TFDS in "not all in memory mode"), the instance crashes with an OOM error. Since we are iteratively using TFDS in batches, this should not be the case, right ... ?
Thus, we can conclude that the model tries to load the entire dataset into memory. Is this behavior normal?
How can we scale this to big-data usage ?
If we load a sufficiently big dataset (using
tf.data.dataset==> TFDS in "not all in memory mode"), the instance crashes with an OOM error. Since we are iteratively usingTFDSin batches, this should not be the case, right ... ?Thus, we can conclude that the model tries to load the entire dataset into memory. Is this behavior normal? How can we scale this to big-data usage ?
THNX!