search

StevenLiuWen / ano_pred_cvpr2018

Official implementation of Paper Future Frame Prediction for Anomaly Detection -- A New Baseline, CVPR 2018
433 stars 126 forks source link

Training Generator Failed #55

Open wiamadaya opened 3 years ago

wiamadaya commented 3 years ago

Hi, i'm trying to train ped2 dataset and always fail when training the generator, i'm using google colab for this, any idea what could be the problem? RAM issue?

No checkpoint file found.
Training discriminator...
2021-04-20 03:13:26.925774: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:145] Filling up shuffle buffer (this may take a while): 785 of 1000
2021-04-20 03:13:29.551348: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:195] Shuffle buffer filled.
2021-04-20 03:13:29.621960: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-04-20 03:13:34.690085: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
Training generator...
2021-04-20 03:13:51.004137: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:145] Filling up shuffle buffer (this may take a while): 500 of 1000
^C
StevenLiuWen commented 3 years ago

hi @wiamadaya , it seems that the buffer is not full, Filling up shuffle buffer (this may take a while): 500 of 1000, and there might not be enough space for the buffer data. You could try to reduce the buffer_size. The default buffer_size is 1000, you could change it to 128.