train_and_eval : eval is not working anymore for big dataset

google / automl

Google Brain AutoML

Apache License 2.0

6.25k stars 1.45k forks source link

train_and_eval : eval is not working anymore for big dataset #526

Open fitoule opened 4 years ago

fitoule commented 4 years ago

Train_and_eval works like a charm on small dataset --num_examples_per_epoch=5000 or less but if I try on 25000 images or 64000 (person cocodataset). Eval is not working any more .. it is bypassed ... So question on a 50 000 Images dataset if I set num_examples_per_epoch=5000 Is the training will finallly browse all the 50 000 images if I have a lot of Epoch : ( above 10 ) ? Thank You My GPU is a NVIDIA 2060RTX 8GO RAM

fitoule commented 4 years ago

same behaviour on colab :(

Samjith888 commented 4 years ago

Increase the num_examples_per_epoch when you are using bigger dataset

fitoule commented 4 years ago

Increase the num_examples_per_epoch when you are using bigger dataset

I know but I have more 200 000 images. And I made tests on my RTX 8Go RAM and same on colab above 5000 - 10000 num_examples_per_epoch I have a cupti message error lib not found and there is no eval at all. So I decrease the num_examples_per_epoch under 8000 and then EVAL works again ...