search

ucbdrive / few-shot-object-detection

Implementations of few-shot object detection benchmarks
Apache License 2.0
1.1k stars 223 forks source link

How much times during pretraing #108

Closed huhan233 closed 3 years ago

huhan233 commented 3 years ago

I use three days when I train the first stage(pretraing),but it not finish,I want to know is it normal?

xinw1012 commented 3 years ago

Hey sorry for the delayed response!

It's largely dependent on the compute resources you have. We were using 8 GPU P100 GPUs on a single machine and it took us a day or two for the pretraining stage. Also, you may try different training schedules (e.g., 1X) to speed up your training time for quick experiments. This is adopted in COCO training. https://github.com/facebookresearch/detectron2/blob/master/MODEL_ZOO.md

Hope it helps!

MartinPedersenpp commented 2 years ago

A quick question: I am doing the base training on the Pascal VOC2007-2012 datasets with a GTX 1060 6GB GPU and it only says it will take around 10 hours. I can't see how that is possible if it took 8 P100s more than a day. Any thoughts?

Edit: I just realized that I reduced the batch size from the original 16 to 2 to fit the training in my GPU, I am guessing that I would then need to increase my max_iterations to 18.000 * 8 = 144.000 which would then take an infinity to finish? If anyone can confirm, I would love to hear if I am correct.