search

megvii-research / FSCE

Apache License 2.0
275 stars 47 forks source link

Question about training time #23

Closed ZiyeHu closed 3 years ago

ZiyeHu commented 3 years ago

I cannot experiment with 8 GPUs. Can you tell me how long it takes to train in an 8-GPU environment?

Chauncy-Cai commented 3 years ago

If you have finished base training, you will take less than 3 hour for most of the tasks in finetune stage.

ZiyeHu commented 3 years ago

So how long for the base training? 18110860007 @.*** 签名由 网易邮箱大师 定制 On 04/28/2021 20:43, Chauncy-Cai wrote: If you have finished base training, you will take less than 3 hour for most of the tasks in finetune stage. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

ZiyeHu commented 3 years ago

I have tried to run the "Stage 1: Training base detector" under one GPU, but still "Starting training for iteration 0" after running for 1-2 days: 1 Here is the code: python tools/train_net.py --num-gpus 1 \ --config-file configs/PASCAL_VOC/base-training/R101_FPN_base_training_split1.yml

Chauncy-Cai commented 3 years ago

For your specified task, base training for voc split1, we can finish training in 2.5 hours. I haven't try to train with one gpu. Maybe you can try to decrease the number of image to 2 images per batch. Meanwhile, you should adjust your training step and leanrning rate to maintain the result.

ZiyeHu commented 1 year ago

关<淀叔窦九蔼倾投涂>于2<愁鹤桩谬耘薛强药>02<看渊明>2<寂寞>年<浑未办>个人劳<蚯蚓变蛟纵变不高>动补<世事明如镜，前程暗似漆。>助<料得来宵>（<嫖逗帷蒲柯哉忱全帷怖>补<孜虫泳敏酬宗繁>贴<以国家之务为己任。>）<花明月黯笼轻雾>通知，<针宽帧甘驼>详情请<上了套子的猴子由人玩耍>及时查<多么沉重地我在旅途上跋涉，>收附<裂辨眷皂筋腐短窦强或>件<闺泌漾卤硬>！