search

princeton-vl / CornerNet-Lite

BSD 3-Clause "New" or "Revised" License
1.78k stars 431 forks source link

The speed of training is slower when I use more gpus #110

Closed Jack-Cherish closed 5 years ago

Jack-Cherish commented 5 years ago

Thanks for your nice work.

I meet one issue that the speed of training is lowser when I use more gpus.

I trained CornerNet-Squeeze and the training took around 91 hours when I used one gpu. 64311296-36bfc800-cfd6-11e9-9c79-e068730103ae 64314242-e863f680-cfe0-11e9-9337-da57fb7ed02f

I trained CornerNet-Squeeze and the training took around 113 hours when I used four gpus. 64314358-57d9e600-cfe1-11e9-910c-18c5d51249f5 64314367-5c9e9a00-cfe1-11e9-941e-e803ab56a0b7

It will take around 156 hours if I train CornerNet-Squeeze with eight gpus.

I try to increase the number of prefetch_size in CornerNet-Squeeze.json to get more training_queue for loading dataset, but it`s useless.

I also try to increase the number of workers, it does`t work either.

What should I do to solve this problem?

Look forward to your reply.

Jack-Cherish commented 5 years ago

Have replied in e-mail.

MurrayC7 commented 4 years ago

Have replied in e-mail.

I have the same problem as you do. Could you please tell me the solution? Thank you :)

Jack-Cherish commented 4 years ago

Have replied in e-mail.

I have the same problem as you do. Could you please tell me the solution? Thank you :)

FYI. MacHi 2019-12-04 15-22-06

MurrayC7 commented 4 years ago

@Jack-Cherish Okay, thank you.