search

OlafenwaMoses / ImageAI

A python library built to empower developers to build applications and systems with self-contained Computer Vision capabilities
https://www.genxr.co/#products
MIT License
8.48k stars 2.18k forks source link

distributed training #382

Open plenari opened 4 years ago

plenari commented 4 years ago

Does keras support distributed training? Can I use tensorflow's distributed training tools?

LeeHounshell commented 4 years ago

You could use something like the 'BatBot' architecture to implement distributed training. It captures images and associated training text for later batch submission. Then, once new models are generated, it is easy to move them into place.

https://github.com/LeeHounshell/BatBot

rola93 commented 4 years ago

Distribution over multiple GPUs is already provided for custom object detection. For custom object there is no chance to use any other tool apart from setting different number of GPUs.

you can also edit the source code to apply them anyway

plenari commented 4 years ago

I'm trying to distribution train imageai with horovod .

Edwin-Aguirre92 commented 2 years ago

Hello ,

I'm trying to train a Yolov3 architecture, with multiple GPUs, with no luck. However , with one GPU it works as observed below: image

I've tried using the tf.dsitribute.Strategy class as suggested by tensorflow (this can be observed here https://www.tensorflow.org/guide/distributed_training) , but have had no luck. Further , there seems to be a method in the DetectionModelTrainer class , which is named SetGpuUsage. However, if I type in either a string(with comma separated values or a int ) it does not use more than one GPU available. This can be observed below : image

Lastly, there is a multiple_gpu model function in the detection.custom.util, as observed here : image This function or method gets used in the _create model method inside the DetectionModelTrainer class, so I'm not sure if I should change the code in the SetGpuUsage method or on that multiple_gpu util function?

Any help would be greatly appreciated

Best Reagrds,

Edwin Aguirre @OlafenwaMoses @zacr0 @ @ZachNagengast @mludvig