Open marcoleewow opened 7 years ago
If the machines have GPU's, it should automatically allocate the GPU's during training. Let me know if it doesn't work. Personally I wouldn't use DOWNPOUR due to "implicit momentum" as the number of asynchronous processes is too high, but rather AGN or ADAG, see https://github.com/JoeriHermans/master-thesis/blob/master/thesis/master_thesis_joeri_hermans.pdf for more information.
Joeri
If the community is interested, I can make a Spark-less script which does the thing you want. Should be rather trivial to implement.
I don't have problems with running multiple models in each of my GPUs, and I can also use synchronous SGD using tensorflow (see here), but so far there is no asynchronous SGD code available for multi GPU system. It would be amazing if you can implement the codes :) I will be more than happy to help you out! Note that I am still a newbie though...
I wouldn't use synchronous SGD since it implicitly increases the size of a mini-batch :) If you want we can work together (I don't have multiple GPU's to test atm), drop me an e-mail: joeri@joerihermans.com. Then we can make a Gist or something for everyone.
@JoeriHermans Hi, Hermans. I will be very interested if you could make a Multi-GPUs without Spark environment!
I am working on the same thing from few weeks. Methods I currently found to use model to train on Multiple GPUs on a single instance:
While using all these libraries dont forget to use TENSORFLOW-GPU version not regular TENSORFLOW with CUDNN libraries. CUDNN is must to solve the training time. It reduces so well.
Let me know if I am wrong understanding all these. But i have used MXNET, ELEPHAS for now. Please also let me know if you do find some other libraries. I am willing to contribute and help you guys to the best I can. Looking forward to hear your opinion on this.
@imranshaikmuma Is it possible to use ELEPHAS for distributed training in my local machine? My local machine has 8 cores and a 4GB NVIDIA GPU. I would like to do the training in multiple workers for test purpose. Could you please tell me if it possible? I am trying it but it is giving me exception.
Hi, are there any plans for multi-GPUs on a single machine support? I would like to use downpour (asynchronous) SGD in my training, which is not supported in Keras.