search

kundajelab / dragonn

A toolkit to learn how to model and interpret regulatory sequence data using deep learning.
http://kundajelab.github.io/dragonn/
MIT License
254 stars 71 forks source link

DragoNN with multiple GPUs #27

Closed satroz closed 7 years ago

satroz commented 7 years ago

Hello,

Great framework ! I was wondering if it's possible make DragoNN work with multiple GPUs? When I try to run the example script with the theano flags Iike this: THEANO_FLAGS='contexts=dev0->cuda0;dev1->cuda1,floatX=float32' python simple_motif_detection.py

I get the following error (repeated multiple times):

ERROR (theano.gof.opt): Optimization failure due to: LocalOptGroup(local_abstractconv_cudnn,local_abstractconv_gw_cudnn,local_abstractconv_gi_cudnn,local_abstractconv_gemm,local_abstractconv3d_gemm,local_abstractconv_gradweights_gemm,local_abstractconv3d_gradweights_gemm,local_abstractconv_gradinputs_gemm,local_abstractconv3d_gradinputs_gemm) ERROR (theano.gof.opt): node: AbstractConv2d{convdim=2, border_mode='valid', subsample=(1, 1), filter_flip=True, imshp=(None, 1, 4, 500), kshp=(45, 1, 4, 10), filter_dilation=(1, 1)}(Assert{msg='AbstractConv shape mismatch: shape of image does not match given imshp.'}.0, Assert{msg='AbstractConv shape mismatch: shape of filters does not match given kshp.'}.0) ERROR (theano.gof.opt): TRACEBACK: ERROR (theano.gof.opt): Traceback (most recent call last): File "/isdata/nalcaraz/Programs/anaconda2/lib/python2.7/site-packages/Theano-0.9.0b1-py2.7.egg/theano/gof/opt.py", line 1964, in process_node replacements = lopt.transform(node) File "/isdata/nalcaraz/Programs/anaconda2/lib/python2.7/site-packages/Theano-0.9.0b1-py2.7.egg/theano/gof/opt.py", line 1316, in transform new_repl = opt.transform(node) File "/isdata/nalcaraz/Programs/anaconda2/lib/python2.7/site-packages/Theano-0.9.0b1-py2.7.egg/theano/gpuarray/dnn.py", line 2641, in local_abstractconv_cudnn ctx = infer_context_name(*node.inputs) File "/isdata/nalcaraz/Programs/anaconda2/lib/python2.7/site-packages/Theano-0.9.0b1-py2.7.egg/theano/gpuarray/basic_ops.py", line 122, in infer_context_name raise ValueError("Could not infer context from inputs") ValueError: Could not infer context from inputs

It works fine when I use just use one GPU.
I'm using dragonn version 0.1.3 and Theano 0.9.0beta1.

jisraeli commented 7 years ago

@satroz DragoNN was designed mainly as a learning toolkit so there is no support for multiple GPUs. We are developing a production-level version of DragoNN for large-scale modeling where it would be more convenient to use multiple GPUs, I expect a release in a few months.