photocyte
commented
5 years ago INFO (theano.gof.compilelock): Waiting for existing lock by process '20565' (I am process '10149')
INFO (theano.gof.compilelock): To manually release the lock, delete /home/tfallon/.theano/compiledir_Linux-4.15--generic-x86_64-with-debian-buster-sid-x86_64-3.6.8-64/lock_dir
INFO (theano.gof.compilelock): Waiting for existing lock by process '3683' (I am process '10149')
INFO (theano.gof.compilelock): To manually release the lock, delete /home/tfallon/.theano/compiledir_Linux-4.15--generic-x86_64-with-debian-buster-sid-x86_64-3.6.8-64/lock_dir
INFO (theano.gof.compilelock): Waiting for existing lock by process '6431' (I am process '10149')
INFO (theano.gof.compilelock): To manually release the lock, delete /home/tfallon/.theano/compiledir_Linux-4.15--generic-x86_64-with-debian-buster-sid-x86_64-3.6.8-64/lock_dir
INFO (theano.gof.compilelock): Waiting for existing lock by process '12591' (I am process '10149')
INFO (theano.gof.compilelock): To manually release the lock, delete /home/tfallon/.theano/compiledir_Linux-4.15--generic-x86_64-with-debian-buster-sid-x86_64-3.6.8-64/lock_dir
INFO (theano.gof.compilelock): Waiting for existing lock by process '12591' (I am process '10149')
INFO (theano.gof.compilelock): To manually release the lock, delete /home/tfallon/.theano/compiledir_Linux-4.15--generic-x86_64-with-debian-buster-sid-x86_64-3.6.8-64/lock_dir
INFO (theano.gof.compilelock): Waiting for existing lock by process '15766' (I am process '10149')
INFO (theano.gof.compilelock): To manually release the lock, delete /home/tfallon/.theano/compiledir_Linux-4.15--generic-x86_64-with-debian-buster-sid-x86_64-3.6.8-64/lock_dirJust noticed this. That's not good. Seems there is a lock on a cluster shared filesystem, so everything starts locking each other.
Solution seems to be here. Unclear if you can give theano that parameter without having to edit that file https://github.com/Theano/Theano/issues/3723
tomas-pluskal
By my estimate deeploc is 10-50 times slower than signalp or tmhmm. So, that is why in the flowgraph it splits into smaller chunks for deeploc versus signalp/tmhmm.
Even still, the test run on a real dataset is taking much longer than I would expect.
It is possible (since deeploc is a deep learning based thing), that the speed of execution might be quite particular on libraries and settings, and so could be going much slower in my test execution than in a 'properly configured' environment.
It is also possible that certain sequences (e.g. with X amino acids), or certain sequence compositions run extra slowly. Seems unlikely since the deep learning neural network model should just be a straightforward bit of math on the sequence.