mcsolve hangs when script is imported as a module

[GoogleCodeExporter]

What steps will reproduce the problem?

This is something that sometimes happens to me, and sometimes not, while 
running the same code, so it is hard to give an exact recipe that reproduces 
it. It has happened on a couple of different platforms, and with several 
different codes. Here is an example code for which I will often experience 
mcsolve hanging:

import numpy as np
from qutip import *

gamma = 1
neq = 2
psi0 = basis(neq,0)
H = sigmax()
c_ops = [sqrt(gamma)*sigmax()]

# Times
T = 10.0
dt = 0.1
nstep = int(T/dt)
tlist = np.linspace(0,T,nstep)

ntraj=10

# set options
opts = Odeoptions()
opts.num_cpus=1 # in case problem has to do with parallelization
opts.gui=False # in case problem is gui

sol_mc = mcsolve(H,psi0,tlist,c_ops,[],ntraj=ntraj,options=opts)

What is the expected output? What do you see instead?

The process simply hangs, and I need to kill the python instance.

What version of the product are you using? On what operating system?

I've had this problem on qutip 2.0.0 and 2.1.0. My python version is 2.73. I've 
had the problem on both ubuntu and os x mountain lion.

Please provide any additional information below.

So, I've been able to pinpoint where the solver hangs. In the MC_class class in 
mcsolve.py, the funciton run() makes a call to the function parallell(). This 
is on line 381 (assuming no gui) on my verison of mcsolve.py 
(QuTiP==2.1.0.dev25872.1.0.dev2587). The code looks like:

if not odeconfig.options.gui:
  self.parallel(args,self)
else:

And this is where it hangs. That is, it never gets past the call to parallel. I 
tried rewriting a "serial" version of parallel, that don't make use of the 
multiprocessing module at all, to see if this was the problem, and indeed this 
worked.

Could the problem be zvode? zvode does not allow you to have several 
scipy.integrated.ode instances that call zvode at the same time. From the scipy 
documentation:

"Warning: This integrator is not re-entrant. You cannot have two ode instances 
using the “zvode” integrator at the same time."

See: 
http://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.ode.html

Original issue reported on code.google.com by arne.gri...@gmail.com on 25 Sep 2012 at 6:00

[GoogleCodeExporter]

Arne,

Try getting the source version and giving it a try.  I just tested it on my 
Linux box at work and I had no problems.  In fact we have never had any 
problems using zvode.  So far, every example has worked.  I am running it on a 
12-processor machine without error.  It worked fine as well on my macbook air 
running 10.8.2

If you still have issues after checking out the latest source let us know.  

Paul

Original comment by nonhermitian on 25 Sep 2012 at 6:08

[GoogleCodeExporter]

Hi Paul,

So I got the code with svn checkout http://qutip.googlecode.com/svn/ 
qutip-read-only

I tried running qutip.test(), and I got no errors. I then also tried running 
through all the Monte Carlo examples via qutip.demos(), and they worked. But my 
program still crashes sometimes--and sometimes not.

Right now, I ran it with gui, and it worked but spat out some exceptions:

Exception AttributeError: AttributeError("'_DummyThread' object has no 
attribute '_Thread__block'",) in <module 'threading' from 
'/usr/lib/python2.7/threading.pyc'> ignored
Exception AttributeError: AttributeError("'_DummyThread' object has no 
attribute '_Thread__block'",) in <module 'threading' from 
'/usr/lib/python2.7/threading.pyc'> ignored
Exception AttributeError: AttributeError("'_DummyThread' object has no 
attribute '_Thread__block'",) in <module 'threading' from 
'/usr/lib/python2.7/threading.pyc'> ignored
Exception AttributeError: AttributeError("'_DummyThread' object has no 
attribute '_Thread__block'",) in <module 'threading' from 
'/usr/lib/python2.7/threading.pyc'> ignored

Don't know if this tells you something...

I've had this problem popping up whenever I've used the mc solver since I 
started using qutip a few months ago, so I know it isn't just something I've 
recently changed that is causing it. I'll try it again on my mac later tonight.

Arne

Original comment by arne.gri...@gmail.com on 25 Sep 2012 at 7:19

[GoogleCodeExporter]

Arne,

I have run your code 100 times in a for loop and did not run into any issues.  
As for the threading stuff, this is an issue with Python, not our code:

http://bugs.python.org/issue14308

Maybe try running the code with an expectation value, say sigmax(), and see 
what you get.  Perhaps it has to do with recent changes we made to the state 
vector code?

Paul

Original comment by nonhermitian on 25 Sep 2012 at 7:55

[GoogleCodeExporter]

I tried just running with an expectation value now without sucess. I also had 
this problem with qutip 2.0.0, so it probably isnt't due to any recent change 
in the code.

It is starting to seem like a problem with my system, and not the qutip code... 
It's just so weird I have the same problem on two different computers. All I 
know is that it is related to the parallelization, as the program never crash 
if I run mc_alg_evolve in serial, without the use of Pool etc. Other than that 
I'm at a loss.

I think I will start a thread on the discussion forum. Maybe someone else have 
run into this.

By the way, what version of python are you using?

Cheers,

Arne

Original comment by arne.gri...@gmail.com on 25 Sep 2012 at 11:15

[GoogleCodeExporter]

Arne,

I am running 2.7.3 on my mac, and I believe it is the same on Ubuntu 12.04.  
Just to be specific, I am using MacPorts on the mac and not the built in 
Python.  A discussion is probably the best idea. as I can not reproduce what 
you are seeing at all.  As far as I know, Robert never mentioned any problems 
as well.  If some more people speak up, then at least we can see whether it is 
indeed a qutip problem, or maybe there is some funny business going on at your 
end.  But like you said, it is strange to have the same issue on two computers.

Regards,

Paul

Original comment by nonhermitian on 25 Sep 2012 at 12:41

[GoogleCodeExporter]

Hi

I've also tried to your test program but have not been able to reproduce the 
problem, and I've never seen any similar problems with other programs. I've 
tried on various versions of ubuntu, with python 2.7.2, 2.7.3, 3.2.3. Perhaps 
it would help if you give the versions on all dependencies as reported by the 
about() function. Then perhaps we can reproduce the your environment and the 
problem.

Regarding scipy.integrated.ode and that it is non-reentrant, note that in qutip 
we are using the multiprocessing package which forks new processes (as opposed 
to using threads), so the ode solver will never be called multiple times from 
within the same process (by different threads), so that cannot be the problem.

Rob

Original comment by jrjohans...@gmail.com on 26 Sep 2012 at 12:56

[GoogleCodeExporter]

I've tried various environments now. I just tried to set up a minimalistic 
virutalenv. The about() function gives:

>>> about()

QuTiP: The Quantum Toolbox in Python
Copyright (c) 2011-2012
Paul D. Nation & Robert J. Johansson

QuTiP Version:       2.1.0.dev2642
Numpy Version:       1.6.2
Scipy Version:       0.10.1
Matplotlib Version:  1.2.0rc2

PySide Version:      None
PyQt4 Version:       None

As far as python packages go, pip freeze gives:

QuTiP==2.1.0.dev26422.1.0.dev2642
argparse==1.2.1
ipython==0.13
matplotlib==1.2.0rc2
nose==1.2.1
numpy==1.6.2
scipy==0.10.1
wsgiref==0.1.2

Another environment where I get the same problem has the following about() 
output:

QuTiP Version:       2.0.0
Numpy Version:       1.6.1
Scipy Version:       0.9.0
Matplotlib Version:  1.2.0rc2

PySide Version:      1.1.0
PyQt4 Version:       4.9.1

A third one:

QuTiP Version:       2.1.0.dev2642
Numpy Version:       1.6.1
Scipy Version:       0.9.0
Matplotlib Version:  1.2.0rc1

PySide Version:      1.1.0
PyQt4 Version:       4.9.1

All these are python 2.7.3, but I also just tried it on 3.2.3.

Original comment by arne.gri...@gmail.com on 26 Sep 2012 at 1:10

[GoogleCodeExporter]

OK, so I've figured out the issue finally. I was pulling my hair trying to 
understand why the qutip examples were working, but not my simple script. Then 
it occured to me that it might be a problem when importing the module. So I 
wrapped everything in a function, and it no longer hanged. Similarly, it will 
hang if you fire up python and import the script (what I was doing), or 
equivalently

python -c "import test"

but doing

python test.py

works fine. I bet what you guys have been testing was the latter? Would you 
mind trying to import the script instead and see if it hangs?

What I suspect to be the problem is that if you import it as a module, and do 
not wrap the call to mcsolve in a function, Pool will try to recursively import 
the module forever.. or something. Actually, there is a warning about this in 
the multiprocessing documentation, but weirdly the warning is issued for 
Windows platforms. See "Safe importing of main module" here:

http://docs.python.org/library/multiprocessing.html#windows

I've had a friend test this too, he is using linux (ubuntu I think), so we have 
at least two linux platforms and one mac os x where this has crashed now.

Arne

Original comment by arne.gri...@gmail.com on 27 Sep 2012 at 12:03

[GoogleCodeExporter]

I can confirm that using python -c "import test" does hang.  The solution to 
this may take some time though as a fix is not as easy as using 
__name__=='__main__', at least as far as I know.  We have never tired to import 
a test script as a module, so I am curious as to why one would want to do this?

Paul

Original comment by nonhermitian on 27 Sep 2012 at 12:47

• Changed title: mcsolve hangs when script is imported as a module
• Changed state: Accepted

[GoogleCodeExporter]

When testing stuff I like using python in interactive mode so I can look at the 
data. After having imported the above as say "test" i can do print test.tlist, 
or plot(test.tlist,test.sol.expect[0]) or whatever. I thought that was a pretty 
standard way of working.

Original comment by arne.gri...@gmail.com on 27 Sep 2012 at 1:11

[GoogleCodeExporter]

You can safely import your example scripts, like ex_32.py etc., because they 
wrap everything in a run() function which avoids the above problem.

Original comment by arne.gri...@gmail.com on 27 Sep 2012 at 1:13

[GoogleCodeExporter]

The mcsolver has quite a bit of extra stuff going on compared to the example 
scripts.  Making sure everything works the way it is supposed to may be a 
challenge.  Perhaps wrapping the entire MC_class may be a solution?

Original comment by nonhermitian on 27 Sep 2012 at 1:17

[GoogleCodeExporter]

Yeah, I don't know how this could be written in a fool proof way. It is maybe 
more of an issue with the multiprocessing module and the way I wrote my script 
than the qutip code? Perhaps just a warning / style guide-line in the docu 
about how to write scripts? For example, my script above should be written

def run():
# put code here

if __name___='__main__':
  run()

and it would have been safe both with "python test.py" and "import test; 
test.run()" in interactive mode.

Original comment by arne.gri...@gmail.com on 27 Sep 2012 at 1:47

[GoogleCodeExporter]

Hi 

Thanks for finding and tracking down this problem! I also don't really know why 
the programs is that it hangs when being imported, so I guess we have some 
debugging to do.

By the way, if you want to run a script interactively and access its variable 
afterwards I'd recommend that you use ipython and its %run command, with which 
you can do 

>>> %run test

and then have access to all the variables in the global address space. 

Original comment by jrjohans...@gmail.com on 27 Sep 2012 at 8:40

[GoogleCodeExporter]

Hi, 

I was wondering if there are any news about this issue. I have the same 
problem, I need to run a loop in which I use mcsolve, and my program will crash 
after few iterations, giving me the error:

Exception AttributeError: AttributeError("'_DummyThread' object has no 
attribute '_Thread__block'",) in <module 'threading' from 
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threa
ding.pyc'> ignored

repetitively, until it crashes, and I need to terminate it.

I have tried to run one of the qutip example ex_31.py (about Coupled Driven 
Oscillators) simply modified by copying an extra line: 

data = mcsolve(H, psi0, tlist, c_op_list, [na, nb],ntraj=5)
data = mcsolve(H, psi0, tlist, c_op_list, [na, nb],ntraj=5)

below the first mcsolve command, so to execute the mcsolve twice, and I do get 
the same error:

Exception AttributeError: AttributeError("'_DummyThread' object has no 
attribute '_Thread__block'",) in <module 'threading' from 
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threa
ding.pyc'> ignored
Exception AttributeError: AttributeError("'_DummyThread' object has no 
attribute '_Thread__block'",) in <module 'threading' from 
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threa
ding.pyc'> ignored

the program ends as wanted producing the figure, but I guess it would crash if 
I would repeat it several times and involving more memory as I do in my code 
with the mcsolve loop.

I get the same error either using python or ipython  (python ex_31.py, ipython 
ex_31.py) or if I import the module in python:
>>import ex_31
>>ex_31.run()

I have a Mac OS X 10.7.5 and I have installed qutip with macports, my qutip 
info are:

QuTiP Version:       2.2.0

Numpy Version:       1.8.0.dev-436a28f
Scipy Version:       0.10.1
Matplotlib Version:  1.2.x

PySide Version:      1.1.2
PyQt4 Version:       4.9.6
PyObjc Installed:    Yes

Thank you.

Best,
Claudia

Original comment by claudia....@gmail.com on 11 Apr 2013 at 3:12