Disable Variable Explorer auto-refresh feature by default

[spyder-bot]

From lemoine....@gmail.com on 2014-09-06T14:29:24Z

Spyder Version: 2.2.5 Python Version: 2.7.5 Qt Version : 4.8.5, PyQt4 (API v2) 4.10.3 on Darwin pyflakes >=0.5.0: 0.5.0 (OK) pep8 >=0.6 : 1.4.6 (OK) IPython >=0.13 : 1.1.0 (OK) rope >=0.9.2 : 0.9.4 (OK) sphinx >=0.6.6 : 1.1.3 (OK) matplotlib >=1.0: 1.3.0 (OK) sympy >=0.7.0 : 0.7.3 (OK) pylint >=0.25 : 1.0.0 (OK)

I'm running a model with nested for loops, something that looks like this (for simplicity, I'm not showing the full code as it's not really important, I don't think):

make a vector of mu

imm = range(101)

Make a a vector the desired simulations

nsim = range(10)

Number of generations

gen = 500

container to store the mu values

mu_bif = []

set up containers to store the output

Nbif = [] Pbif = [] Qbif = []

set initial conditions

Ni, Pi, Qi = 100., 10., 10.

RUN MODEL

start = time.time() for mu in imm: for sim in nsim: state = [Ni, Pi, Qi] for i in range(gen): state = Ptoid_Model(mu, state[0], state[1], state[2]) mu_bif.append(mu) Nbif.append(state[0]) Pbif.append(state[1]) Qbif.append(state[2])

end = time.time() print end - start

If I run this in Terminal, it only takes a minute or so. However, if I run this in Spyder's interactive interpreter, it can take anywhere between 10 - 20 minutes. If I increase nsim to range(100), Spyder can run for over 12 hours without finishing the task, whereas the task finishes in about 15 minutes.

Is there a reason the interactive interpreter in Spyder would be so much slower in for loops than running Python through the Terminal?

-Nate

Original issue: http://code.google.com/p/spyderlib/issues/detail?id=1958

[spyder-bot]

From contrebasse on 2014-09-07T14:50:28Z

Try to profile your code to see where exactly the computation is slow. You can use the profiler from spyder or line_profiler. I guess that the slow part is in Ptoid_Model.

[spyder-bot]

From lemoine....@gmail.com on 2014-09-08T09:54:38Z

I profiled the code and the Ptoid_Model part did take up the most time (~95% of elapsed time taken up by running the model). I used both cProfile in Terminal and the spyder built-in profiler. Both returned roughly the same amount of elapsed time to run the code (~50 seconds for a reduced number of iterations).

Ive tried running this code in the Spyder internal console, and it runs fine (~ 45 seconds). It also runs fine in the iPython console (~ 50 seconds). The same code in a regular" Python console/interpreter ran for about 15 - 20 minutes and didnt finish. So I guess the question is what is different about the Python interpreter in Spyder that it hangs on these kinds of problems, whereas both the internal console and iPython consoles work fine?

[spyder-bot]

From contrebasse on 2014-09-08T11:21:07Z

Ptoid_Model uses which library ? What kind of computation does it do ? Sorry for my questions, but I think we need more indices. I run for loops daily in spyder without any problem !

The best would be to have a runnable minimum example of your problem.

[spyder-bot]

From lemoine....@gmail.com on 2014-09-08T11:27:58Z

Sure! The Ptoid_Model is a recursive model that takes the previous time steps output and applies it to generate output at the current time step for 500 time steps (gen). Since some of the processes are stochastic, I do this for some number of simulations (nsim). Since Im interested in how the parameter mu affects model output, I repeat this simulations over a gradient of mu. The rest of the code (everything at the top) is simply setting up parameters to the right values, which are constant and never change in any run of the model.

import numpy as np import pylab as py import pandas as pd

define the model function

def Ptoid_Model(mu, N, P, Q): p, q = np.random.poisson(mu, 1), 40 N = round(N) P = round(P) Q = round(Q)

P_prob = (1 + aP * P / kP) ** -kP
Q_prob = (1 + aQ * Q / kQ) ** -kQ
p_prob = (1 + ap * p / kp) ** -kp
q_prob = (1 + aq * q / kq) ** -kq

Total = aP * P + aQ * Q + ap * p + aq * q

f = P_prob * Q_prob * p_prob * q_prob # negative binomial

Nt = lamb * N * np.exp(-delta * N) * f
Pt = wP * N * (1 - f) * (aP * P) / Total + wp * N * (1 - f) * (ap * p) / Total
Qt = wQ * N * (1 - f) * (aQ * Q) / Total + wq * N * (1 - f) * (aq * q) / Total
return [Nt, Pt, Qt, f]

def qConv(x): return 0.1884 * x

def pConv(x): return 0.1898 * x

lamb, delta = 20, 0.0015 wP, wp, wQ, wq = 1, 1, 1, 1 kP, kp, kQ, kq = 0.8, 0.8, 0.8, 0.8 P_ovi, p_ovi, Q_ovi, q_ovi = 4.05, 4.25, 4.25, 3.86 P_head, p_head = pConv(P_ovi), pConv(p_ovi) Q_head, q_head = qConv(Q_ovi), qConv(q_ovi)

def headToSearch(x): return float('-6.1e-4') + float('26.1e-4') * x

aP, aQ, ap, aq = headToSearch(P_head), headToSearch(Q_head), headToSearch(p_head), headToSearch(q_head)

make a vector of mu

imm = range(1, 401, 2)

Make a a vector the desired simulations

nsim = range(5)

Number of generations

gen = 500

container to store the mu values

mu_bif = []

set up containers to store the output

Nbif = [] Pbif = [] Qbif = []

set initial conditions

Ni, Pi, Qi = 100., 10., 10.

for mu in imm: for sim in nsim: state = [Ni, Pi, Qi] for i in range(gen): state = Ptoid_Model(mu, state[0], state[1], state[2]) mu_bif.append(mu) Nbif.append(state[0]) Pbif.append(state[1]) Qbif.append(state[2])

[spyder-bot]

From ccordoba12 on 2014-09-10T18:35:40Z

I think I found the problem. You need to go to

Tools > Preferences > Variable explorer

and deactivate the option called

[ ] Enable autofresh

This will avoid refreshing the Variable Explorer every 2 secs, which adds a lot of time to the simulation in the long run. The problem is you're creating very big lists, and building a view for them in such a short time is quite costly! :)

For things to run even faster, you also need to not keep visible the Variable Explorer while the simulation is running because Spyder also refreshes it every time the main windows loses focus (e.g. when you move from Spyder to Chrome/Firefox and then back to Spyder again). For that you can close it or just give focus to another pane that is placed next to it.

[spyder-bot]

From lemoine....@gmail.com on 2014-09-10T19:12:13Z

Yep that fixed it! Thanks!

[spyder-bot]

From contrebasse on 2014-09-14T12:25:13Z

Should we disable the refresh when the corresponding console is busy ? It's not really useful to view variable contents while the script is still running (or is it ?).

Maybe the correct thing to do is update only when a new prompt appears.

[spyder-bot]

From ccordoba12 on 2014-09-16T12:27:17Z

Yeah, I think we can disable it. But we shouldn't remove the option because it can be useful for educational purposes.

I think the variable explorer is refreshed automatically when the prompt comes back (at least that's what I've seen in our IPython consoles).

Summary: Disable Variable Explorer auto-refresh feature by default (was: Spyder hangs in for loop (or takes a long time))
Status: Accepted
Labels: -Usability Cat-VariableExplorer MS-v2.3.2

[spyder-bot]

From ccordoba12 on 2014-09-21T10:37:13Z

Labels: Easy

[spyder-bot]

From ccordoba12 on 2014-11-02T12:54:14Z

This issue was closed by revision cfced284987e .

Status: Fixed