output in cached cells is delayed

[ihrke]

Hi,

when running a cell under the %%cache magic, the output is delayed until the evaluation has finished. I.e., in the following code, 1 shows up after 10 secs.

%%cache 'test.pkl' var
import time
for i in range(10):
    time.sleep(1)
    print i
var=2

I usually insert progress-bars or other on-the-fly output to long computations so that I'm able to estimate how long they are going to take. Is it possible to enable dynamic output in ipycache? I'm not sure what would be the best way to proceed since I'm not very familiar with ipython's internals.

[ihrke]

Sorry for the fuzz. Turns out, it IS an issue of ipycache, after all. The same problem was also caused by a different plugin in my setup which is why I thought that it was not caused by ipycache...

[rossant]

I'm not sure why this bug is happening actually... I don't see why the standard output would be captured. That requires some testing...

[ihrke]

Maybe you just have to add a sys.stdout.flush() someplace? Unfortunately, I don't know how ipython handles this, so I'm not a big help.

[ihrke]

I checked out ipycache.py and the problem seems to be because you capture the cells' output using the capture_output() handler:

with capture_output() as io:
   try:
      ip_run_cell(cell)
   except:
      # Display input/output.
      io()
      return
# code, code
io()

which is used to inject the captured output in case the cell is not evaluated.

So I guess, to make the output show up in real-time, the capture-output handler would have to be modified?

[ihrke]

I experimented a bit with the capture_output() handler and found a solution that resolves the "bug" (see ihrke/ipycache@8d05a45b620ead800a6acf9379b8051213d4e545). The branch is here: https://github.com/ihrke/ipycache/tree/capture_and_print However, the solution is quite ugly as it requires redefining ipython's capture_output() handler to use a custom derivation of StringIO which writes out its input and stores it internally.

Do you have a better idea how that could be handled?

[rossant]

I'm wondering if it would be possible to just not use IPython's code but an entirely custom context manager? something like http://stackoverflow.com/questions/16571150/how-to-capture-stdout-output-from-a-python-function-call

I initially used whatever IPython did because that was just quicker, but if it's too limited we can use something else.

[ihrke]

Actually, that's kind of what I did (by copying and adapting the code from ipython). I thought that you used ipython's capture handler for some deeper reason but I guess we are alright with my solution, then. We need all the ipython features anyway because we need to handle output graphics etc. Is it ok if I merge my code into the main repo?

[rossant]

If that works sure, we can always refine later if needed. Thanks.

[ktran9891]

This is still not working for me for some reason. Any suggestions?