search

mila-iqia / blocks-extras

A collection of extensions to the Blocks framework
MIT License
27 stars 40 forks source link

IPython extension #17

Closed janchorowski closed 9 years ago

janchorowski commented 9 years ago

Upon receiving the hup signal gets you an IPython shell that can load the environment of the function calling main-loop. Perfect to change a few hyperparameters by hand, or just to force a checkpoint.

rizar commented 9 years ago

Very nice! Since this is blocks-extras, I am happy to merge it, if you wish, you can add some docs later.

nouiz commented 9 years ago

I have a report that Theano don't allow to kill execution with ctrl-C in ipython. Do you know about this?

https://github.com/Theano/Theano/issues/3064

It is dangerous to interrupt a Theano function call. I'm not sure this always go well. If a function have many updates, it could happend that some of them get applyed, but not all.

Just in case you have some information on that.

janchorowski commented 9 years ago

The code does two things: install a tiny signal handler which just sets a flag and the IPython runner which is run between batches, when it should be safe.

I don't know if the signal handler can break theano calls - it should exit very fast.

nouiz commented 9 years ago

Ok, I think what you do is safe at a high level. I think the problem with Theano is that it is a C function and that isn't interuptible by default. At least, not in the way we do it. If you know how to do this, it could be useful.

On Thu, Aug 6, 2015 at 6:58 PM, Jan Chorowski notifications@github.com wrote:

The code does two things: install a tiny signal handler which just sets a flag and the IPython runner which is run between batches, when it should be safe.

I don't know if the signal handler can break theano calls - it should exit very fast.

— Reply to this email directly or view it on GitHub https://github.com/mila-udem/blocks-extras/pull/17#issuecomment-128534788 .