search

mila-iqia / blocks

A Theano framework for building and training neural networks
Other
1.16k stars 351 forks source link

Postpost compilation in DatasetEvaluator #1185

Closed rizar closed 7 years ago

rizar commented 7 years ago

For debugging it's much better if the compilation in DatasetEvaluator happens when it's first called, not when it's created.

dmitriy-serdyuk commented 7 years ago

Hmm. As for me, it is better to crash sooner (if something is wrong with evaluation) rather than later.

rizar commented 7 years ago

You can then trigger DataStreamMonitoring on before_training. This way you will first do the monitoring and only then compile the training algorithm.

On Thu, 30 Mar 2017 at 11:12 dmitriy-serdyuk notifications@github.com wrote:

Hmm. As for me, it is better to crash sooner (if something is wrong with evaluation) rather than later.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/mila-udem/blocks/pull/1185#issuecomment-290442233, or mute the thread https://github.com/notifications/unsubscribe-auth/AAn8YrhA6Gi83gorGUOc0LOMKIw9NFb3ks5rq8ZngaJpZM4Mucvr .

dmitriy-serdyuk commented 7 years ago

Yes, but evaluation might be very slow. It is always frustrating when model compiles for several minutes, then evaluates for half an hour and then crashes with a first batch of training. And if you interrupt it during evaluation you'll have to wait again for compilation.

I would explicitly compile the evaluator on before_training callback.

rizar commented 7 years ago

Let's discuss this offline.

On Thu, 30 Mar 2017 at 11:24 dmitriy-serdyuk notifications@github.com wrote:

Yes, but evaluation might be very slow. It is always frustrating when model compiles for several minutes, then evaluates for half an hour and then crashes with a first batch of training. And if you interrupt it during evaluation you'll have to wait again for compilation.

I would explicitly compile the evaluator on before_training callback.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/mila-udem/blocks/pull/1185#issuecomment-290446089, or mute the thread https://github.com/notifications/unsubscribe-auth/AAn8YgGDofc4RnlinPR5vSan3GGNe86dks5rq8lAgaJpZM4Mucvr .

dmitriy-serdyuk commented 7 years ago

LGTM then