Control knobs for sending commands back to the running job / controlling live variables from the dashboard

[turian]

🚀 Feature

There should be a way to attach variables to the logger, that you can modify live from from controls in the dashboard.

Motivation

When you have a running job and are monitoring the progress, you sometimes want to adjust the learning rate or other hyperparameter (should we switch to fine-tuning mode, etc.).

Pitch

This is a bit of a unspoken black-magic deep learning technique. However, if you read papers from Meta, etc. or talk to hardcore old-school practitioners, they have these super long-running difficult optimization problems, and say something like: "Well we trained the generator for X thousand epochs, then we enabled the discriminator, then Y thousand epochs later we dropped the learning rate, etc." This is ideally done by monitoring a live, running job and modifying the variables in situ.

Alternatives

• The non-agile way to do this is let your run go for a while, decide afterwards that you should have changed something at some point in time, code that, run it again and cross your fingers. This is obviously pretty slow and requires luck.
• A hacky way to do this is to create a DSL with sentinel files that the running job reads and applies. However, the workflow is useful enough that there should be a common way to do this.

Additional context

I'm not aware of any logging library that does this. So it would make great blog posts to show off and attract more users.

[gorarakelyan]

@turian Thanks for the awesome suggestion! We totally got it on our plans. Right now we're laying down the foundation that will enable it to be implemented. I will share the progress here.