Ability to disable/limit training messages

[arpieb]

While training messages are helpful, in some cases it would be nice to be able to either limit the verbosity or disable the output completely.

Use cases:

• When running in a Docker container using an external IDE like VSCode, the stream of per-batch messages bogs down training run output, in turn bogging down the runtime of the app (e.g. MNIST takes ~2s/epoch running directly within a Docker container, ~20-30s/epoch running as a remote container against GPUs). In this case per-epoch data would be great, per-batch not so much
• When running something like an RL model where you're more interested in logging the completion status, state and rewards of an agent per traversal than the incremental loss/accuracy/validation, it's often desirable to disable logging completely.

Maybe being able to set reporting level like :per_batch, ':per_epoch or :none?

[seanmor5]

Agreed completely. I would like to implement that IO completely as a callback, so I think configuration would fall there.

[seanmor5]

As far as the configuration I would probably do something like: log_every: with the option of a number for logging every n steps and then maybe accept :epoch and :none. WDYT?

[batate]

Maybe a log_every option as scripted, and a generic callback fn accepting an epoch and the metadata for the training step?

-bt

On Fri, Apr 9, 2021 at 5:39 PM Sean Moriarity @.***> wrote:

As far as the configuration I would probably do something like: log_every: with the option of a number for logging every n steps and then maybe accept :epoch and :none. WDYT?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/elixir-nx/axon/issues/49#issuecomment-816986116, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABFDOKSDNDZS3DE4OGQNUDTH5XYTANCNFSM42VVWPHQ .

--

Regards, Bruce Tate CEO

https://bowtie.mailbutler.io/tracking/hit/f8218219-d2a8-4de4-9fef-1cdde6e723f6/c7c97460-016e-45fb-a4ab-0a70318c7b97

Groxio, LLC. 512.799.9366 @.*** grox.io

[arpieb]

I think there is value in a reporting frequency setting for users fine with the default reporting mechanism, but do agree with @batate that a callback option would also have value, especially if we would like to ultimately open logging up to systems like TensorBoard or other logging systems in the general case.

[seanmor5]

I've added the ability to limit messages in f2981ca0518dbdacc44c71d71a6851820b139019 with the :log_every option. You can see a usage example in the MNIST example.

I'm going to mark this closed because callbacks will be addressed in: https://github.com/elixir-nx/axon/issues/21