Training Enablement Tools Telegram Interface

[ghost]

This issue should be used to discuss the commands in Telegram.

List of planned commands:

• /list - Returns a time-sorted list of training processes e.g:

  Name  |  Status  |  Start Date  |  Stop Date
  resnet-base-replacex03  |  running  |  17th Oct 1:43 pm  |  N/A
  resnet-base-replacecrap  |  stopped  |  17th Oct 1:43 pm  | 17th Oct 1:43 pm
  resnet-base-replacecr1ap  |  crashed  |  17th Oct 1:43 pm  | 17th Oct 1:43 pm

• /output <model-id> - Returns the log output of the model specified with the model name<model-id>
• /shutdown <model-id> - Initiates the shutdown of the given training process

Update

• /nvidia-smi <model-id>- Get the output of nvidia-smi for given training

Furthermore the Telegram Bot will automatically send notifications on following events:

• Training process has crashed
• Training process has finished successfully
• Training process was shutdown successfully after shutdown request

@Simsso @FlorianPfisterer Any other desires from your deepest dreams?

[Simsso]

Dreaming of /nvidia-smi to get the memory consumption and GPU util.

[FlorianPfisterer]

That's funny, I've once had the same dream!

[ghost]

Initial /list-command:

There is still a long way to go!

[Simsso]

A link to TensorBoard, where only the given run is shown would be helpful. Since TensorBoard is not really mobile-compatible, it's always a pain to change the checkbox selection on the left-hand side.

[ghost]

Closing this one. Nearly all requested features are implemented.

Beside @Simsso request to generate links for the given training run 👎. Sorry for that!