Keyboard interrupt catch

[johnpaulbin]

Sometimes, things don't go your way, and your already deep down the rabbit hole. Maybe you've spent 2 hours training a model, and you just realized now you want to change some config, but you have set your epoch checkpoint interval to be waay too much. Would it be possible to catch a keyboard interrupt while training, to do a graceful shutdown (save checkpoint n such?)

[sjkoelle]

Not quite sure how to do this. Any suggestions?

[johnpaulbin]

Here: https://github.com/uberduck-ai/uberduck-ml-dev/blob/75d47f0a95bf7bc4cd3babc558d3c4df086c8de0/uberduck_ml_dev/exec/train_tacotron2.py#L45

Wrapping the if else statement with a try catch, making sure exception is KeyboardInterrupt

    try:
        main()
    except KeyboardInterrupt:
        print("Graceful shutdown, saving current checkpoint")

[sjkoelle]

ahh makes sense. want to put in a PR?

[johnpaulbin]

fixed in latest pr