Learning Rate finder

[torzdf]

Adds a learning rate finder for training in Faceswap.

Inspired by: https://fastai1.fast.ai/callbacks.lr_finder.html

The learning rate finder is enabled by the command line option: -r or --use-lr-finder

Configuration options are available:

• lr_finder_iterations - Set the number of iterations for discovering optimal learning rate (default: 1000)
• lr_finder_mode - Controls the mode. Either:
  • set the optimal learning rate
  • provide a graph of discovered optimal learning rate and set the optimal learning rate
  • provide a graph of discovered optimal learning rate and exit
    • lr_finder_strength - How aggressively to set the learning rate.

Example graph output: