Run hyperparameter tuning on AzML using HyperDriveConfig

[stevehadd]

Once #9 is complete, we will have a pipeline for running experiments on AzureML. The next stage is to make use of the AzML hyperparameter tuning functionality i the HyperDriveConfig class: https://docs.microsoft.com/en-gb/python/api/azureml-train-core/azureml.train.hyperdrive.hyperdriveconfig?view=azure-ml-py

This can make good use of the AzML compute cluster to run jobs in parallel.

This should happen after the planned refactoring as the code is getting a bit messy.

Some addtional refactoring of the current AzML implementation that might be required:

• thinking about how to specify the parameters to the entry point script for the run. current the hyperparameters are specified in the JSON config. A good idea might to make the hyperparameter just a list of different config files to use. These could be generated by a template. Alternatively, we can use the built in mechanism of passing the values as arguments to the script through the CLI args, read in the JSON file, and overwrite the default values in the JSON file with those from the CLI.

[stevehadd]

I've been investigating this today. The way I have been passing parameters into the pipeline (through experiment definition JSON file) doesn't fit very well with the AzML hyper parameter tuning API (where one passes arguments through the command line. A naive implementation would require adding the relevant hyperparameters as command line options, but the hyperparameters vary between different models, so it would break flexibility of the code to switch between different algorithms based on the settings in the JSON file. To maintain this flexibility, I intend to make the following changes

• create a new AzML Experiment class for hyperparameter tuning experiments
• in this class, use the process_known_arguments functions to process the current CLI args, then manually process the remaining arguments, which will be assumed to be hyperparameter related
  • https://docs.python.org/3/library/argparse.html#argparse.ArgumentParser.parse_known_args
• the derived class should override read JSON function. It should call the parent class function and then afterwards overwrite any hyper parameters defined on the command line
• we should make sure we register the current hyperparameters with the Run class
• create another AzML class derived from Experiment called AzuremlLauncher,
  • this class should handle the AzML setup stuff that happens IN the notebook, to reduce boilerplate.
  • this include reading the JSON file. It should be responsible for creating a parameter sampling object to be used with the HyperDriveConfig object. https://docs.microsoft.com/en-gb/python/api/azureml-train-core/azureml.train.hyperdrive.hyperparametersampling?view=azure-ml-py
  • the run_experiment should do the launching of the experiment.