AzureML and ML Flow demo

[stevehadd]

I've created some extra notebooks to demo more advanced use of the Azure ML and ML Flow tools. The following changes / developments are included in this PR:

• updated requirements file for ML
• demo of running pipeline on a cluster. This consists of a 2 notebooks (one a CPU cluster, the other GPU), a file containing the training code, and a separate entry script.
  • the entry script is separate so the functional code can be called from the notebook. the whole if __name__ == 'main' doesn't seem to work well
• demo using ML flow with a ML flow tracking server. This includes instruction on running the tracking server and tensor board server.
• demo of using mlflow with an azureml backend. All this requires is setting the tracking URI to a URI supplied by the AzureML workspace, in the same way in the demo of a local tracking server mentioned above, we pointed to a localhost URI.

Closes #7 Closes #8 Closes #28

[review-notebook-app[bot]]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[hannahbrown7]

Clustering demos

• Just a note on something we have already discussed, need to be careful with tensorflow versions otherwise run into an error when reloading the trained model (version 2.8.0 definitely works).
• I found that it can sometimes take a bit of time before the model is ready to load for evaluation and if I was being impatient got errors that file or directory didn't exist when running evaluation code. May be worth just adding a comment to say wait until you can see the model/run output in experiments before trying to load.

Mlflow on azure demo

• This is fab!
• May be clearer to remove the run_name='run1' when starting the experiment as then each run name is called run1 in experiment dashboard and is difficult to tell them apart.
• The plotting cell below the tensorflow callback seems to be in the wrong place and doesn't work where it in when notebook is run in sequence. Need to be removed or moved down.

prd_pipeline.py / Data on Azure

• The data on Azure seems to have a lot more NaN values then previous dataset and so applying df.dropna in the prd_pipeline.py reduces the dataset from 211680 samples to 62388 samples, whereas previously dropna was only dropping ~540 samples. There nan values seem to be coming from 'forecast_reference_time_x' (41400 NaN values) and 'time_bnds' (149130 NaN values). As a temporary workaround, using dropna(subset='rainfall_rate_composite') goes back to previous sample sizes.

[hannahbrown7]

• I found that it can sometimes take a bit of time before the model is ready to load for evaluation and if I was being impatient got errors that file or directory didn't exist when running evaluation code. May be worth just adding a comment to say wait until you can see the model/run output in experiments before trying to load.

While having a play with Hyperdrive, I came across a useful function prd_run.wait_for_completion(), which holds the notebook until the scripts running on the cluster have completed then continues to run the subsequent cells in the notebook as expected