update azureml mlops demo notebook to include tensorboard

[stevehadd]

Currently we have a demo of using tensorboard, but only only a local installation. There is now support for using tensor board with azureml, as described on this page: https://docs.microsoft.com/en-us/azure/machine-learning/how-to-monitor-tensorboard

This requires adding azureml-tensorboard to the environment. We should do this as part of our standard MLOps toolset

[stevehadd]

Todo:

• update notebooks
  • prd_mlflow_on_azure_demo.ipynb
  • prd_mlops_azml_demo.ipynb,
  • prd_mlops_azml_cluster_demo.ipynb
• Update conda environments to include new packages and run with code as it currently exists in branch #
• Separate out azure ml and non azure ml environments
• Ensure notebooks and cluster scripts run with new datasets
• Add tensorboard callback to code for fitting models in notebooks and scripts
• Add tensorboard initiation code to notebooks, test that dashboard can be viewed
• Add mlflow experiment tracking to cluster scripts
• Add run monitoring widgets to notebooks

[stevehadd]

it seems like we may have to set the log directory for tensorboard to work, and maybe also add more tensorflow logging stuff.

[stevehadd]

having tried to get tensorboard running several times with my own notebooks and environments as well as the sample code, I'm coming to the conclusion that one of the following is true:

• it doesn't work currently
  • Given that this is true for the sample code, this seems at least partially likely
• the sample is missing some crucial detail or I have misunderstood a crucial detail
  • there are often obscure config details with AzML, but again since taking the sample code and running it as is doesn't work, its hard to see how this could be true
• the way we're logging things in tensorflow (with a tensorboard callback) isn't working with tensorboard on azureml
  • its working fine for tensorflow on SPICE or on a mac, so should be fine as far as I can see on AzML as well

I'm waiting to hear from Steve Gibson on a contact from Microsoft, I think this is something that I need to check with whoever that is.