Hannah/data exploration

[hannahbrown7]

This pull request includes code for data exploration and setting up an initial model pipeline in order to explore tools such as ML flow

Files:

• data_exploration_Storm_Dennis.ipynb - overview of data and exploratory data visualisation.
• precip_dediagnosis_model.ipynb - initial model pipeline which is able to take either only module profile features, or a combination of model profile features and single level features, with some simple evaluation of the trained model at the end.

Future work:

• Link the model up to MLOps tools such as MLflow or AzureML experiments
• Refine the model, including hyperparameter tuning (currently set to initial hyperparameters)
• Expand evaluation to include more explainability techniques.

[review-notebook-app[bot]]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[stevehadd]

That looks very good. Some thoughts

• As we've discussed before, it will be very interesting to see how much of what you've foundin the data exploration is specific to the data from this event compared to data from a broader range of weather regimes and types
• I wasnt sure what environment you were using to run this notebook. I've created a conda env requirements file and added it to the PR, which I've used to run this notebook. So take a look and see if thats OK.
• Some of the comparison plots e.g. comparing range of air temp for columns with and without precip, it might be good to have them a little smaller and side by side perhaps?
• I like that correlation heatmap, very interesting how that differs between precip / no precip. Will be interesting to compare similarly for different weather regimes etc.
• you might have to give more explaination of the precip accumulation/ cloud fraction connection. I couldn't make anything out from that plot.
• you might want to fix the axis dimensions for the wind speed/direction plots so they don't change between plots, to make comparison easier.
• it might be worth adding plots and some text at the start about the large scale setup for this event, to give some context for some of the later analsyis.

This is a nice template. I think we should produce a notebook for this for each "interesting" event we include in our training dataset.

[stevehadd]

I should have added, I'm, happy for you to merge when you're ready to do, so all, look OK code wise

[stevehadd]

the model pipeline directory stuff in the this PR addresses and closes #6