Data assimilation notebooks

dhruvbalwada commented 1 year ago

Hi @feiyulu and @MitchBushuk and @William-gregory -

You had all shown some interest in improving the DA section of our demo (https://m2lines.github.io/L96_demo/notebooks/DA_demo_L96.html). Would it be possible for you all to self-organize and split the work around this? I am happy to discuss more on this as needed. I think Mitch has some slides, so some of the work might just be in converting from slides to notebook format. Maybe skimming the initial part of the machine learning section in the book, can help provide a sense of what the expectations of pedagogy are.

I think the goal is to try to and wrap everything up before the end of the month. I hope this is not too much of an ask.

feiyulu commented 1 year ago

Hi @dhruvbalwada

In terms of the code part, where do you think it needs improvement or more documentation?

dhruvbalwada commented 1 year ago

Honestly, since I am not an expert in DA - I can't really tell you.
If you take a first pass at adding an intro, I will look over it and then tell you if I understood it. Maybe the expectation should be that someone who knows something about modeling and math (beginnner like me) but not much experience with DA can follow. You will see that this is roughly how the machine learning intro was structured. What do you think?

dhruvbalwada commented 1 year ago

@NoraLoose - since you seem to be diving into this book, maybe you would like to take a look here too (given your DA background).

NoraLoose commented 1 year ago

Sure, I'm happy to help. @feiyulu, @MitchBushuk, @William-gregory have you already discussed what still needs to be done?

feiyulu commented 1 year ago

Hi @NoraLoose, I think the plan is to adapt the slides we used during the demo into markdowns at the beginning of the notebook. Then the code might need a bit more documentation and streamlining.

Hi @dhruvbalwada, sorry I won't be able to work on this before the end of June. I had to finish a couple of things after the annual meeting and will have some time off until after July 4th. I will have time to work on this in July.

MitchBushuk commented 1 year ago

Hi all, the slides we used for the presentation are here: https://github.com/MitchBushuk/L96_demo/blob/main/slides/M2LInES%20L96%20DA%20Demo.pdf

These show how DA works in the 1-D case, and then introduce the L96 DA toy problem. I am not claiming they are fully complete, but I think something like this would be a good starting point for the notebook. In addition to the slide content, I think it would be useful to (briefly) describe how the covariance matrix is obtained in 3D-VAR, EnKF, and Hybrid-EnKF, since these all appear in the notebook. Also, a few lines on the concepts of localization and inflation would be helpful, since these are also in the notebook.

I am busy with some deadlines the next few weeks, but think I may have a bit more time in July (pre-baby) to work on this.

dhruvbalwada commented 1 year ago

Hi all, was wondering if there were any updates on this?

feiyulu commented 1 year ago

Hi @MitchBushuk, it seems like your repository is private. Could you share the access?

MitchBushuk commented 1 year ago

@feiyulu, I just shared access. Let me know if you have any issues.

feiyulu commented 1 year ago

Hi all, was wondering if there were any updates on this?

Hi @dhruvbalwada , I'm starting to add some description to the notebook. This is my first new commit: https://github.com/feiyulu/L96_demo/commit/eb1ef925b2e998795c409cc81c2291f53e21c3eb

Do you think this is in the right direction? Since I'm not very familiar with Markdown, it might require some editing eventually.

dhruvbalwada commented 1 year ago

@feiyulu This is starting to look good. Here are few comments:

We don't need section on what is L96. Instead just point to https://m2lines.github.io/L96_demo/notebooks/L96-two-scale-description.html
In the section on "What is DA?", please define what is 'X'.
While it is good to have all the functions in the notebook. It would be even better if these functions were explained and defined when they are first used in the notebook; explaining how that functions fits in with the 6 step process put forwards in the "Design of the data assimilation ..." section. At the moment the reader is presented with functions, without any background on them.
In section "Apply localization to the background model covariance", could you say more about what the covariance matrix is?
In the section "Run data assimilation", could you explain what is the operator matrix?

dhruvbalwada commented 1 year ago

Also don't worry too much about the markdown editing, we will be able to do that later. @IamShubhamGupto is an expert in that and can help sort it out once all the information content is in.

dhruvbalwada commented 1 year ago

Also might be good to reference these tutorials (or similar ones) in the notebook somewhere: https://dart.ucar.edu/tutorials/

dhruvbalwada commented 1 year ago

@feiyulu - any updates on the status of this notebook?

feiyulu commented 1 year ago

@feiyulu - any updates on the status of this notebook?

Yes, I did make some more changes a couple of weeks ago. I moved all the functions to the python module file to declutter the notebook, and added some explanation in the notebook.

dhruvbalwada commented 1 year ago

Is it ready to make a pull request then to the main L96? or are you planning to make some more changes before?

feiyulu commented 1 year ago

I think I can do a PR and check how it looks so far.

m2lines / L96_demo

Data assimilation notebooks #119