search

nansencenter / DAPPER

Data Assimilation with Python: a Package for Experimental Research
https://nansencenter.github.io/DAPPER
MIT License
356 stars 124 forks source link

Stocahstic Differential Equation Models / Integration Schemes #70

Open cgrudz opened 3 years ago

cgrudz commented 3 years ago

Hey Y'all,

I wanted to post a proposal about integrating some new features into DAPPER. I will be teaching with DAPPER over the summer and in the fall, and I plan to implement a number of Jupyter example worksheets that will follow experiment-driven learning on various topics in DA and numerical simulation. Because these will be wide survey courses, I would like to add some additional features into DAPPER to teach a variety of topics.

The first feature that I'd like to propose integrating is some functionality around stochastic differential equations. I have a number of Python functions related to this in an old repository that allow for the rigorous simulation of SDEs generally, and some that are designed specifically for the Lorenz-96 model with additive noise. The mathematical details are included here. The L96-s model discussed in the manuscript could be implemented as its own model in DAPPER with the configuration that we studied and rigorously validated in our work.

I will be available for development of the methods and merging these into DAPPER, but if there is time available, I'd be quite happy to develop these methods and a variety of examples with support from others in the project to merge these features. Likewise, I have a number of additional features to start working on upcoming over the next few months in preparation for the fall course.

Let me know if that sounds good, I'll want to have these SDE features and probably some other minor additions merged by the end of June.

Cheers, Colin

patnr commented 3 years ago

So, to summarise my understanding: we can expect quite a few contributions from your part in the next few months. These should be high-quality, as they build on published work, and will be used for teaching in a course. The first one is on stochastic integration methods. We've already discussed this on the phone, and providing it is fairly modular or does not require very major changes the the configuration system, it should be a doable (and valuable) thing! Moreover, for high efficiency, you should have some assistance in implementing the features in DAPPER. I want to be part of this, but am quite busy right now, so I'll take the liberty to ping @yumengch to ask if he could imagine working on this?

The next steps would be to outline below how the implementation would look, more or less, which we can then discuss. Or, if you're feeling confident, you can create a draft PR with some changes, and the discussion can take place there.

To start the discussion: Maybe a good way to start would be to implement the stochastic integration schemes as part of a specific model, and later migrate them into the general codebase. Right now there is some (not much) modularity that fits with this feature: the ensemble methods all use the method da_methods.ensemble.add_noise. I suppose that would have to be moved into the model integrator instead. On the other hand, some DA algorithms like the KF and some of the varieties of the particle filters have their own particular way of dealing with model noise, and it is now clear to me how that could be cleanly unified in the code. 🤔

yumengch commented 3 years ago

Hi all,

I'm learning a bit of sea ice modelling recently, but I think I will have some time for the DAPPER development.

I agree that integrating SDE into the core code base will need some change, especially when it comes to the DA algorithms. Adding an option for SDE integration can be a first step.

cgrudz commented 3 years ago

That all sounds good to me, I'll start reviewing the project structure in the next two weeks and thinking about how to make a merge of code.

Maybe a good way to start would be to implement the stochastic integration schemes as part of a specific model, and later migrate them into the general codebase. 

That seems like a good idea to me, I'll look at implementing the
L96-s model from the manuscript as its own model within other
choices of models.  This can be a good demo of how to merge the
scheme more widely, and I can push my teaching materials that rely
on the model.

I'll keep everyone posted when I get some real plans sketched.

Cheers,

Colin

On 5/13/21 7:38 AM, Yumeng Chen wrote:

Hi all,

I'm learning a bit of sea ice modelling recently, but I think I will have some time for the DAPPER development.

I agree that integrating SDE into the core code base will need some change, especially when it comes to the DA algorithms. Adding an option for SDE integration can be a first step.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/nansencenter/DAPPER/issues/70#issuecomment-840605609, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABXIPFXRBFXYHE3NGRR4RYLTNPP5NANCNFSM44ZIBHNA.