Support for non-uniform observations?

[Balinus]

Hello!

Before I try to call an external model in C++, I wanted to know if there is a support for non-uniform observations? For example, here's our constraint with a hydrological model. We want to assimilate snow data that are gathered "randomly" by human.

• Our observations are not "periodic" : most years, but not always, observations are available on February 15th, March 1st-15th, April 1st
• Rest of the year, there are no observations
• Some year, we have continuous obs between February and May

Cheers|

[baggepinnen]

Are you talking about non-uniform sampling? (as opposed to non-uniformly distributed observation noise) There is no explicit support for non-uniform sampling, but the dynamics can be time varying so it's possible to accommodate this use case albeit in a slightly manual and awkward fashion. See https://baggepinnen.github.io/LowLevelParticleFilters.jl/dev/discretization/#Non-uniform-sample-rates-1 for more details.

[Balinus]

Thanks!

Yes, it was about about the time and periodicity of sampling : sometimes we have data for for X dates throughout the year and sometimes it is Y dates. Since it is snow data, we don't have data in summer. Hence, observation noise is time-dependent. Ideally, we would like to have "non-uniformly distributed observation noise" in the sense that the data is positively defined (bounded at 0) and the uncertainty is a function of the value (not the same uncertainty for a 1cm snow data versus 40cm snow data). Hope it make sense!

Thanks for the link, I think it answer the non-uniform sample rate. We just need to call correct!(...,t) when y(t) is defined

[baggepinnen]

All Kalman-filter type algorithms depend on Gaussian noise, but particle filters do not. There are several other state-estimation algorithms available elsewhere as well that can do away with the restriction of Gaussian noise, but the particle filters are the only ones available in this package.

All estimators in this package should support time and state dependent measurement noise covariance though. There is a tutorial in the docs https://baggepinnen.github.io/LowLevelParticleFilters.jl/dev/beetle_example/ that illustrates advanced usage of a particle filter with custom noise properties.

[Balinus]

Thanks!

Yes indeed, for KF algos, we were aware of this limitation and this is one of the reason we are actively looking at PF. Thanks for the link, we will try to build something, using the external model.

A side-question for the external model : can we use the variable p to store file information for the external model? i.e.

dynamics(x, u, p, t) -> x(t+1)

[baggepinnen]

The p field can be anything, and no matter what it is, it is just passed along to all functions that require it, so I think you should be able to put your file information in there and write the dynamics function to make use of the information. This is at least how it is supposed to work, so if you run into problems doing this we should fix it ;)

[Balinus]

Thanks! :) I'll write something up and see where it leads 😄

[baggepinnen]

Did you manage to make this work for you?

[Balinus]

I didn't tried enough sadly. The project decided to go with something else, hence I haven't spent lots of time on this. We might revisit it in a couple of months. Thanks again!