search

pySTEPS / pysteps

Python framework for short-term ensemble prediction systems.
https://pysteps.github.io/
BSD 3-Clause "New" or "Revised" License
468 stars 169 forks source link

Implement support for three-dimensional input data #129

Open pulkkins opened 4 years ago

pulkkins commented 4 years ago

Using three-dimensional input data from full radar volumes (instead of lowest-angle PPI or a low-level CAPPI) for extrapolation-based nowcasting is showing great promise. A number of papers have already been published about this. Therefore, it would be highly useful to implement support for three-dimensional input data into pysteps. There are several interesting ideas that could be implemented:

1) Computation of the vertically integrated liquid (VIL), which results into a two-dimensional field. This can be easily implemented into pysteps. However, an estimate for the freezing level height is needed for this. Also, if the VIL is used as the forecast variable, it needs to be converted into an estimate of surface rain rate. This can be done by simple linear regression. 2) Extension of advection field estimation into three dimensions. As far as I know, the OpenCV implementation of Lucas-Kanade does not support three-dimensional inputs, and a custom version needs to be implemented from scratch. For the other optical flow methods, the extension should be straightforward. 3) Extension of the cascade decomposition and the autoregressive model into three dimensions. A vector autoregressive (VAR) model, where the rain rate/reflectivity at each altitude level is a separate variable, could also be tried. 4) Three-dimensional noise generators.

What do you think?

dnerini commented 4 years ago

Hi Seppo

This sounds all very exciting and would represent an excellent contribution to pysteps! Similar developments are also being explored in machine learning approches to nowcasting (eg. this paper from Tran and Song).

Concerning point 1, I'm not sure how this fits within pysteps: computing the VIL is somewhat equivalent to computing the QPE field, as it represents a 2d analysis field from volumetric radar observations. In this sense, I would not consider it within the scope of pystpes. The 2d field of VIL can already be used for nowcasting with the current pysteps routines.

Concerning point 2, it is true that the OpenCV LK implementation does not support 3d arrays, but I don't think that it will be necessary to support 3d inputs in all the optical flow methods. Clearly, it would be interesting to implement a custom version of LK which supports it, but this should not slow down the rest of the developments.

Point 3 is really interesting and relates to #115. Maybe we should add a new nowcasting method instead of trying to include this idea as an option in the existing STEPS routine (in the same way SSEPS represents the localization of STEPS).

Point 4 is fairly straightforward for the non-parametric generator (assuming a regular spacing in the vertical dimension, too). The parametric generator currently will assume the same resolution on all dimensions, but this won't be difficult to generalize, I assume.

We should also think whether the pysteps.utils routines can ingest volumetric arrays, but my guess is that in most cases this is already possible.

Just one general concern: there are now quite a number of important developments still open (e.g. #115 for advanced AR models, including the use of an ARI model for STEPS, #78 for the spectral implementation and #12 for the xarray support). Perhaps it could be wise to complete those before opening yet another new large development?

What do you think?

pulkkins commented 4 years ago

Hi,

Thanks for your insightful comments. I have been thinking this a little bit more and came up with the following.

Point 1: I completely agree that computation of the VIL should be kept outside pysteps. I just wanted to point out that if you have reflectivity interpolated to a 3d grid, computation of the VIL is trivial to implement.

Point 2: Computation of a 3d advection field is an interesting idea but probably not very practical. In my opinion, the best approach is to model the horizontal advection and the vertical evolution separately to keep the computational requirements reasonable.

Point 3: Yes, this should be implemented in a separate module. In fact, I have already done this. The new method is called autoregressive nowcasting using VIL (ANVIL). The paper has been submitted to IEEE TGRS, and I will implement ANVIL to pysteps in the near future.

Finally, I agree that this issue should be addressed after the ones you mentioned have been completed.