Raise warning on NaN input

[emiliano-gelati]

Question raised by @bissebe:

"Last week i was running LISFLOOD with the new meteo dataset (1990-2018) and got unrealistic results which took me a while to find out why. I found that in the generated LISVAP e0, et and es maps missing values appear at some timesteps (for e.g., 27-01-1999), which are caused by the square root of negative vapor pressure (pd) values (EmNet = (0.56 - 0.079 * operations.sqrt(self.EAct)).

Both LISVAP and LISFLOOD are generating output which is not useful and therefore I suggest to build in a warning to prevent unnecessarily post-processing work.

Can the negative values in the file EMO-5-pd_1990_2017.nc be fixed?"

[domeniconappo]

@bissebe @emiliano-gelati

We confirmed that problem relies in input files. In this case, erroneous negative values in PD - probably originated by interpolation - generating NaN values in lisvap output (e0, es) and, in turn, in lisflood output.

So the proposal is actually to Raise a warning on NaN output? Which can be a red raising flag that can help to detect the problem quickly?

[emiliano-gelati]

@domeniconappo @bissebe Yes - for instance, we could have something like https://github.com/ec-jrc/lisflood-code/commit/445102fb9887368e49fa6022d0406c3cb98947c2 or https://github.com/ec-jrc/lisflood-code/commit/04a892cec805d21d4b703c983fe8ce2784cf6318

In https://github.com/ec-jrc/lisflood-code/commit/04a892cec805d21d4b703c983fe8ce2784cf6318, the NaN check is nested under if Flags['check']. Users should be advised to use "-c" flag the first time they run the model with new input (python2 lisf1.py settings.xml -c). However, this generates a lot of screen output, which may be confusing.

If we go for https://github.com/ec-jrc/lisflood-code/commit/445102fb9887368e49fa6022d0406c3cb98947c2, the NaN check is executed at each time step (it is not computationally expensive though).

In the longer term, the input check procedure could be redefined.

[domeniconappo]

@emiliano-gelati Yes it's worth to define better the check procedure and to put in documentation some details.

What I was saying is that we also should add that check to output variables (both for Lisvap and Lisflood). Because we saw that NaN in output can happen even if there are no NaN values in input maps, as it's for LISVAP where negative values are causing illegal math operations leading to NaN in output values.

@emiliano-gelati Could you add that in Lisflood? I can do the same for Lisvap.

[emiliano-gelati]

Sorry @domeniconappo, I misread output for input.

I agree with your idea. I will propose a value-check procedure for input and output in LISFLOOD: tentatively, I will add a key to the Flags dictionary in https://github.com/ec-jrc/lisflood-code/blob/bf56ca0283f0ba4dc142b473f19d853a67eff2c1/src/lisflood/global_modules/globals.py#L103-L131

[domeniconappo]

@emiliano-gelati Thanks. I also agree with this approach. let us know when it's ready to merge :)

[domeniconappo]

For tracking here (as we are using yet another issue tracker)... Problem was originated by interpolation algorithm spheremap

by Markus Ziese

negative values could be caused by the gradient calculated in SPHEREMAP. I do not assume that it is caused by the height correction.

SPHEREMAP combines an inverse distance weighting, angular weighting and calculation of a gradient based on the location and value of the input values. This was introduced as it is well known that observations not always catch the real maximum of an event. In some cases the gradient should produce an overshot (grid cell values higher/lower than input values), where negative values are taken into account for precipitation but probably not for water vapor pressure. Therefore you should find no negative precipitation values.

I propose to use the same principle for water vapor pressure as for precipitation to avoid negative grid cell values.

We should also apply this to wind speed, as negative wind speed should be a return of the wind direction.

I have to analyse our procedure to find the right place for the correction: either in the SPHEREMAP source code or in the script.

[emiliano-gelati]

@domeniconappo Proposed changes in 8fd7dba6a194f19999859f4c6180831969a39d80 Ready to merge if OK for you

[domeniconappo]

Great. Thanks Emiliano. For me it's ok.

[emiliano-gelati]

thanks, merged.