Noise Adjustment Coefficient -> ValueError: field contains non-finite values

5fcgdaeb commented 2 years ago

I am performing nowcasts with the setup below but I am getting an error during the noise adjustment process.

Here is the setup:

Methods:
--------
extrapolation:          semilagrangian
bandpass filter:        gaussian
decomposition:          fft
noise generator:        parametric
noise adjustment:       yes
velocity perturbator:   bps
conditional statistics: yes
precip. mask method:    incremental
probability matching:   cdf
FFT method:             numpy
domain:                 spatial

The error trace looks like:

Computing noise adjustment coefficients... Traceback (most recent call last): File "historicalNowcastPerformerScript.py", line 96, in <module> R_forecasts = nowcaster.nowcast(R, metadata) File "/root/projects/lib/python3.7/site-packages/pysteps/nowcasts/steps.py", line 451, in forecast num_workers=num_workers, File "/root/projects/lib/python3.7/site-packages/pysteps/noise/utils.py", line 95, in compute_noise_stddev_adjs decomp_R = decomp_method(R, F, mask=MASK_) File "/root/projects/lib/python3.7/site-packages/pysteps/cascade/decomposition.py", line 191, in decomposition_fft raise ValueError("field contains non-finite values") ValueError: field contains non-finite values

From what I have analyzed, the problem here is that the last observation input contains all zeros; meaning there is no precipitation. In that case, the noise adjustment algorithm fails. To be more precise, inside the compute_noise_stddev_adjs method of the utils.py file, the calculated mu value is nan; making the calculations non-finite.

Would there be a way to skip noise adjustment in case there is no precipitation?

dnerini commented 2 years ago

Thanks for reporting the issue! I will have a look into it, can you maybe share a minimal code to reproduce the error? From what I understand, it should be enough to set the last precipitation field to all zeros?

Also, what pysteps version are you using?

5fcgdaeb commented 2 years ago

@dnerini Thanks for the quick reply.

My pysteps version is 1.6.1.

The same error also occurred inside the nowcasting module, in the temporal_autocorrelation method (line number 107). This is probably where the forecasts (R_fct) are being verified so it is not just about the input values but also about the forecasted precipitation fields. If the forecast contains no precipitation, the same crash is happening.

noise std. dev. coeffs:   [0.89172898 1.04151636 0.93850691 0.97760406 0.89743188 0.85347049]
Traceback (most recent call last):
  File "/Users//Library/Application Support/JetBrains/Toolbox/apps/PyCharm-C/ch-0/211.7442.45/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/pydevd.py", line 1483, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script

  File "/Users//Library/Application Support/JetBrains/Toolbox/apps/PyCharm-C/ch-0/211.7442.45/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)

  File "/Users//Documents//scripts/utility/radar/historicalNowcastPerformerScript.py", line 101, in <module>
    R_forecasts = nowcaster.nowcast(R, metadata)

  File "../../../library/radar/nowcaster/algorithms/STEPSNowcaster.py", line 70, in nowcast
    seed=seed)

  File "/Users//Documents//lib/python3.7/site-packages/pysteps/nowcasts/steps.py", line 492, in forecast
    GAMMA[i, :] = correlation.temporal_autocorrelation(R_c[i], mask=MASK_thr)

  File "/Users//Documents//lib/python3.7/site-packages/pysteps/timeseries/correlation.py", line 107, in temporal_autocorrelation
    raise ValueError("x contains non-finite values")

ValueError: x contains non-finite values

Yeah, the minimal code would be tough to share but I can provide a bit more details:

Feeding in 8 observation timestamps, only 1-2 of them contains precipitation that is above the R_threshold (which is 10.0) for me. The observation doesn't have to be all zeros, but they should be below 10.0.
If the last observed timestamp contains no precipitation, there is a crash in during the noise correction.
If the forecasted values have no precipitation, there is a crash during the temporal correlation.

Very minimal code looks like:

R_fct = nwc_method(R, UV, n_timesteps, n_ens_members,  # 6 timesteps, 1 ensemble
                           n_cascade_levels,
                           kmperpixel=metadata["xpixelsize"] / 1000,
                           timestep=metadata["timestep"],   
                           R_thr=metadata["threshold"], # 10.0 for me
                           extrap_method=adv_method,
                           decomp_method=decomp_method,
                           bandpass_filter_method=bandpass_filter,
                           noise_method=noise_method,
                           noise_stddev_adj=noise_stddev_adj,
                           ar_order=ar_order,
                           conditional=conditional,
                           mask_method=mask_method,
                           seed=seed)

I highly doubt that the library didn't consider the case where there is no precipitation; this might be a user error. It would be great if you could double-check. When there is precipitation, the forecasts look pretty good.

dnerini commented 2 years ago

Great, thanks for the detailed information! I hope to be able to look into this in the coming days, will let you know!

dnerini commented 4 months ago

closing following resolution of #309 with #351, feel free to reopen if you still notice a problem.

pySTEPS / pysteps

Noise Adjustment Coefficient -> ValueError: field contains non-finite values #282