search

monocongo / climate_indices

Climate indices for drought monitoring
https://monocongo.github.io/climate_indices/
Other
335 stars 160 forks source link

invalid L-moments SPEI #441

Open mabelcalim opened 2 years ago

mabelcalim commented 2 years ago

I'm using the PyPi version of climate indices. I'm getting this " Unable to calculate Pearson Type III parameters due to invalid L-moments" trying to submit SPEI and SPI calculation .

Describe the bug ValueError: Unable to calculate Pearson Type III parameters due to invalid L-moments To Reproduce This is my code: process_climate_indices --index spei --periodicity monthly --netcdf_precip /Users/calim/code/CEMADEN/data/prec_pant_axis_new_latpet_ascend.nc --var_name_precip precip --netcdf_pet /Users/calim/code/CEMADEN/data/PET_thornthwaite_ascend.nc --var_name_pet pet_thornthwaite --output_file_base /Users/calim/code/CEMADEN/data/nclimgrid_lowres --scales 6 12 --calibration_start_year 1981 --calibration_end_year 2021 --multiprocessing all

Error: 2021-08-26 17:55:33 INFO Start time: 2021-08-26 17:55:33.291960 2021-08-26 17:55:33 INFO Computing monthly SPI 2021-08-26 17:55:33 INFO Computing 1-month SPI (Gamma) 2021-08-26 17:55:53 INFO Computing 1-month SPI (Pearson) 2021-08-26 17:56:04 ERROR Unable to calculate Pearson Type III parameters due to invalid L-moments 2021-08-26 17:56:04 ERROR Unable to calculate Pearson Type III parameters due to invalid L-moments 2021-08-26 17:56:04 ERROR Unable to calculate Pearson Type III parameters due to invalid L-moments 2021-08-26 17:56:04 ERROR Unable to calculate Pearson Type III parameters due to invalid L-moments 2021-08-26 17:56:04 ERROR Failed to complete File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/pool.py", line 768, in get raise self._value ValueError: Unable to calculate Pearson Type III parameters due to invalid L-moments

prec_pant_axis_new_latpet_ascend.nc - lat and lon values in ascending order, shape(lat,lon,time) = (33, 41, 487) PET_thornthwaite_ascend.nc - get from climate-indices PET- also lat and lon values in ascending order, shape(lat,lon,time) = (33, 41, 487) Files from Pantanal (Brasil) region from 1981-2021.

Expected behavior SPEI netcdf files

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

Additional context I'm using the PyPi version of climate-indices==1.0.9

monocongo commented 2 years ago

Thanks @mabelcalim

Can you please post a link to your input datasets? This will allow me to attempt to reproduce and debug the error.

monocongo commented 2 years ago

@mabelcalim can you please try again with the latest code from the master branch? There were changes made recently which may have fixed your issue.

bennyistanto commented 2 years ago

@mabelcalim can you please try again with the latest code from the master branch? There were changes made recently which may have fixed your issue.

I also experiencing the same error: invalid L-moments. Btw how to install the latest code from master branch? Currently I use pip install climate-indices

UPDATE I tried to install using pip install https://github.com/monocongo/climate_indices/archive/refs/heads/master.zip and unfortunately the error remain same.

(climate_indices) INEB:Tile11 bennyistanto$ spi --periodicity monthly --scales 1 --calibration_start_year 1958 --calibration_end_year 2020 --netcdf_precip /Users/bennyistanto/Temp/TERRACLIMATE/CDO/Tile11/Tile11_TerraClimate_ppt_1958_2020_llt.nc --var_name_precip ppt --output_file_base /Users/bennyistanto/Temp/TERRACLIMATE/SPI/Outputs/nc_original/tile11_wld_cli --multiprocessing all --save_params /Users/bennyistanto/Temp/TERRACLIMATE/SPI/Fitting/tile11_wld_cli_spei_fitting.nc --overwrite
2021-11-05  17:01:42 INFO Start time:    2021-11-05 17:01:42.828205
2021-11-05  17:01:42 INFO Computing monthly SPI
2021-11-05  17:04:02 INFO Computing 1-month SPI (Gamma)
2021-11-05  18:12:28 INFO Computing 1-month SPI (Pearson)
2021-11-05  18:13:58 ERROR Unable to calculate Pearson Type III parameters due to invalid L-moments
2021-11-05  18:15:43 ERROR Unable to calculate Pearson Type III parameters due to invalid L-moments
2021-11-05  18:15:43 ERROR Failed to complete
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/Users/bennyistanto/opt/anaconda3/envs/climate_indices/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/Users/bennyistanto/opt/anaconda3/envs/climate_indices/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/Users/bennyistanto/opt/anaconda3/envs/climate_indices/lib/python3.7/site-packages/climate_indices/__spi__.py", line 1309, in _apply_to_subarray_pearson
    periodicity=args["periodicity"],
  File "/Users/bennyistanto/opt/anaconda3/envs/climate_indices/lib/python3.7/site-packages/climate_indices/lmoments.py", line 30, in fit
    raise ValueError(message)
ValueError: Unable to calculate Pearson Type III parameters due to invalid L-moments
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/bennyistanto/opt/anaconda3/envs/climate_indices/lib/python3.7/site-packages/climate_indices/__spi__.py", line 1523, in main
    _compute_write_index(kwrgs)
  File "/Users/bennyistanto/opt/anaconda3/envs/climate_indices/lib/python3.7/site-packages/climate_indices/__spi__.py", line 709, in _compute_write_index
    args=args,
  File "/Users/bennyistanto/opt/anaconda3/envs/climate_indices/lib/python3.7/site-packages/climate_indices/__spi__.py", line 1018, in _parallel_fitting
    pool.map(_apply_to_subarray_pearson, chunk_params)
  File "/Users/bennyistanto/opt/anaconda3/envs/climate_indices/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/Users/bennyistanto/opt/anaconda3/envs/climate_indices/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
  File "/Users/bennyistanto/opt/anaconda3/envs/climate_indices/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/Users/bennyistanto/opt/anaconda3/envs/climate_indices/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/Users/bennyistanto/opt/anaconda3/envs/climate_indices/lib/python3.7/site-packages/climate_indices/__spi__.py", line 1309, in _apply_to_subarray_pearson
    periodicity=args["periodicity"],
  File "/Users/bennyistanto/opt/anaconda3/envs/climate_indices/lib/python3.7/site-packages/climate_indices/lmoments.py", line 30, in fit
    raise ValueError(message)
ValueError: Unable to calculate Pearson Type III parameters due to invalid L-moments
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/Users/bennyistanto/opt/anaconda3/envs/climate_indices/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/Users/bennyistanto/opt/anaconda3/envs/climate_indices/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/Users/bennyistanto/opt/anaconda3/envs/climate_indices/lib/python3.7/site-packages/climate_indices/__spi__.py", line 1309, in _apply_to_subarray_pearson
    periodicity=args["periodicity"],
  File "/Users/bennyistanto/opt/anaconda3/envs/climate_indices/lib/python3.7/site-packages/climate_indices/lmoments.py", line 30, in fit
    raise ValueError(message)
ValueError: Unable to calculate Pearson Type III parameters due to invalid L-moments
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/bennyistanto/opt/anaconda3/envs/climate_indices/bin/spi", line 8, in <module>
    sys.exit(main())
  File "/Users/bennyistanto/opt/anaconda3/envs/climate_indices/lib/python3.7/site-packages/climate_indices/__spi__.py", line 1523, in main
    _compute_write_index(kwrgs)
  File "/Users/bennyistanto/opt/anaconda3/envs/climate_indices/lib/python3.7/site-packages/climate_indices/__spi__.py", line 709, in _compute_write_index
    args=args,
  File "/Users/bennyistanto/opt/anaconda3/envs/climate_indices/lib/python3.7/site-packages/climate_indices/__spi__.py", line 1018, in _parallel_fitting
    pool.map(_apply_to_subarray_pearson, chunk_params)
  File "/Users/bennyistanto/opt/anaconda3/envs/climate_indices/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/Users/bennyistanto/opt/anaconda3/envs/climate_indices/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
  File "/Users/bennyistanto/opt/anaconda3/envs/climate_indices/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/Users/bennyistanto/opt/anaconda3/envs/climate_indices/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/Users/bennyistanto/opt/anaconda3/envs/climate_indices/lib/python3.7/site-packages/climate_indices/__spi__.py", line 1309, in _apply_to_subarray_pearson
    periodicity=args["periodicity"],
  File "/Users/bennyistanto/opt/anaconda3/envs/climate_indices/lib/python3.7/site-packages/climate_indices/lmoments.py", line 30, in fit
    raise ValueError(message)
ValueError: Unable to calculate Pearson Type III parameters due to invalid L-moments
(climate_indices) INEB:Tile11 bennyistanto$

Data: https://drive.google.com/file/d/1GaVFVux2rH1fUNh9w6u6SRUun-Rs9a_i/view?usp=sharing

mabelcalim commented 2 years ago

Hi James and Benny,

This issue was solved by reevaluating the shape of the input time series. Check if the prec, temp have the same shape and format of evapotransp (from Thornthwaite PET script) and the time dimension is monotonically ascendent.

Mabel Calim Costa GitHub: https://github.com/mabelcalim ORCID: https://orcid.org/0000-0003-3787-3983 Lattes: http://lattes.cnpq.br/3002998002368589

On Fri, 5 Nov 2021 at 06:52, Benny Istanto @.***> wrote:

@mabelcalim https://github.com/mabelcalim can you please try again with the latest code from the master branch? There were changes made recently which may have fixed your issue.

I also experiencing the same error: invalid L-moments. Btw how to install the latest code from master branch? Currently I use pip install climate-indices

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/monocongo/climate_indices/issues/441#issuecomment-961759613, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABRW6DJ2CIFG6UMPG2BTBCLUKOSN7ANCNFSM5C4IC5EA .

bennyistanto commented 2 years ago

Hi James and Benny, This issue was solved by reevaluating the shape of the input time series. Check if the prec, temp have the same shape and format of evapotransp (from Thornthwaite PET script) and the time dimension is monotonically ascendent. …

Hi Mabel, thank you for the hint. I have check both data and its correct, above example is for SPI calculation and I also try the SPEI too and got the same error.

monocongo commented 2 years ago

I have updated the version in PyPI to 1.0.10, @bennyistanto please try installing this new version, please report if you are still having issues.

bennyistanto commented 2 years ago

I have updated the version in PyPI to 1.0.10, @bennyistanto please try installing this new version, please report if you are still having issues.

Thank you James! I have tested to calculate SPEI and it's worked.

But strange that the SPI calculation remain failed with same error ValueError: Unable to calculate Pearson Type III parameters due to invalid L-moments

UPDATE I have create new environment and install 1.0.10, it's worked for general SPI but failed if I want to use pre-compute SPI distribution fitting variable by including this --save_params /data/path/fitting.nc --overwrite and the error message is same.

bennyistanto commented 2 years ago

Hi James, I would like to report back that I experienced this "ERROR Unable to calculate Pearson Type III parameters due to invalid L-moments" again.

For this case I used:

  1. Mac mini server 2012, i7 16GB RAM and TB's of SSD, running on macOS Catalina. With Anaconda installed on the machine.
  2. Jupyter Notebook Server from my office (I don't know the specs). Using this machine I only need 2 minutes to run single tile SPEI, while using Mac mini it takes 20 minutes.

As a background:

  1. I am using global TerraClimate precipitation and pet data (both is monthly data from 1958-2020, with 4km spatial resolution) to calculate global SPEI. I have upload the data in here: https://drive.google.com/drive/folders/1-E09-bSp9gmPZruqcqswbQaL4G6NekP3?usp=sharing
  2. For testing in Mac mini, I decide to divided the global data into 32 tiles, each file covering 45x45deg.
  3. I decided to reorder the dimension for all of the data from time,lat,lon to lat,lon,time, so the script will accept the data nicely without forcing to reorder it. The data is available in here: https://drive.google.com/drive/folders/1--1u7_-sfRxC-UeLEvci4zIicCuZ5s8_?usp=sharing
  4. To calculate SPEI, I used below code process_climate_indices --index spei --periodicity monthly --netcdf_precip Tile31_TerraClimate_ppt_1958_2020.nc --var_name_precip ppt --netcdf_pet Tile31_TerraClimate_pet_1958_2020.nc --var_name_pet pet --output_file_base ../outputs/Tile31_wld_cli --scales 1 2 3 6 9 12 18 24 36 48 60 72 --calibration_start_year 1958 --calibration_end_year 2020 --multiprocessing all

Unfortunately, 7 out of 32 tiles are failed in the calculation process (highlighted orange below). Screen Shot 2022-01-05 at 2 33 01 PM

I have try in both Mac mini and Jupyter server, and failed.

monocongo commented 2 years ago

@bennyistanto I apologize in advance it may take me a while to look into and resolve this issue. Have you used the latest version from the master branch? I will soon make an upload to PyPI with the latest version which may fix this for you (there were good fixes added for the PE3 distribution fitting which is perhaps relevant here, although I think those changes are already in the version on PyPI, I should verify that).

bennyistanto commented 2 years ago

@bennyistanto I apologize in advance it may take me a while to look into and resolve this issue. Have you used the latest version from the master branch? I will soon make an upload to PyPI with the latest version which may fix this for you (there were good fixes added for the PE3 distribution fitting which is perhaps relevant here, although I think those changes are already in the version on PyPI, I should verify that).

Yes, I used the latest version 1.0.10. Thanks you @monocongo looking forward to the update.

MarceloZeri commented 2 years ago

Hi, I have the exact same problem (ValueError: Unable to calculate Pearson Type III parameters due to invalid L-moments) while trying to calculate SPI using CHIRPS data (on an iMac Pro Intel). By the way, first I had to include the units mm/month in the original code for SPI, and change the names of dimensions and variables from "latitude" and "longitude" to lat,lon (following a python code in another issue here).

I just re-installed anaconda with python 3.9, the climate-indices 1.0.10, etc. The error:

[10:16:16 marcelozeri]$ spi --periodicity monthly --scales 1 --netcdf_precip chirps-v2.0.monthly_BR2.nc --output_file_base SPIs --var_name_precip precip --calibration_start_year 1981 --calibration_end_year 2011 --multiprocessing all_but_one 2022-04-29 10:16:38 INFO Start time: 2022-04-29 10:16:38.949039 2022-04-29 10:17:40 INFO Computing monthly SPI 2022-04-29 10:17:55 INFO Computing 1-month SPI (Gamma) 2022-04-29 10:18:59 INFO Computing 1-month SPI (Pearson) 2022-04-29 10:19:11 ERROR Unable to calculate Pearson Type III parameters due to invalid L-moments 2022-04-29 10:19:18 ERROR Unable to calculate Pearson Type III parameters due to invalid L-moments 2022-04-29 10:19:48 ERROR Failed to complete multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/Users/marcelozeri/opt/anaconda3/lib/python3.9/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, *kwds)) File "/Users/marcelozeri/opt/anaconda3/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar return list(map(args)) File "/Users/marcelozeri/opt/anaconda3/lib/python3.9/site-packages/climate_indices/spi.py", line 1296, in _apply_to_subarray_pearson compute.pearson_parameters( File "/Users/marcelozeri/opt/anaconda3/lib/python3.9/site-packages/climate_indices/lmoments.py", line 30, in fit raise ValueError(message) ValueError: Unable to calculate Pearson Type III parameters due to invalid L-moments """

Update after this: Actually the SPIs_spi_gamma_1_month.nc was being generated, but not the Pearson version. That was the case for the SPI 3 months too. However, the command above worked for SPI 6 months, generating both the Gamma and Pearson SPIs, but I had to run the command individually for each scale (1, 3, 6) because the error in the Pearson version would stop the process while moving to the other scales.

peterataylor commented 1 year ago

Hi @monocongo, @bennyistanto ,

I also ran into this issue while generating SPEI using TerraClimate, with a Pearson distribution. I've been able to isolate and reproduce the issue using some specific cells within TerraClimate. It appears to be in cells that either have a high count of zero pET or all zero pET. The error varies depending on the scale used (1 vs 3 month SPEI).

I've written a notebook that shows the working here: https://github.com/peterataylor/climate_indices/blob/master/notebooks/SPEI_pearson_lmoment_bug_example.ipynb. I show the issue and then demonstrate you can remove the issue by filling with a distribution of low pET values for the zeros. Also show how this case affects 1-month but not 3-month SPEI.

Not sure of the ultimate fix in terms of adjustments to the l-moment calculation, but this should provide a test case.

lbwheeler commented 1 year ago

I am also running into an L-moment related error. The example case for SPI works as expected but when I attempt an SPI calculation on a different file, I get the following error:

process_climate_indices --index spi --periodicity monthly --netcdf_precip data/test.nc --var_name_precip PRECC --output_file_base output/test_case_001 --scales 3 --calibration_start_year 2011 --calibration_end_year 2015 --multiprocessing single

2022-10-11  15:01:26 INFO Start time:    2022-10-11 15:01:26.036058
2022-10-11  15:01:26 INFO Computing 3-month SPI/Pearson
2022-10-11  15:02:56 ERROR Unable to calculate Pearson Type III parameters due to invalid L-moments
2022-10-11  15:02:56 ERROR Failed to complete
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/lib/python3.9/site-packages/climate_indices/__main__.py", line 1342, in _apply_along_axis
    computed_array = np.apply_along_axis(func1d,
  File "<__array_function__ internals>", line 180, in apply_along_axis
  File "/lib/python3.9/site-packages/numpy/lib/shape_base.py", line 402, in apply_along_axis
    buff[ind] = asanyarray(func1d(inarr_view[ind], *args, **kwargs))
  File "/lib/python3.9/site-packages/climate_indices/__main__.py", line 1116, in _spi
    return indices.spi(values=precips,
  File "/lib/python3.9/site-packages/climate_indices/compute.py", line 549, in transform_fitted_pearson
    pearson_parameters(
  File "/lib/python3.9/site-packages/climate_indices/lmoments.py", line 36, in fit
    raise ValueError(message)
ValueError: Unable to calculate Pearson Type III parameters due to invalid L-moments
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/lib/python3.9/site-packages/climate_indices/__main__.py", line 1692, in main
    _compute_write_index(kwrgs)
  File "/lib/python3.9/site-packages/climate_indices/__main__.py", line 1015, in _compute_write_index
    _parallel_process(keyword_arguments["index"],
  File "/lib/python3.9/site-packages/climate_indices/__main__.py", line 1305, in _parallel_process
    pool.map(_apply_along_axis, chunk_params)
  File "/lib/python3.9/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
ValueError: Unable to calculate Pearson Type III parameters due to invalid L-moments
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/lib/python3.9/site-packages/climate_indices/__main__.py", line 1342, in _apply_along_axis
    computed_array = np.apply_along_axis(func1d,
  File "<__array_function__ internals>", line 180, in apply_along_axis
  File "/lib/python3.9/site-packages/numpy/lib/shape_base.py", line 402, in apply_along_axis
    buff[ind] = asanyarray(func1d(inarr_view[ind], *args, **kwargs))
  File "/lib/python3.9/site-packages/climate_indices/__main__.py", line 1116, in _spi
    return indices.spi(values=precips,
  File "/lib/python3.9/site-packages/climate_indices/compute.py", line 549, in transform_fitted_pearson
    pearson_parameters(
  File "/lib/python3.9/site-packages/climate_indices/lmoments.py", line 36, in fit
    raise ValueError(message)
ValueError: Unable to calculate Pearson Type III parameters due to invalid L-moments
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/bin/process_climate_indices", line 8, in <module>
    sys.exit(main())
  File "/lib/python3.9/site-packages/climate_indices/__main__.py", line 1692, in main
    _compute_write_index(kwrgs)
  File "/lib/python3.9/site-packages/climate_indices/__main__.py", line 1015, in _compute_write_index
    _parallel_process(keyword_arguments["index"],
  File "/lib/python3.9/site-packages/climate_indices/__main__.py", line 1305, in _parallel_process
    pool.map(_apply_along_axis, chunk_params)
  File "/lib/python3.9/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
ValueError: Unable to calculate Pearson Type III parameters due to invalid L-moments

The same line works just fine if I use the example code and have tested it with a similarly short duration reference period. I have also taken the dataset and modified the dims to match that of the example. Below is the header for the file:

dimensions:
    lat = UNLIMITED ; // (192 currently)
    lon = 288 ;
    time = 120 ;
variables:
    float PRECC(lat, lon, time) ;
        PRECC:units = "m/s" ;
        PRECC:long_name = "Convective precipitation rate (liq + ice)" ;
        PRECC:cell_methods = "time: mean" ;
    double lat(lat) ;
        lat:units = "degrees_north" ;
        lat:long_name = "latitude" ;
    double lon(lon) ;
        lon:units = "degrees_east" ;
        lon:long_name = "longitude" ;
    double time(time) ;
        time:units = "days since 2010-01-01 00:00:00" ;
        time:long_name = "time" ;
        time:calendar = "noleap" ;
        time:bounds = "time_bnds" ;

I should note that I modified main.py to convert m/s to mm from the monthly input file.