ARCCSS-extremes / climpact

Calculate the ET-SCI climate extremes indices
GNU General Public License v3.0
42 stars 29 forks source link

Error in ncdf4.helpers #85

Closed dumontgoulart closed 2 years ago

dumontgoulart commented 2 years ago

First of all, thank you very much for this incredible tool.

I ran the test with the sample and everything worked well, but when testing with my three files, the following error appears:

Loading required package: PCICt
Loading required package: lmomco
Loading required package: parallel
Loading required package: ggplot2
# Package SPEI (1.7) loaded [try SPEINews()].
Error in ncdf4.helpers::get.cluster.worker.subsets(max.vals.millions *  :
  rows.per.slice >= 1 is not TRUE
Calls: create.indices.from.files -> <Anonymous> -> stopifnot
Execution halted

A quick view on the files themselves:

netcdf pr_day_gswp3-w5e5_1971_2016 {
dimensions:
        time = UNLIMITED ; // (16802 currently)
        lon = 720 ;
        lat = 360 ;
variables:
        double time(time) ;
                time:standard_name = "time" ;
                time:long_name = "time" ;
                time:units = "days since 1860-1-1 00:00:00" ;
                time:calendar = "proleptic_gregorian" ;
                time:axis = "T" ;
        double lon(lon) ;
                lon:standard_name = "longitude" ;
                lon:long_name = "Longitude" ;
                lon:units = "degrees_east" ;
                lon:axis = "X" ;
        double lat(lat) ;
                lat:standard_name = "latitude" ;
                lat:long_name = "Latitude" ;
                lat:units = "degrees_north" ;
                lat:axis = "Y" ;
        float pr(time, lat, lon) ;
                pr:standard_name = "precipitation_flux" ;
                pr:long_name = "Precipitation" ;
                pr:units = "kg m-2 d-1" ;
                pr:_FillValue = 1.e+20f ;
                pr:missing_value = 1.e+20f ;

// global attributes:
                :CDI = "Climate Data Interface version 1.9.9rc1 (https://mpimet.mpg.de/cdi)" ;
                :Conventions = "CF-1.6" ;
                :title = "GSWP3 global meteorological forcing data bias-adjusted to W5E5 with ISIMIP3BASD v2.4.1 (1971-1978) and W5E5 global meteorological forcing data (1979-1980) merged for ISIMIP2a (info@isimip.org)" ;
                :version = "This version of the GSWP3-W5E5 dataset was published on 18 June 2020, see <https://www.isimip.org/gettingstarted/input-data-changelog/isimip2a-3a-gswp3-ewembi-and-gswp3-w5e5-data-replaced/>" ;
                :CDO = "Climate Data Operators version 1.9.9rc1 (https://mpimet.mpg.de/cdo)" ;
                :history = "Wed Jul 21 20:28:12 2021: ncatted -O -a units,pr,o,c,kg m-2 d-1 pr_day_gswp3-w5e5_1971_2016.nc\n",
                        "Wed Jul 21 20:06:48 2021: cdo mulc,86400 pr_gswp3-w5e5_1971_2016.nc pr_day_gswp3-w5e5_1971_2016.nc\n",
                        "Wed Jul 21 10:23:56 2021: cdo mergetime pr_gswp3-w5e5_1971_1980.nc4 pr_gswp3-w5e5_1981_1990.nc4 pr_gswp3-w5e5_1991_2000.nc4 pr_gswp3-w5e5_2001_2010.nc4 pr_gswp3-w5e5_2011_2016.nc4 pr_gswp3-w5e5_1971_2016.nc" ;
                :NCO = "netCDF Operators version 4.9.1 (Homepage = http://nco.sf.net, Code = http://github.com/nco/nco)" ;
}
netcdf tasmin_gswp3-w5e5_1971_2016 {
dimensions:
        time = UNLIMITED ; // (16802 currently)
        lon = 720 ;
        lat = 360 ;
variables:
        double time(time) ;
                time:standard_name = "time" ;
                time:long_name = "time" ;
                time:units = "days since 1860-1-1 00:00:00" ;
                time:calendar = "proleptic_gregorian" ;
                time:axis = "T" ;
        double lon(lon) ;
                lon:standard_name = "longitude" ;
                lon:long_name = "Longitude" ;
                lon:units = "degrees_east" ;
                lon:axis = "X" ;
        double lat(lat) ;
                lat:standard_name = "latitude" ;
                lat:long_name = "Latitude" ;
                lat:units = "degrees_north" ;
                lat:axis = "Y" ;
        float tasmin(time, lat, lon) ;
                tasmin:standard_name = "air_temperature" ;
                tasmin:long_name = "Daily Minimum Near-Surface Air Temperature" ;
                tasmin:units = "K" ;
                tasmin:_FillValue = 1.e+20f ;
                tasmin:missing_value = 1.e+20f ;

// global attributes:
                :CDI = "Climate Data Interface version 1.9.9rc1 (https://mpimet.mpg.de/cdi)" ;
                :Conventions = "CF-1.6" ;
                :history = "Wed Jul 21 09:51:06 2021: cdo mergetime tasmin_gswp3-w5e5_1971_1980.nc4 tasmin_gswp3-w5e5_1981_1990.nc4 tasmin_gswp3-w5e5_1991_2000.nc4 tasmin_gswp3-w5e5_2001_2010.nc4 tasmin_gswp3-w5e5_2011_2016.nc4 tasmin_gswp3-w5e5_1971_2016.nc" ;
                :title = "GSWP3 global meteorological forcing data bias-adjusted to W5E5 with ISIMIP3BASD v2.4.1 (1971-1978) and W5E5 global meteorological forcing data (1979-1980) merged for ISIMIP2a (info@isimip.org)" ;
                :version = "This version of the GSWP3-W5E5 dataset was published on 18 June 2020, see <https://www.isimip.org/gettingstarted/input-data-changelog/isimip2a-3a-gswp3-ewembi-and-gswp3-w5e5-data-replaced/>" ;
                :CDO = "Climate Data Operators version 1.9.9rc1 (https://mpimet.mpg.de/cdo)" ;
}
netcdf tasmax_gswp3-w5e5_1971_2016 {
dimensions:
        time = UNLIMITED ; // (16802 currently)
        lon = 720 ;
        lat = 360 ;
variables:
        double time(time) ;
                time:standard_name = "time" ;
                time:long_name = "time" ;
                time:units = "days since 1860-1-1 00:00:00" ;
                time:calendar = "proleptic_gregorian" ;
                time:axis = "T" ;
        double lon(lon) ;
                lon:standard_name = "longitude" ;
                lon:long_name = "Longitude" ;
                lon:units = "degrees_east" ;
                lon:axis = "X" ;
        double lat(lat) ;
                lat:standard_name = "latitude" ;
                lat:long_name = "Latitude" ;
                lat:units = "degrees_north" ;
                lat:axis = "Y" ;
        float tasmax(time, lat, lon) ;
                tasmax:standard_name = "air_temperature" ;
                tasmax:long_name = "Daily Maximum Near-Surface Air Temperature" ;
                tasmax:units = "K" ;
                tasmax:_FillValue = 1.e+20f ;
                tasmax:missing_value = 1.e+20f ;

// global attributes:
                :CDI = "Climate Data Interface version 1.9.9rc1 (https://mpimet.mpg.de/cdi)" ;
                :Conventions = "CF-1.6" ;
                :history = "Wed Jul 21 09:45:19 2021: cdo mergetime tasmax_gswp3-w5e5_1971_1980.nc4 tasmax_gswp3-w5e5_1981_1990.nc4 tasmax_gswp3-w5e5_1991_2000.nc4 tasmax_gswp3-w5e5_2001_2010.nc4 tasmax_gswp3-w5e5_2011_2016.nc4 tasmax_gswp3-w5e5_1971_2016.nc" ;
                :title = "GSWP3 global meteorological forcing data bias-adjusted to W5E5 with ISIMIP3BASD v2.4.1 (1971-1978) and W5E5 global meteorological forcing data (1979-1980) merged for ISIMIP2a (info@isimip.org)" ;
                :version = "This version of the GSWP3-W5E5 dataset was published on 18 June 2020, see <https://www.isimip.org/gettingstarted/input-data-changelog/isimip2a-3a-gswp3-ewembi-and-gswp3-w5e5-data-replaced/>" ;
                :CDO = "Climate Data Operators version 1.9.9rc1 (https://mpimet.mpg.de/cdo)" ;
}

Also, the only piece of code I changed from the original R script is the input: infiles= c("./www/input_gswp3/pr_day_gswp3-w5e5_1971_2016.nc","./www/input_gswp3/tasmax_gswp3-w5e5_1971_2016.nc","./www/input_gswp3/tasmin_gswp3-w5e5_1971_2016.nc")

I appreciate if anyone could provide some help.

Best,

Henrique

heroldn commented 2 years ago

Hi, try adjusting the "maxvals" variable in the wrapper script.

On 22/07/2021 05:11 Henrique Goulart ***@***.***> wrote:

First of all, thank you very much for this incredible tool.

I ran the test with the sample and everything worked well, but when testing with my three files, the following error appears:

Loading required package: PCICt
Loading required package: lmomco
Loading required package: parallel
Loading required package: ggplot2
# Package SPEI (1.7) loaded [try SPEINews()].
Error in ncdf4.helpers::get.cluster.worker.subsets(max.vals.millions *  :
  rows.per.slice >= 1 is not TRUE
Calls: create.indices.from.files -> <Anonymous> -> stopifnot
Execution halted

A quick view on the files themselves:

netcdf pr_day_gswp3-w5e5_1971_2016 {
dimensions:
        time = UNLIMITED ; // (16802 currently)
        lon = 720 ;
        lat = 360 ;
variables:
        double time(time) ;
                time:standard_name = "time" ;
                time:long_name = "time" ;
                time:units = "days since 1860-1-1 00:00:00" ;
                time:calendar = "proleptic_gregorian" ;
                time:axis = "T" ;
        double lon(lon) ;
                lon:standard_name = "longitude" ;
                lon:long_name = "Longitude" ;
                lon:units = "degrees_east" ;
                lon:axis = "X" ;
        double lat(lat) ;
                lat:standard_name = "latitude" ;
                lat:long_name = "Latitude" ;
                lat:units = "degrees_north" ;
                lat:axis = "Y" ;
        float pr(time, lat, lon) ;
                pr:standard_name = "precipitation_flux" ;
                pr:long_name = "Precipitation" ;
                pr:units = "kg m-2 d-1" ;
                pr:_FillValue = 1.e+20f ;
                pr:missing_value = 1.e+20f ;

// global attributes:
                :CDI = "Climate Data Interface version 1.9.9rc1 (https://mpimet.mpg.de/cdi)" ;
                :Conventions = "CF-1.6" ;
                :title = "GSWP3 global meteorological forcing data bias-adjusted to W5E5 with ISIMIP3BASD v2.4.1 (1971-1978) and W5E5 global meteorological forcing data (1979-1980) merged for ISIMIP2a ***@***.***)" ;
                :version = "This version of the GSWP3-W5E5 dataset was published on 18 June 2020, see <https://www.isimip.org/gettingstarted/input-data-changelog/isimip2a-3a-gswp3-ewembi-and-gswp3-w5e5-data-replaced/>" ;
                :CDO = "Climate Data Operators version 1.9.9rc1 (https://mpimet.mpg.de/cdo)" ;
                :history = "Wed Jul 21 20:28:12 2021: ncatted -O -a units,pr,o,c,kg m-2 d-1 pr_day_gswp3-w5e5_1971_2016.nc\n",
                        "Wed Jul 21 20:06:48 2021: cdo mulc,86400 pr_gswp3-w5e5_1971_2016.nc pr_day_gswp3-w5e5_1971_2016.nc\n",
                        "Wed Jul 21 10:23:56 2021: cdo mergetime pr_gswp3-w5e5_1971_1980.nc4 pr_gswp3-w5e5_1981_1990.nc4 pr_gswp3-w5e5_1991_2000.nc4 pr_gswp3-w5e5_2001_2010.nc4 pr_gswp3-w5e5_2011_2016.nc4 pr_gswp3-w5e5_1971_2016.nc" ;
                :NCO = "netCDF Operators version 4.9.1 (Homepage = http://nco.sf.net, Code = http://github.com/nco/nco)" ;
}

netcdf tasmin_gswp3-w5e5_1971_2016 {
dimensions:
        time = UNLIMITED ; // (16802 currently)
        lon = 720 ;
        lat = 360 ;
variables:
        double time(time) ;
                time:standard_name = "time" ;
                time:long_name = "time" ;
                time:units = "days since 1860-1-1 00:00:00" ;
                time:calendar = "proleptic_gregorian" ;
                time:axis = "T" ;
        double lon(lon) ;
                lon:standard_name = "longitude" ;
                lon:long_name = "Longitude" ;
                lon:units = "degrees_east" ;
                lon:axis = "X" ;
        double lat(lat) ;
                lat:standard_name = "latitude" ;
                lat:long_name = "Latitude" ;
                lat:units = "degrees_north" ;
                lat:axis = "Y" ;
        float tasmin(time, lat, lon) ;
                tasmin:standard_name = "air_temperature" ;
                tasmin:long_name = "Daily Minimum Near-Surface Air Temperature" ;
                tasmin:units = "K" ;
                tasmin:_FillValue = 1.e+20f ;
                tasmin:missing_value = 1.e+20f ;

// global attributes:
                :CDI = "Climate Data Interface version 1.9.9rc1 (https://mpimet.mpg.de/cdi)" ;
                :Conventions = "CF-1.6" ;
                :history = "Wed Jul 21 09:51:06 2021: cdo mergetime tasmin_gswp3-w5e5_1971_1980.nc4 tasmin_gswp3-w5e5_1981_1990.nc4 tasmin_gswp3-w5e5_1991_2000.nc4 tasmin_gswp3-w5e5_2001_2010.nc4 tasmin_gswp3-w5e5_2011_2016.nc4 tasmin_gswp3-w5e5_1971_2016.nc" ;
                :title = "GSWP3 global meteorological forcing data bias-adjusted to W5E5 with ISIMIP3BASD v2.4.1 (1971-1978) and W5E5 global meteorological forcing data (1979-1980) merged for ISIMIP2a ***@***.***)" ;
                :version = "This version of the GSWP3-W5E5 dataset was published on 18 June 2020, see <https://www.isimip.org/gettingstarted/input-data-changelog/isimip2a-3a-gswp3-ewembi-and-gswp3-w5e5-data-replaced/>" ;
                :CDO = "Climate Data Operators version 1.9.9rc1 (https://mpimet.mpg.de/cdo)" ;
}

Also, the only piece of code I changed from the original R script is the input: infiles= "./www/input_gswp3/tasmax_gswp3-w5e5_1971_2016.nc" #c("./www/input_gswp3/pr_day_gswp3-w5e5_1971_2016.nc","./www/input_gswp3/tasmax_gswp3-w5e5_1971_2016.nc","./www/input_gswp3/tasmin_gswp3-w5e5_1971_2016.nc") #"./www/input_gswp3/*"

I appreciate if anyone could provide some help.

Best,

Henrique

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub https://github.com/ARCCSS-extremes/climpact/issues/85 , or unsubscribe https://github.com/notifications/unsubscribe-auth/ACW3FAIQ2NRXU2KXFZN3JRTTY4LWZANCNFSM5AYSAVWA .
dumontgoulart commented 2 years ago

Hi Herold,

thank you for the answer. It helped a lot. Only now I noticed there was this commented in the script. If I may suggest, perhaps it would be good to have this also specified in the main page here in github.

Cheers,

Henrique

heroldn commented 2 years ago

Hi Henrique, noted. At some point we should look at the documentation again with fresher eyes (and more time).

Cheers

Nick.

On 22/07/2021 19:34 Henrique Goulart ***@***.***> wrote:

Hi Herold,

thank you for the answer. It helped a lot. Only now I noticed there was this commented in the script. If I may suggest, perhaps it would be good to have this also specified in the main page here in github.

Cheers,

Henrique

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/ARCCSS-extremes/climpact/issues/85#issuecomment-884779218 , or unsubscribe https://github.com/notifications/unsubscribe-auth/ACW3FAMCYFCJ42VZNRQLP63TY7Q3PANCNFSM5AYSAVWA .
dumontgoulart commented 2 years ago

May I ask you one additional thing: What does it change when we alter the 'maxvals'? I wonder why it was originally selected as 10, why would I change to 20 or 50? Is there any drawback, such as slower progress, with a larger maxvals?

Cheers,

henrique

heroldn commented 2 years ago

Sorry for the very late response to this. And sorry I don't have a good answer, but looking at the code (this is from climdex.pcic.ncdf) this relates to how small the chunks are that the grid is broken down into for processing. Quoting "The number of data values to process at one time (length of time dim number of values number of variables).".

Following parallel computing logic, there is always overhead in breaking things down into chunks. So having too many (small) chunks, while maybe good from a memory perspective, might lead to unnecessary overhead, but while having too few chunks (i.e. large chunk sizes) would result in less overhead but needing more memory. This hopefully is not an issue for most people unless you need to run the code often (e.g. a forecasting context).