Mark-MengZhang commented 2 years ago

Dear contributor,

I have two questions. One is I would like to use CMIP6 historical dataset (e.g. ACCESS-ESM1-5) to calculate thresholds. The base period is 1961-1990. However, I have faced "dims [product xx] do not match the length of object [xx]" problem.

The sample data can work well in my computer. And the program without thresholds (file name: ACCESS-ESM1-5) also works well. The two files are as follows:

file name: ACCESS-ESM1-5

! /usr/bin/env Rscript

------------------------------------------------

This wrapper script calls the 'create.indices.from.files' function from the modified climdex.pcic.ncdf package

to calculate ETCCDI, ET-SCI and other indices, using data and parameters provided by the user.

Note even when using a threshold file, the base.range parameter must still be specified accurately.

------------------------------------------------

assign(".lib.loc", "/home/PUBLIC1/zhangm/R/site-library", envir = environment(.libPaths)) library(climdex.pcic.ncdf)

list of one to three input files. e.g. c("a.nc","b.nc","c.nc")

infiles=c("a.nc","b.nc","c.nc")

list of variable names according to above file(s)

vars=c(prec="pr", tmax="tasmax", tmin="tasmin")

output directory. Will be created if it does not exist.

outdir="/historical/climdex/ACCESS-ESM1-5"

Output filename format. Must use CMIP5 filename convention. i.e. "var_timeresolution_model_scenario_run_starttime-endtime.nc"

file.template="var_daily_ACCESS-ESM1-5_historical_NA_1850-2014.nc"

author data

author.data=list(institution="My University", institution_id="MU")

reference period

base.range=c(1961,1990)

number of cores to use, or FALSE for single core.

cores=FALSE

list of indices to calculate, or NULL to calculate all.

indices=NULL

input threshold file to use, or NULL for none.

thresholds.files=NULL

Directory where Climpact is stored. Use full pathname. Leave as NULL if you are running this script from the Climpact directory (where this script was initially stored).

root.dir="/climpact-master/"

#######################################################

Esoterics below, do not modify without a good reason.

definition used for Excess Heat Factor (EHF). "PA13" for Perkins and Alexander (2013), this is the default. "NF13" for Nairn and Fawcett (2013).

EHF_DEF = "PA13"

axis to split data on. For chunking up of grid, leave this.

axis.name="Y"

Number of data values to process at once. If you receive "Error: rows.per.slice >= 1 is not TRUE", try increasing this to 20. You might have a large grid.

maxvals=20

output compatible with FCLIMDEX. Leave this.

fclimdex.compatible=FALSE

Call the package.

create.indices.from.files(infiles,outdir,file.template,author.data,variable.name.map=vars,base.range=base.range,parallel=cores,axis.to.split.on=axis.name,climdex.vars.subset=indices,thresholds.files=thresholds.files,fclimdex.compatible=fclimdex.compatible,root.dir=root.dir, cluster.type="SOCK",ehfdef=EHF_DEF,max.vals.millions=maxvals,wsdin_n=5,csdin_n=5,hddheatn_n=18,cddcoldn_n=18,gddgrown_n=10,rxnday_n=7,rnnmm_n=30,ntxntn_n=3,ntxbntnb_n=3,project.lat2d.coords=TRUE, thresholds.name.map=c(tx05thresh="tx05thresh",tx10thresh="tx10thresh", tx50thresh="tx50thresh", tx90thresh="tx90thresh",tx95thresh="tx95thresh", tn05thresh="tn05thresh",tn10thresh="tn10thresh",tn50thresh="tn50thresh",tn90thresh="tn90thresh",tn95thresh="tn95thresh", tx90thresh_15days="tx90thresh_15days",tn90thresh_15days="tn90thresh_15days",tavg90thresh_15days="tavg90thresh_15days", tavg05thresh="tavg05thresh",tavg95thresh="tavg95thresh", txraw="txraw",tnraw="tnraw",precraw="precraw", r95thresh="r95thresh", r99thresh="r99thresh"))

file name: ACCESS-ESM1-5_thresholds

! /usr/bin/env Rscript

------------------------------------------------

This wrapper script calls the 'create.thresholds.from.file' function from the modified climdex.pcic.ncdf package

to create thresholds, using data and parameters provided by the user.

------------------------------------------------

assign(".lib.loc", "/home/PUBLIC1/zhangm/R/site-library", envir = environment(.libPaths)) library(climdex.pcic.ncdf)

list of one to three input files. e.g. c("a.nc","b.nc","c.nc")

input.files=c("a.nc","b.nc","c.nc")

list of variable names according to above file(s)

vars=c(prec="pr", tmax="tasmax", tmin="tasmin")

output file name

output.file="/historical/climdex/ACCESS-ESM1-5/thresholds_1961-1990.nc"

author data

author.data=list(institution="My University", institution_id="MU")

reference period

base.range=c(1961,1990)

number of cores to use (or FALSE)

cores=FALSE

print messages?

verbose=TRUE

Directory where Climpact is stored. Use full pathname. Leave as NULL if you are running this script from the Climpact directory (where this script was initially stored).

root.dir="/climpact-master/"

######################################

Do not modify without a good reason.

Number of data values to process at once. If you receive "Error: rows.per.slice >= 1 is not TRUE", try increasing this to 20. You might have a large grid.

maxvals=50

fclimdex.compatible=FALSE

create.thresholds.from.file(input.files,output.file,author.data,variable.name.map=vars,base.range=base.range,parallel=cores,verbose=verbose,fclimdex.compatible=fclimdex.compatible,root.dir=root.dir,max.vals.millions=maxvals)

Another question is the contributor once said "If your base period is 1981-2010 and your calculating indices on 1951-2014 then just remove the threshold file from your wrapper script (i.e. don't specify it). That does confuse climpact. The in-base-period thresholds will be calculated at runtime so you don't need to specify a thresholds file." in other issue. However, if the runtime of two files are same (e.g. 1850-2014), and I need to specify a threshold file based on one model to run another model. So I am wondering if this can work in climpact.

Thanks a lot!

heroldn commented 1 year ago

Hi, it's probably best if you can upload a few years of your data to test on.

As for the second question. You shouldn't be using thresholds from one model on a different model. If you are calculating indices on 5 models then you'll need 5 thresholds files. But, the thresholds file only needs to be created and specified if the base period is outside of the temporal range of your input data.

Mark-MengZhang commented 1 year ago

Hi heroldn, Thanks for your reply.

The data I used are all from CMIP6 official website. https://esgf-node.llnl.gov/search/cmip6 Due to the poor network transmission speed, I am not able to upload my data.

I think this problem may sometime happens. For example, If I want to compare the differences of TX10p among historical, hist-nat, hist-GHG, I should create threshold file from historical model data. Assume that the time period is 1961-1990. Then use this threshold file to calculate TX10p indices of hist-nat and hist-GHG.

I have used cdo "shifttime" command to solve this problem. cdo shifttime,-1000years input.nc output.nc It can change the time period of threshold file.

I have tested it and works well. If this method have any problems, please do not hesitate and let me know. Thanks again for your help.

Cheers, Mark

heroldn commented 1 year ago

Hi Mark, ah, I think I'm finally understanding what you're doing. You're not testing different time periods you're testing the same time period in different experiments (the 3 historical scenarios). Which explains why you need to shifttime. I've never done this before but it seems what you would need to do to get it working.

I don't recall what the hist-nat and hist-GHG scenarios are, but assumedly their boundary conditions are different and thus their climates too. It depends on your specific hypothesis that you're testing. If you use thresholds from historical in the hist-nat and hist-GHG experiments then you're seeing how often hot days/heatwaves occur in these alternative climates assuming the historical scenario thresholds (so you could compare differences in number of days between alternate histories).

On 15/08/2022 12:50 ACST Mark-MengZhang @.***> wrote:

Hi heroldn, Thanks for your reply.

The data I used are all from CMIP6 official website. https://esgf-node.llnl.gov/search/cmip6 Due to the poor network transmission speed, I am not able to upload my data.

I think this problem may sometime happens. For example, If I want to compare the differences of TX10p among historical, hist-nat, hist-GHG, I should create threshold file from historical model data. Assume that the time period is 1961-1990. Then use this threshold file to calculate TX10p indices of hist-nat and hist-GHG.

I have used cdo "shifttime" command to solve this problem. cdo shifttime,-1000years input.nc output.nc It can change the time period of threshold file.

I have tested it and works well. If this method have any problems, please do not hesitate and let me know. Thanks again for your help.

Cheers, Mark

— Reply to this email directly, view it on GitHub https://github.com/ARCCSS-extremes/climpact/issues/98#issuecomment-1214572144, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACW3FAMFY44I4ME54776VZDVZGZQFANCNFSM5YXAFFZQ. You are receiving this because you commented.Message ID: @.***>

Mark-MengZhang commented 1 year ago

Yes, I am testing the same time period in different experiments. I want to compare the differences of threshold indices between alternate histories.

ARCCSS-extremes / climpact

Get "dims [product xx] do not match the length of object [xx]" when calculating thresholds #98

! /usr/bin/env Rscript

------------------------------------------------

This wrapper script calls the 'create.indices.from.files' function from the modified climdex.pcic.ncdf package

to calculate ETCCDI, ET-SCI and other indices, using data and parameters provided by the user.

Note even when using a threshold file, the base.range parameter must still be specified accurately.

------------------------------------------------

list of one to three input files. e.g. c("a.nc","b.nc","c.nc")

list of variable names according to above file(s)

output directory. Will be created if it does not exist.

Output filename format. Must use CMIP5 filename convention. i.e. "var_timeresolution_model_scenario_run_starttime-endtime.nc"

author data

reference period

number of cores to use, or FALSE for single core.

list of indices to calculate, or NULL to calculate all.

input threshold file to use, or NULL for none.

Directory where Climpact is stored. Use full pathname. Leave as NULL if you are running this script from the Climpact directory (where this script was initially stored).

Esoterics below, do not modify without a good reason.

definition used for Excess Heat Factor (EHF). "PA13" for Perkins and Alexander (2013), this is the default. "NF13" for Nairn and Fawcett (2013).

axis to split data on. For chunking up of grid, leave this.

Number of data values to process at once. If you receive "Error: rows.per.slice >= 1 is not TRUE", try increasing this to 20. You might have a large grid.

output compatible with FCLIMDEX. Leave this.

Call the package.

! /usr/bin/env Rscript

------------------------------------------------

This wrapper script calls the 'create.thresholds.from.file' function from the modified climdex.pcic.ncdf package

to create thresholds, using data and parameters provided by the user.

------------------------------------------------

list of one to three input files. e.g. c("a.nc","b.nc","c.nc")

list of variable names according to above file(s)

output file name

author data

reference period

number of cores to use (or FALSE)

print messages?

Directory where Climpact is stored. Use full pathname. Leave as NULL if you are running this script from the Climpact directory (where this script was initially stored).

Do not modify without a good reason.

Number of data values to process at once. If you receive "Error: rows.per.slice >= 1 is not TRUE", try increasing this to 20. You might have a large grid.