Switch to clef

[DamienIrving]

It's unlikely that a symlink tree similar to the one created/maintained for CMIP5 will exist for CMIP6. As such, commands such as the following in shell scripts:

files_symlink="/g/data/ua6/DRSv3/CMIP5/${model}/${experiment}/mon/ocean/r1i1p1/so/latest/so_Omon_${model}_${experiment}_r1i1p1_??????-??????.nc"

will need to be replaced with:

files_clef=$(clef --local cmip5 --variable so --experiment ${experiment} --table Omon --format file -m ${model} | sort)

There is clef documentation (the README might be more up-to-date) and tutorials available.

[DamienIrving]

Also note that there are two levels of help information:

$ clef --help
Usage: clef [OPTIONS] COMMAND [ARGS]...

Options:
  --remote   returns only ESGF search results
  --local    returns only local files matching ESGF search
  --missing  returns only missing files matching ESGF search
  --request  send NCI request to download missing files matching ESGF search
  --debug    Show debug info
  --help     Show this message and exit.

Commands:
  cmip5  Search ESGF and local database for CMIP5 files Constraints can be...
  cmip6  Search ESGF and local database for CMIP6 files Constraints can be...
  ds     Search local database for non-ESGF datasets

$ clef cmip5 --help
Usage: clef cmip5 [OPTIONS] [QUERY]...

  Search ESGF and local database for CMIP5 files

  Constraints can be specified multiple times, in which case they are
  combined    using OR: -v tas -v tasmin will return anything matching
  variable = 'tas' or variable = 'tasmin'. The --latest flag will check ESGF
  for the latest version available, this is the default behaviour

Options:
  -e, --experiment x              CMIP5 experiment: piControl, rcp85, amip ...
  --experiment_family [Atmos-only|Control|Decadal|ESM|Historical|Idealized|Paleo|RCP]
                                  CMIP5 experiment family: Decadal, RCP ...
  -m, --model x                   CMIP5 model acronym: ACCESS1.3, MIROC5 ...
  -t, --table, --mip [Amon|Omon|OImon|LImon|Lmon|6hrPlev|6hrLev|3hr|Oclim|Oyr|aero|cfOff|cfSites|cfMon|cfDay|cf3hr|day|fx|grids]
  -v, --variable x                Variable name as shown in filanames: tas,
                                  pr, sic ...
  -en, --ensemble, --member TEXT  CMIP5 ensemble member: r#i#p#
  --frequency [mon|day|3hr|6hr|fx|yr|monClim|subhr]
  --realm [atmos|ocean|land|landIce|seaIce|aerosol|atmosChem|ocnBgchem]
  --institution TEXT              Modelling group institution id: MIROC, IPSL,
                                  MRI ...
  --cf_standard_name TEXT         CF variable standard_name, use instead of
                                  variable constraint
  --format [file|dataset]         Return output for datasets (default) or
                                  individual files
  --latest / --all-versions       Return only the latest version or all of
                                  them. Default: --latest
  --replica / --no-replica        Return both original files and replicas.
                                  Default: --no-replica
  --distrib / --no-distrib        Distribute search across all ESGF nodes.
                                  Default: --distrib
  --debug / --no-debug            Show debug output. Default: --no-debug
  --help                          Show this message and exit.

[DamienIrving]

Also, there are multiple versions. The development version isn't compatible with simultaneously using another conda environment, but this probably doesn't matter in a shell script because you can just be specific when you call python (i.e. call the python from the other environment):

$ module use /g/data3/hh5/public/modules
$ module load conda
$ source activate clef-test

Then there's the version pre-installed at NCI:

$ module use /g/data3/hh5/public/modules
$ module load conda/analysis3-unstable