How to access the quickest server?

[SHEN-Cheng]

Hi,

I am using accmip6 to downloading CMIP6 GCMs.

I found a file can't be accessed in the default server(https://handle-esgf.dkrz.de/lp/21.14100/dab45247-bb68-49db-8f32-7a3424a10170), but the is OK in other replicas and the speed is really quick(10MB/s). So I am wondering that can I select the server(default or other replicas) or is there possible to add a function to select the relative quick server to access?

Thanks

[TaufiqHassan]

Hi,

acccmip6 doesn't have the feature to select fastest server yet. By default, it accesses https://esgf-node.llnl.gov node. What I've added (use the clone of acccmip6) for now is to select a node (using -serv option) from the following list: (1) https://esgf-node.llnl.gov (2) https://esgf-node.ipsl.upmc.fr (3) https://esgf-data.dkrz.de (4) https://esgf-index1.ceda.ac.uk So, you can select a preferred node as such: python acccmip6/acccmip6.py -o S -m MIROC6 -f mon -e historical -v prc -serv https://esgf-index1.ceda.ac.uk

This is a temporary solution for now.

Taufiq

[TaufiqHassan]

Hello,

The latest version (5.3.0) includes both server selection (-serv) and data node selection (-n) options.

Checking out the data nodes using module: acccmip6 -o M -n show

Currently  34  nodes has outputs!

Available nodes:

['aims3.llnl.gov', 'cmip.bcc.cma.cn', 'cmip.dess.tsinghua.edu.cn', 'cmip.fio.org.cn', 'crd-esgf-drc.ec.gc.ca', 'dist.nmlab.snu.ac.kr', 'dpesgf03.nccs.nasa.gov', 'esg-cccr.tropmet.res.in', 'esg-dn1.nsc.liu.se', 'esg-dn2.nsc.liu.se', 'esg-dn3.nsc.liu.se', 'esg.camscma.cn', 'esg.lasg.ac.cn', 'esg1.umr-cnrm.fr', 'esgdata.gfdl.noaa.gov', 'esgf-cnr.hpc.cineca.it', 'esgf-data.csc.fi', 'esgf-data.ucar.edu', 'esgf-data1.llnl.gov', 'esgf-data2.diasjp.net', 'esgf-data2.llnl.gov', 'esgf-nimscmip6.apcc21.org', 'esgf-node2.cmcc.it', 'esgf.bsc.es', 'esgf.ceda.ac.uk', 'esgf.dwd.de', 'esgf.ichec.ie', 'esgf.nci.org.au', 'esgf.rcec.sinica.edu.tw', 'esgf3.dkrz.de', 'lapdat02-p.gfdl.noaa.gov', 'noresg.nird.sigma2.no', 'polaris.pknu.ac.kr', 'vesg.ipsl.upmc.fr']

Selecting nodes for data search: acccmip6 -o S -e historical -v abs550aer -f mon -n aims3.llnl.gov,esgf-data.ucar.edu,esgf3.dkrz.de

TIPS: Use the check (-c) argument to check your inputs.

 Currently available models based on your search:

 ['HadGEM3-GC31-MM', 'GFDL-CM4', 'CanESM5', 'IPSL-CM6A-LR', 'AWI-ESM-1-1-LR', 'MPI-ESM1-2-LR', 'MRI-ESM2-0', 'CESM2-WACCM-FV2', 'CanESM5-CanOE', 'HadGEM3-GC31-LL', 'MIROC-ES2L', 'INM-CM4-8', 'CESM2-WACCM', 'CNRM-ESM2-1', 'E3SM-1-0', 'UKESM1-0-LL', 'E3SM-1-1-ECA', 'CESM2-FV2', 'INM-CM5-0', 'MPI-ESM1-2-HR', 'MPI-ESM-1-2-HAM', 'CESM2', 'NorESM2-LM', 'E3SM-1-1', 'CNRM-CM6-1']

Currently available variables based on your search:

 ['abs550aer']

Currently available experiments based on your search:

 ['historical']

Number of files: 1072

Available realizations: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32]

The same can be applied to data downloads (-o D).

Best, Taufiq

[SHEN-Cheng]

Hi,

I am wondering what acccmip will do when the speed of downloading is too slow, will it do a loop in all the available nodes?

I recommend acccmip can do a loop in all the available nodes when speed is less than 100kb/s, but my download process will stop for an hour and then skip the file, the experience was not good as I need to run the same code again and select some nodes to try it one by one.

Other question still confuse me: different node seems contains different result in respond to the same requirement, like 10 model in esgf-data1.llnl.gov, 20 in another node (Just an example). How to make sure I can download all different models in a same command?

Wish the update can be added in the new version.