search

icenet-ai / icenet

The icenet library is a pip installable python package containing the commands and code you need to produce forecasts
MIT License
21 stars 7 forks source link

CEDA: ESGF DAP server errors #26

Closed JimCircadian closed 2 years ago

JimCircadian commented 2 years ago

Getting persistent DAP server errors from esgf.ceda.ac.uk, are we misusing this endpoint? 

DEBUG:root:Query: {'source_id': 'MRI-ESM2-0', 'member_id': 'r1i1p1f1', 'frequency': 'day', 'variable_id': 'siconca', 'table_id': 'SIday', 'grid_label': 'gn', 'experiment_id': 'ssp245', 'data_node': 'esgf.ceda.ac.uk'}
DEBUG:root:Found ssp245: ['https://esgf.ceda.ac.uk/thredds/dodsC/esg_cmip6/CMIP6/ScenarioMIP/MRI/MRI-ESM2-0/ssp245/r1i1p1f1/SIday/siconca/gn/v20210329/siconca_SIday_MRI-ESM2-0_ssp245_r1i1p1f1_gn_20150101-20641231.nc', 'https://esgf.ceda.ac.uk/thredds/dodsC/esg_cmip6/CMIP6/ScenarioMIP/MRI/MRI-ESM2-0/ssp245/r1i1p1f1/SIday/siconca/gn/v20210329/siconca_SIday_MRI-ESM2-0_ssp245_r1i1p1f1_gn_20650101-21001231.nc']
INFO:root:Found 5 siconca results from ESGF search
syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: <html^><head><title>500 Internal Server Error</title></head><body><center><h1>500 Internal Server Error</h1></center><hr><center>nginx</center></body></html>
ERROR:root:Error encountered: [Errno -70] NetCDF: DAP server error: b'https://esgf.ceda.ac.uk/thredds/dodsC/esg_cmip6/CMIP6/CMIP/MRI/MRI-ESM2-0/historical/r1i1p1f1/SIday/siconca/gn/v20210311/siconca_SIday_MRI-ESM2-0_historical_r1i1p1f1_gn_19000101-19491231.nc'
ERROR:root:Thread failure: [Errno 2] No such file or directory: b'/data/hpcdata/users/jambyr/icenet/green/data/cmip6.MRI-ESM2-0.r1i1p1f1/north/siconca/download.MRI-ESM2-0.r1i1p1f1.nc'
Traceback (most recent call last):
...

Discovery is via core nodes so a bit confusing:

INFO:root:Querying ESGF
DEBUG:root:ESGF search URL: https://esgf-node.llnl.gov/esg-search/search/?source_id=MRI-ESM2-0&member_id=r1i1p1f1&freque
ncy=day&variable_id=tas&table_id=day&grid_label=gn&experiment_id=historical&data_node=esgf.ceda.ac.uk&project=CMIP6&type
=File&latest=true&format=application%2Fsolr%2Bjson&offset=0
INFO:root:Querying ESGF
DEBUG:root:ESGF search URL: https://esgf-node.llnl.gov/esg-search/search/?source_id=MRI-ESM2-0&member_id=r1i1p1f1&freque
ncy=day&variable_id=ta&table_id=day&grid_label=gn&experiment_id=historical&data_node=esgf.ceda.ac.uk&project=CMIP6&type=
File&latest=true&format=application%2Fsolr%2Bjson&offset=0
INFO:root:Querying ESGF
DEBUG:root:ESGF search URL: https://esgf-node.llnl.gov/esg-search/search/?source_id=MRI-ESM2-0&member_id=r1i1p1f1&freque
ncy=day&variable_id=psl&table_id=day&grid_label=gn&experiment_id=historical&data_node=esgf.ceda.ac.uk&project=CMIP6&type
=File&latest=true&format=application%2Fsolr%2Bjson&offset=0
INFO:root:Querying ESGF
DEBUG:root:ESGF search URL: https://esgf-node.llnl.gov/esg-search/search/?source_id=MRI-ESM2-0&member_id=r1i1p1f1&freque
ncy=day&variable_id=tos&table_id=Oday&grid_label=gr&experiment_id=historical&data_node=esgf.ceda.ac.uk&project=CMIP6&typ
e=File&latest=true&format=application%2Fsolr%2Bjson&offset=0
DEBUG:root:id: CMIP6.CMIP.MRI.MRI-ESM2-0.historical.r1i1p1f1.Oday.tos.gr.v20191210.tos_Oday_MRI-ESM2-0_historical_r1i1p1
f1_gr_18500101-18991231.nc|esgf.ceda.ac.uk
JimCircadian commented 2 years ago

Attempting to use this again under dev testing for #45

JimCircadian commented 2 years ago

Some variables have come out, so I've re-enabled the node. These errors are transient and to be expected. Ultimately the mechanism for doing these lookups isn't ideal and too broadly selecting the endpoints, which is why I raised #43. Therefore this is worth closing and on a case-by-case basis, the -xs switch needs to be used (e.g. -xs esgf.ceda.ac.uk -xs esgf-data2.diasjp.net) to remove transiently down endpoints.

With the refactor for #43, we can make this more resilient, but only if the need warrants the development effort.

JimCircadian commented 2 years ago

It needs to be noted that ESGF nodes are justifiable offline, rendering some variables unavailable (in this case MRI-ESM2-0 r3i1p1f1 ssp245 hus:

Query: {'source_id': 'MRI-ESM2-0', 'member_id': 'r3i1p1f1', 'frequency': 'day', 'variabl
e_id': 'hus', 'table_id': 'day', 'grid_label': 'gn', 'experiment_id': 'ssp245', 'data_node': 'esgf-data2.diasjp.net'}
[05-08-22 10:52:55 :DEBUG   ] - Found ssp245: ['http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMI
P/MRI/MRI-ESM2-0/ssp245/r3i1p1f1/day/hus/gn/v20210830/hus_day_MRI-ESM2-0_ssp245_r3i1p1f1_gn_20150101-20241231.nc', 'http
://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MRI/MRI-ESM2-0/ssp245/r3i1p1f1/day/hus/gn/v2021083
0/hus_day_MRI-ESM2-0_ssp245_r3i1p1f1_gn_20250101-20301231.nc', 'http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/
CMIP6/ScenarioMIP/MRI/MRI-ESM2-0/ssp245/r3i1p1f1/day/hus/gn/v20210830/hus_day_MRI-ESM2-0_ssp245_r3i1p1f1_gn_20310101-204
01231.nc', 'http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MRI/MRI-ESM2-0/ssp245/r3i1p1f1/day
/hus/gn/v20210830/hus_day_MRI-ESM2-0_ssp245_r3i1p1f1_gn_20410101-20501231.nc', 'http://esgf-data2.diasjp.net/thredds/dod
sC/esg_dataroot/CMIP6/ScenarioMIP/MRI/MRI-ESM2-0/ssp245/r3i1p1f1/day/hus/gn/v20210830/hus_day_MRI-ESM2-0_ssp245_r3i1p1f1
_gn_20510101-20601231.nc', 'http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MRI/MRI-ESM2-0/ssp
245/r3i1p1f1/day/hus/gn/v20210830/hus_day_MRI-ESM2-0_ssp245_r3i1p1f1_gn_20610101-20701231.nc', 'http://esgf-data2.diasjp
.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MRI/MRI-ESM2-0/ssp245/r3i1p1f1/day/hus/gn/v20210830/hus_day_MRI-ESM2-0
_ssp245_r3i1p1f1_gn_20710101-20801231.nc', 'http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MR
I/MRI-ESM2-0/ssp245/r3i1p1f1/day/hus/gn/v20210830/hus_day_MRI-ESM2-0_ssp245_r3i1p1f1_gn_20810101-20901231.nc', 'http://e
sgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MRI/MRI-ESM2-0/ssp245/r3i1p1f1/day/hus/gn/v20210830/hu
s_day_MRI-ESM2-0_ssp245_r3i1p1f1_gn_20910101-21001231.nc']
[05-08-22 10:52:55 :INFO    ] - Found 16 hus results from ESGF search

...
RuntimeError: NetCDF: DAP server error

syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: <!DOCTYPE^ HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"><html><head><title>503 Service Temporarily Unavailable</title></head><body><h1>Service Temporarily Unavailable</h1><p>The server is temporarily unable to service yourrequest due to maintenance downtime or capacityproblems. Please try again later.</p></body></html>