Strange, not consistent HTTP 500 error

[jgamper]

I am running a call to download one dataset using a docker image with synda (see below).

docker run -i -v $(pwd)/conf/credentials.conf:/root/sdt/conf/credentials.conf -v $(pwd)/data/ESGF/:/root/sdt/data/ synda get -d /root/sdt/data/ -f cmip5.output1.MPI-M.MPI-ESM-LR.decadal1995.mon.land.Lmon.r2i1p1.v20120529

Running this command once this morning I was able to download the dataset. But then running the same command a few minutes later I am getting the following erros:

Retrieving parameters from ESGF..

*** Error occured at 2020-07-17 08:40:43.331658 ***

==================
*   Error code   *
==================

SDNETUTI-002

=====================
*   Error message   *
=====================

HTTP query failed (url=http://esgf-data.dkrz.de/esg-search/search?limit=0&facets=*&type=Dataset&fields=*&format=application%2Fsolr%2Bjson,exception=HTTP Error 500: 500,timeout=60)

==================
*   Stacktrace   *
==================

  File "bin/synda", line 174, in <module>
    status=sdtiaction.actions[args.subcommand](args)
  File "/root/sdt/lib/sd/sdtiaction.py", line 206, in get
    import sdlogon, sdrfile, sddeferredafter, sddirectdownload, syndautils, humanize, sdconfig, os, sdconst, sdearlystreamutils
  File "/root/sdt/lib/sd/sdrfile.py", line 19, in <module>
    import sdquicksearch
  File "/root/sdt/lib/sd/sdquicksearch.py", line 32, in <module>
    import sdpipeline
  File "/root/sdt/lib/sd/sdpipeline.py", line 22, in <module>
    import sdquerypipeline
  File "/root/sdt/lib/sd/sdquerypipeline.py", line 36, in <module>
    import sdnearestpre
  File "/root/sdt/lib/sd/sdnearestpre.py", line 23, in <module>
    import sdremoteparam
  File "/root/sdt/lib/sd/sdremoteparam.py", line 25, in <module>
    import sddenormodel
  File "/root/sdt/lib/sd/sddenormodel.py", line 22, in <module>
    import sdparam
  File "/root/sdt/lib/sd/sdparam.py", line 306, in <module>
    sdcache.run(reload=True)
  File "/root/sdt/lib/sd/sdcache.py", line 38, in run
    parameters=get_parameters_from_searchapi(host,project)
  File "/root/sdt/lib/sd/sdcache.py", line 210, in get_parameters_from_searchapi
    d=sdremoteparam_light.run(facets_group={'type':['Dataset']},dry_run=dry_run,host=host)
  File "/root/sdt/lib/sd/sdremoteparam_light.py", line 76, in run
    params=sdnetutils.call_param_web_service(url,60)
  File "/root/sdt/lib/sd/sdnetutils.py", line 85, in call_param_web_service
    buf=HTTP_GET(url,timeout)
  File "/root/sdt/lib/sd/sdnetutils.py", line 130, in HTTP_GET
    raise SDException(errcode,errmsg)

What are the underlying causes to this ? And is there a way to anticipate and fix this?

[pjournou-ipsl]

I was not in the team in July 2020.

What I can say today it that the only Synda package that is maintained by the team can be installed with conda (see https://anaconda.org/ipsl/synda).

From July 2020, some new parameters have been added into the sdt.conf file. For instance, the async_http_timeout or direct_http_timeout parameters might have been able to help you to avoid such HTTP 500 error.