`test_real_search` fails on circleci with read timeout

[zklaus]

Describe the bug The test_real_search, introduced recently in #1217, fails on circleci with read timeout. This just may be a fluke, but even so, two thoughts:

• We might want to check network accessibility from circleci and possibly skip the test there or explore circleci configuration
• We might want to catch the exception and give better feedback to the user.

The test result can be inspected here: https://app.circleci.com/pipelines/github/ESMValGroup/ESMValCore/5713/workflows/c1f2d784-4702-4ec4-ba56-862f387e2636/jobs/27482?invite=true#step-103-159 the test is located here: https://github.com/ESMValGroup/ESMValCore/blob/43e6b16ed7f84adbf611f07d3c535e3a3b9a2791/tests/integration/esgf/test_search_download.py#L131

[bouweandela]

It seems more likely that the esgf index node is down than that there is a problem with the circleci network, because then the tests wouldn't run at all. We might want to fall back to another esgf search node automatically if the default one is not available.

[zklaus]

Sure, that happens often enough (with all index nodes, really). I think it would be good to handle the situation more gracefully and possibly to catch the exception and report better to the user.

[valeriupredoi]

ah bollox, I just opened the exact same issue 10min ago :rofl: Gonna file that one. I was suggesting in there we introduce a catch for the timeout error and make the test xfail rather than full on fail. Also, the stack trace output is large from these fails, maybe we can trim that too, or pipe it to a file?

[valeriupredoi]

here's a GA test fail example too :+1:

[remi-kazeroni]

Here is another example of a recent failure. It would be great to have some kind of a "second/third default node" in case of failures. Recently, I also realized the default node was down after starting a cylc suite to test all recipes.

Attention: @valeriupredoi (your question from #1512).