search

cernopendata / opendata.cern.ch

Source code for the CERN Open Data portal
http://opendata.cern.ch/
GNU General Public License v2.0
665 stars 148 forks source link

HTTP 502 response on opendata.cern.ch/eos #2840

Open cbourjau opened 4 years ago

cbourjau commented 4 years ago

Hello,

Requesting files from opendata.cern.ch/eos returns a 502 response. opendata-dev.web.cern.ch/eos/ seems to be working as expected.

Examples: Not working: http://opendata.cern.ch/eos/opendata/alice/2010/LHC10h/000139038/ESD/0001/AliESDs.root Working: http://opendata-dev.web.cern.ch/eos/opendata/alice/2010/LHC10h/000139038/ESD/0001/AliESDs.root

This used to work earlier and is currently breaking the CI tests of https://github.com/cbourjau/alice-rs It is crucial for that project to have a simple http resource which allows requesting byte ranges (Accept-Ranges header set). Has there been an intentional change on where and how these files are hosted?

Best regards, Christian

tiborsimko commented 4 years ago

@cbourjau Thanks for reporting. I occasionally see the ERROR 502: Bad Gateway as well, however if you retry immediately, the connection usually works well the second time. So one quick solution for your CI pipeline could be to retry three times before giving up.

A better option would be to use XRootD protocol instead of HTTP protocol to access the file. This is because there are more front-end servers listening to the XRootD traffic when compared to the HTTP traffic for serving big files. Would this be an option?

Here are three example how to get the same file over HTTP and XRootD protocols:

$ # option 1: using HTTP from CERN Open Data portal
$ wget http://opendata.cern.ch/eos/opendata/alice/2010/LHC10h/000139038/ESD/0001/AliESDs.root

$ # option 2: using HTTPS from EOSPUBLIC
$ wget --no-check-certificate https://eospublichttp.cern.ch//eos/opendata/alice/2010/LHC10h/000139038/ESD/0001/AliESDs.root

$ # option 3: using XRootD from EOSPUBLIC
$ xrdcp root://eospublic.cern.ch//eos/opendata/alice/2010/LHC10h/000139038/ESD/0001/AliESDs.root .
cbourjau commented 4 years ago

Thanks for your response @tiborsimko ! XRootD is not an option for alice-rs for various reasons including complexity, weight, non-Rustness, and lack of wasm-compatibility (see also #2811 ). The latter also disqualifies using self-signed certificates (your second option). I guess retrying is an option as you pointed out. However, besides of the latency and complexity that would introduce I would guess that its potentially very inefficient on the server side to hammer it with 502 provoking requests?

I am pretty sure that this is a recent regression (~months). There were hickups in the past, but now its only about one in four requests that succeeds. Any chance of fixing the root of the problem upstream?