search

dmwm / DBSClient

Apache License 2.0
3 stars 8 forks source link

Proxy Error when calling client functions #78

Closed amanrique1 closed 11 months ago

amanrique1 commented 11 months ago

I'm using the dbs3-client on Python 3.8 and 3.9. Since last week, I've been getting 502 Proxy Error when calling any method on the test and production URL. The code I'm running has nothing special, it is the following:

from dbs.apis.dbsClient import DbsApi

url="https://cmsweb.cern.ch/dbs/prod/global/DBSWriter"
url="https://cmsweb-testbed.cern.ch/dbs2go-writer"
dbsApi = DbsApi(url=url)

file = '/store/results/susy/8TeV_T1T1_2BC_550_100/StoreResults-8TeV_T1T1_2BC_550_100_RECO_532p3_pPF-START53_V7A_PU_S10/8TeV_T1T1_2BC_550_100/USER/StoreResults-8TeV_T1T1_2BC_550_100_RECO_532p3_pPF-START53_V7A_PU_S10/0000/34C19CEE-4A32-E211-B83E-441EA171A696.root'
print(dbsApi.updateFileStatus(logical_file_name=file, is_file_valid=0, lost=0))

The error received is:

Traceback (most recent call last):
  File "/afs/cern.ch/user/a/amanriqu/.conda/envs/dbs3env/lib/python3.8/site-packages/dbs/apis/dbsClient.py", line 474, in __callServer
    self.http_response = method_func(self.url, method, params, data, request_headers)
  File "/afs/cern.ch/user/a/amanriqu/.conda/envs/dbs3env/lib/python3.8/site-packages/RestClient/RestApi.py", line 48, in put
    return http_request(self._curl)
  File "/afs/cern.ch/user/a/amanriqu/.conda/envs/dbs3env/lib/python3.8/site-packages/RestClient/RequestHandling/HTTPRequest.py", line 62, in __call__
    raise HTTPError(effective_url, http_code, http_response.msg, http_response.raw_header, http_response.body)
RestClient.ErrorHandling.RestClientExceptions.HTTPError: HTTP Error 502: Proxy Error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "lab.py", line 11, in <module>
    print(dbsApi.updateFileStatus(logical_file_name=file, is_file_valid=0, lost=0))
  File "/afs/cern.ch/user/a/amanriqu/.conda/envs/dbs3env/lib/python3.8/site-packages/dbs/apis/dbsClient.py", line 396, in wrapper
    return func(*args, **kwargs)
  File "/afs/cern.ch/user/a/amanriqu/.conda/envs/dbs3env/lib/python3.8/site-packages/dbs/apis/dbsClient.py", line 1981, in updateFileStatus
    return self.__callServer("files", params=kwargs, callmethod='PUT')
  File "/afs/cern.ch/user/a/amanriqu/.conda/envs/dbs3env/lib/python3.8/site-packages/dbs/apis/dbsClient.py", line 486, in __callServer
    self.__parseForException(data)
  File "/afs/cern.ch/user/a/amanriqu/.conda/envs/dbs3env/lib/python3.8/site-packages/dbs/apis/dbsClient.py", line 512, in __parseForException
    raise http_error
RestClient.ErrorHandling.RestClientExceptions.HTTPError: HTTP Error 502: 
URL=https://cmsweb-testbed.cern.ch/dbs2go-writer/files?logical_file_name=%2Fstore%2Fresults%2Fsusy%2F8TeV_T1T1_2BC_550_100%2FStoreResults-8TeV_T1T1_2BC_550_100_RECO_532p3_pPF-START53_V7A_PU_S10%2F8TeV_T1T1_2BC_550_100%2FUSER%2FStoreResults-8TeV_T1T1_2BC_550_100_RECO_532p3_pPF-START53_V7A_PU_S10%2F0000%2F34C19CEE-4A32-E211-B83E-441EA171A696.root&is_file_valid=0&lost=0
Code=502
Message=Proxy Error
Header=HTTP/1.1 502 Proxy Error
Date: Thu, 04 Jan 2024 16:58:48 GMT
Server: Apache
Content-Length: 472
Connection: close
Content-Type: text/html; charset=iso-8859-1

Body=<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>502 Proxy Error</title>
</head><body>
<h1>Proxy Error</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
The proxy server could not handle the request <em><a href="/auth/complete/dbs2go-writer/files">PUT&nbsp;/auth/complete/dbs2go-writer/files</a></em>.<p>
Reason: <strong>DNS lookup failure for: dbs2go-writer.dbs.svc.cluster.local</strong></p></p>
</body></html>
vkuznet commented 11 months ago

The issue is that you used https://cmsweb-testbed.cern.ch/dbs2go-writer URL which is only used for dev purposes when we test new dbs2go code. The cmsweb-testbed has official DBS instance, https://cmsweb-testbed.cern.ch/dbs/int/global/DBSWriter. Said that, you can use curl command to check any DBS instance, e.g. here is how you see its APIs:

curl -s -L -k --key ~/.globus/userkey.pem --cert ~/.globus/usercert.pem https://cmsweb-testbed.cern.ch/dbs/int/global/DBSWriter/apis
[{"api":"/dbs/int/global/DBSWriter/physicsgroups","methods":["GET","POST"]},{"api":"/dbs/int/global/DBSWriter/blockorigin","methods":["GET"]},{"api":"/dbs/int/global/DBSWriter/filechildren","methods":["GET"]},{"api":"/dbs/int/global/DBSWriter/blocks","methods":["GET","POST","PUT","GET"]},{"api":"/dbs/int/global/DBSWriter/fileparents","methods":["GET","POST","GET"]},{"api":"/dbs/int/global/DBSWriter/filesummaries","methods":["GET"]},{"api":"/dbs/int/global/DBSWriter/filelumis","methods":["GET","POST"]},{"api":"/dbs/int/global/DBSWriter/acquisitioneras_ci","methods":["GET"]},{"api":"/dbs/int/global/DBSWriter/fileparentsbylumi","methods":["POST","POST","GET"]},{"api":"/dbs/int/global/DBSWriter/dbstats","methods":["GET"]},{"api":"/dbs/int/global/DBSWriter/status","methods":["GET"]},{"api":"/dbs/int/global/DBSWriter/help","methods":["GET"]},{"api":"/dbs/int/global/DBSWriter/datatiers","methods":["GET","POST","GET"]},{"api":"/dbs/int/global/DBSWriter/datasets","methods":["GET","POST","PUT","GET"]},{"api":"/dbs/int/global/DBSWriter/parentDSTrio","methods":["GET"]},{"api":"/dbs/int/global/DBSWriter/acquisitioneras","methods":["GET","POST","PUT","GET"]},{"api":"/dbs/int/global/DBSWriter/outputconfigs","methods":["GET","POST","GET"]},{"api":"/dbs/int/global/DBSWriter/blockchildren","methods":["GET"]},{"api":"/dbs/int/global/DBSWriter/datasetparents","methods":["GET","GET"]},{"api":"/dbs/int/global/DBSWriter/fileArray","methods":["POST"]},{"api":"/dbs/int/global/DBSWriter/healthz","methods":["GET"]},{"api":"/dbs/int/global/DBSWriter/runs","methods":["GET"]},{"api":"/dbs/int/global/DBSWriter/blockdump","methods":["GET"]},{"api":"/dbs/int/global/DBSWriter/blockparents","methods":["GET","POST","GET"]},{"api":"/dbs/int/global/DBSWriter/datasetlist","methods":["POST"]},{"api":"/dbs/int/global/DBSWriter/errors","methods":["GET"]},{"api":"/dbs/int/global/DBSWriter/metrics","methods":["GET"]},{"api":"/dbs/int/global/DBSWriter/datasetchildren","methods":["GET"]},{"api":"/dbs/int/global/DBSWriter/serverinfo","methods":["GET"]},{"api":"/dbs/int/global/DBSWriter/blockTrio","methods":["GET"]},{"api":"/dbs/int/global/DBSWriter/files","methods":["GET","POST","PUT","GET"]},{"api":"/dbs/int/global/DBSWriter/primarydatasets","methods":["GET","POST","GET"]},{"api":"/dbs/int/global/DBSWriter/releaseversions","methods":["GET"]},{"api":"/dbs/int/global/DBSWriter/datatypes","methods":["GET"]},{"api":"/dbs/int/global/DBSWriter/processingeras","methods":["GET","POST","GET"]},{"api":"/dbs/int/global/DBSWriter/blocksummaries","methods":["GET"]},{"api":"/dbs/int/global/DBSWriter/bulkblocks","methods":["POST"]},{"api":"/dbs/int/global/DBSWriter/primarydstypes","methods":["GET"]},{"api":"/dbs/int/global/DBSWriter/datasetaccesstypes","methods":["GET","POST","GET"]},{"api":"/dbs/int/global/DBSWriter/runsummaries","methods":["GET"]},{"api":"/dbs/int/global/DBSWriter/apis","methods":["GET"]},{"api":"/dbs/int/global/DBSWriter/dummy","methods":["GET","POST"]},{"api":"/dbs/int/global/DBSWriter/","methods":["GET"]}]

And, offical testbed URLs are:

Moreover, you don't need per-se python client as everything in DBS is based on RESTful APIs which you can use via curl or any other HTTP library, see https://github.com/dmwm/dbs2go/blob/master/docs/Client.md.

amanrique1 commented 11 months ago

Thanks @vkuznet The other test URL worked