Incosistent output for block that does not exist

[amaltaro]

This is another inconsistency that I noticed while testing the production (python-based) vs testbed (go-based) servers, from a follow up from this WMCore issue: https://github.com/dmwm/WMCore/issues/10842#issuecomment-1033971622

In the python-based server, when we make a call for a block that does not exist in the database, an HTTPError exception is raised in the client. While testing the same call in testbed (go-based), the call actually goes through and the response is an empty list.

This can be reproduced/verified with something like:

dbs = DbsApi(dbsUrl)
dbs.listBlocks(block_name='/HighPileUp/Run2011A-v1/RAW#fabf118a-cbbf-11e0-80a9-003048caaace' + 'BLAH')

If we come to an agreement that this is the best way to deal with inexistent blocks, then a thoroughly investigation of WMCore needs to be performed to evaluate the impact of this.

[vkuznet]

@amaltaro it is issue with regexp in GO server and nothing to do with APIs behavior. If you paid attention to DBS Python server error(which I"m glad I fixed to include full message) it said :

DBS Server error: {'exception': 400, 'message': 'Invalid Input Data /HighPileU...: Not Match Required Format', 'type': 'HTTPError'}

So, it means that your block name is not valid regexp name, while in Go I think I missed that. But if you'll put really non-existing block name, like /HighPileUp/Run2011A-v1/BLA#fabf118a-cbbf-11e0-80a9-003048caaace, then both Py and GO will return the same empty list.

I'll fix in Go server and will report here once I'll deploy it again and re-test using your example.

[vkuznet]

Fixed in this commit https://github.com/dmwm/dbs2go/commit/6d0d5c05e270c5b27a5d77fcd820b1fbd9ca451b, deployed to testbed. Tested again with this example and now dbs2go return proper error

DBS Server error: [{'error': {'reason': 'DBSError Code:114 Description:DBS validation error when wrong pattern is provided Function:dbs.validator.strType Message:wrong length of block hash Error: validation error', 'message': 'not str type', 'function': 'dbs.Validate', 'code': 113}, 'http': {'method': 'GET', 'code': 400, 'timestamp': '2022-02-10 12:43:10.897727652 +0000 UTC m=+192.516705594', 'path': '/dbs2go/blocks?block_name=/HighPileUp/Run2011A-v1/RAW#fabf118a-cbbf-11e0-80a9-003048caaaceBLAH&detail=False', 'user_agent': 'DBSClient/Unknown/', 'x_forwarded_host': 'cmsweb-testbed.cern.ch', 'x_forwarded_for': '67.249.140.245', 'remote_addr': '188.184.72.217:55627'}, 'exception': 400, 'type': 'HTTPError', 'message': 'DBSError Code:113 Description:DBS validation error, e.g. input parameter does not match lexicon rules Function:dbs.Validate Message:not str type Error: nested DBSError Code:114 Description:DBS validation error when wrong pattern is provided Function:dbs.validator.strType Message:wrong length of block hash Error: validation error'}]
Traceback (most recent call last):
  File "/Users/vk/CMS/DMWM/GIT/WMCore/./test_dbsapi.py", line 24, in <module>
    res = api.listBlocks(block_name='/HighPileUp/Run2011A-v1/RAW#fabf118a-cbbf-11e0-80a9-003048caaace' + 'BLAH')
  File "/Users/vk/CMS/DMWM/GIT/DBSClient/src/python/dbs/apis/dbsClient.py", line 921, in listBlocks
    return self.__callServer("blocks", params=kwargs)
  File "/Users/vk/CMS/DMWM/GIT/DBSClient/src/python/dbs/apis/dbsClient.py", line 448, in __callServer
    self.__parseForException(http_error)
  File "/Users/vk/CMS/DMWM/GIT/DBSClient/src/python/dbs/apis/dbsClient.py", line 492, in __parseForException
    raise http_error
  File "/Users/vk/CMS/DMWM/GIT/DBSClient/src/python/dbs/apis/dbsClient.py", line 445, in __callServer
    self.http_response = method_func(self.url, method, params, data, request_headers)
  File "/Users/vk/CMS/DMWM/GIT/PycurlClient/src/python/RestClient/RestApi.py", line 36, in get
    return http_request(self._curl)
  File "/Users/vk/CMS/DMWM/GIT/PycurlClient/src/python/RestClient/RequestHandling/HTTPRequest.py", line 62, in __call__
    raise HTTPError(effective_url, http_code, http_response.msg, http_response.raw_header, http_response.body)
RestClient.ErrorHandling.RestClientExceptions.HTTPError: HTTP Error 400: Bad Request

Please see first line reporting the error. The new DBS error codes are in place and provide comprehensive description of the issue, i.e. DBSError Code:114 Description:DBS validation error when wrong pattern is provided Function:dbs.validator.strType Message:wrong length of block hash Error: validation error

The issue is resolved/ Please confirm and we may close it.

[amaltaro]

Thanks for the prompt fix, Valentin. I confirm that testbed is behaving consistently with production now, in what concerns invalid block name.

Regarding the error message, could you please clarify why we have 3 different codes in it?

DBS Server error: [{'error': {'reason': 'DBSError Code:114 Description:DBS validation error when wrong pattern is provided Function:dbs.validator.strType Message:wrong length of block hash Error: validation error', 'message': 'not str type', 'function': 'dbs.Validate', 'code': 113}, 'http': {'method': 'GET', 'code': 400, 'timestamp': '2022-02-10 12:43:10.897727652 +0000 UTC m=+192.516705594', 'path': '/dbs2go/blocks?block_name=/HighPileUp/Run2011A-v1/RAW#fabf118a-cbbf-11e0-80a9-003048caaaceBLAH&detail=False', 'user_agent': 'DBSClient/Unknown/', 'x_forwarded_host': 'cmsweb-testbed.cern.ch', 'x_forwarded_for': '67.249.140.245', 'remote_addr': '188.184.72.217:55627'}, 'exception': 400, 'type': 'HTTPError', 'message': 'DBSError Code:113 Description:DBS validation error, e.g. input parameter does not match lexicon rules Function:dbs.Validate Message:not str type Error: nested DBSError Code:114 Description:DBS validation error when wrong pattern is provided Function:dbs.validator.strType Message:wrong length of block hash Error: validation error'}]

reason key contains "Code:114", then the code key contains "113", all under the error key. The http nested dictionary also contains something that is apparently inconsistent (same diff codes).

[vkuznet]

ok, let's break down the reason message. It says

DBSError Code:114 Description:DBS validation error when wrong pattern is provided Function:dbs.validator.strType Message:wrong length of block hash Error: validation error

According to dbs error codes, see here we have

• code 114 PatternErrorCode // 114 pattern error
• it happens in Function dbs.validator.strType code base, which you can look-up here, the 'dbsis a module, validator is go code within this module andstrType` is an acttual function
• the message says wrong length of block hash which tells you the reason of failure
• and actual error is validation error Therefore, by looking at dbs/validator.go you can easily find the line which produce the error, line283

Now, due to nested function calls this internal error was happen in upstream code which call validator dbs.Validate, see line310. Because it calls specific validator the code at that place was ValidateErrorCode which is 113. Since Go does not have exceptions, I put errors in nested chain. In other words:

• upstream code calls validator and received error
• validator itself return actual error
• upstream code wrap it up in its error code.

The code 113 ValidationErrorCode is something which happens in validator, while the actual reason for this is code 114 PatternErrorCode. The validation may fail for different reasons, e.g. wrong pattern, wrong data type, etc. Therefore, everything is consistent and moreover more precise. We have validation error and its cause.

[vkuznet]

should we close this issue?

[amaltaro]

I am happy to see WMCore is not the only one stacking error messages :-D Yes, Valentin, from my side it can now be closed. Thanks for getting this fix in!