vkuznet
commented
3 years ago Here are full details of the flaw in DBS python server implementation. Let's start with example of using DBS client API:
cat t.py
#!/usr/bin/env python
import json
import uuid
from dbs.apis.dbsClient import DbsApi
uid = uuid.uuid4().time_mid
processing_version="%s" %(uid if (uid<9999) else uid%9999)
url="https://cmsweb-testbed.cern.ch/dbs/int/global/DBSWriter"
api = DbsApi(url=url)
data={'processing_version': processing_version, 'description':'this_is_a_test'}
print(json.dumps(data))
res = api.insertProcessingEra(data)
print(res)As you can see the code above will use a string value of processing_version. If we'll run this code we'll get the following:
python ./t.py
{"processing_version": "1846", "description": "this_is_a_test"}
NoneNow, we can look-up results in DBS via curl call (scurl here refers to curl command with provided grid certs):
# fetch the results
scurl https://cmsweb-testbed.cern.ch/dbs/int/global/DBSReader/processingeras?processing_version=1846
[{"description": "this_is_a_test", "create_by": "valya", "creation_date": 1632934742, "processing_version": 1846}]As you can see from above, even though we provided processing_version as string data-type the DBS client API is happy to inject it into DBS.
Now, let's confirm that it is an issue with DBS server and not DBS client. For that we'll verify that new value for processing era does not exists in DBS and inject it via curl call using JSON with string value of processing version.
# let 's check if value 1848 does not exists in DBS
scurl https://cmsweb-testbed.cern.ch/dbs/int/global/DBSReader/processingeras?processing_version=1848
[]
# insert new processing era
scurl -v -X POST -H "Content-type: application/json" -d '{"processing_version": "184"description": "this_is_a_test"}' https://cmsweb-testbed.cern.ch/dbs/int/global/DBSWriter/processingeras
...
< HTTP/1.1 200 OK
...
* Connection #0 to host cmsweb-testbed.cern.ch left intact
null
# and now we can see it again
scurl https://cmsweb-testbed.cern.ch/dbs/int/global/DBSReader/processingeras?processing_version=1848
[{"description": "this_is_a_test", "create_by": "valya", "creation_date": 1632935194, "processing_version": 1848}]With dbs2go implementation it is not possible, see the following error during the injection
scurl -v -X POST -H "Content-type: application/json" -d '{"processing_version": "1849", "description": "this_is_a_test"}' https://cmsweb-testbed.cern.ch/dbs2go-writer/processingeras
...
< HTTP/1.1 400 Bad Request
[{"api":"processingeras","error":"json: cannot unmarshal string into Go struct field ProcessingEras.processing_version of type int64","exception":400,"method":"POST","type":"HTTPError"}]The reported error says that we can't unmarshal (the marshal/unmarshal names used for encode/decode in Go) the provided JSON because ProcessingEras.processing_version attribute is type of int64.
Using this example I conclude that there are two kind of issues with DBS python implementation:
- the DBS client API does not properly validate data-types of injected fields
- the DBS server cast wrong data-type to the expected data-type, e.g. in this example string value "1846" to int value 1846
Therefore, I expect that due to issue one none of the Python clients check data-types of JSON request and may pass incorrectly wrong data-type to DBS server. Since DBS server is casting under the hood values we never saw this problem. But if this is the case and when we move to dbs2go the python client calls may fail if they pass wrong data-type.
amaltaro
belforte
Impact of the new feature I found through DBS unit tests that DBS client allows to pass wrong data-types to DBS server, mostly represent integer as string, e.g.
'123'which then is casted to int data-type in DBS server. This is not allowed in Go implementation since it follows strict data-type. The issue here is Python itself which does not enforce strict data-typing. As such a special care should be done on a client side to avoid sending wrong data-type data via HTTP requests.Is your feature request related to a problem? Please describe. Yes, in Go implementation it is impossible to send wrong data-type via HTTP request
Describe the solution you'd like All WMCore tools should ensure to send proper data-types in DBS injection requests.
Full API documentation can be found at: https://github.com/vkuznet/dbs2go/blob/master/docs/apis.md#post-apis
Describe alternatives you've considered None
Additional context You may refer to DBS#657 which demonstrate the issue and provide a fix for DBS unit test client.