search

dmwm / DBSClient

Apache License 2.0
3 stars 8 forks source link

Add support for gzip encoding in HTTP requests #50

Closed vkuznet closed 3 years ago

vkuznet commented 3 years ago

@yuyiguo , here is set of changes to allow DBSClient to use gzip encoding for HTTP POST requests, e.g. bulkblocks API. My original benchmark shows that usage of such encoding can save us 10-20 times on network, e.g. 180MB payload can be easily gzipped to 3MB before sending to server. Of course, the server itself should be aware of such encoding and handle properly such data. That's why I disable it by default (see useGzip=False in DbsApi constructor). But these changes provide a way to start using this encoding of data.

@amaltaro you can also use this as an example for other WMCore client's calls. And, it is related to https://github.com/dmwm/DBS/pull/648 and https://github.com/dmwm/WMCore/issues/10451 issues. I provided concise example of Cherrypy server implementation in my gist.

vkuznet commented 3 years ago

@yuyiguo it is a issue for our frontends though, if we can reduce payload on FE then FE will need less resources and process it faster. So, if you don't mind I'll merge it after I'll perform additional tests on BE servver.

yuyiguo commented 3 years ago

@vkuznet I don't mind to merge it at all.

vkuznet commented 3 years ago

@amaltaro it would be nice if you'll review it before you're going to vacation, such that I can merge and proceed.