HTTP compression (e.g. Gzip encoding) support

[gberche-orange]

Issue

As an api user

• in order to optimize network bandwidth of api calls (especially over the internet)
• I need CC_NG to accept HTTP compression (such as gzip encoding)

Context

This enhancement request was tracked since 2015 on https://github.com/cloudfoundry/cc-api-v3-style-guide/issues/47 and postponed until CAPI V3 is out.

Reopening this issue since https://github.com/cloudfoundry/cc-api-v3-style-guide repo was archived on April 9th, 2021

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/177739041

The labels on this github issue will be updated when the story is started.

[sethboyles]

Looks like this was disabled in 2015: https://github.com/cloudfoundry/capi-release/commit/d20d9c913cb61ca9a047bb686ddc3d17f95a94f4

We'll have to dig into the specifics as to why it was disabled, if those issues are still relevant, and, if so, if we can avoid them with other configuration options (or some other way).

We'll also need to take time to understand BREACH attacks and if CCNG would be susceptible to them.

[gberche-orange]

thanks @sethboyles for looking into this enhancement suggestion. Seems cloudfoundry/capi-release@d20d9c9 was focussing solely on blobs. This issue was more generic and extending to api payloads (json).

[sethboyles]

whoops, you're totally right.

I tried this out on a bosh lite today by adding the gzip section to the nginx conf for creating a lot of ASGs (a notoriously slow endpoint). I had 2233 ASGs with 100 rules each.

 2022-07-22 21:35:14 ⛅️  ruby 3.1.2p20 capi-ws-6 in ~/workspace/capi-release
±  |develop → origin {1} U:1 ✗| → time curl http://api.oil-keeper.capi.land/v3/security_groups?per_page=5000 -H "Authorization: $(cf oauth-token)"  > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 41.4M    0 41.4M    0     0  3019k      0 --:--:--  0:00:14 --:--:-- 10.7M

real    0m14.879s
user    0m0.070s
sys     0m0.087s

 2022-07-22 21:34:53 ⛅️  ruby 3.1.2p20 capi-ws-6 in ~/workspace/capi-release
±  |develop → origin {1} U:1 ✗| → time curl http://api.oil-keeper.capi.land/v3/security_groups?per_page=5000 -H "Authorization: $(cf oauth-token)" --compressed > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  910k    0  910k    0     0  58122      0 --:--:--  0:00:16 --:--:--  237k

real    0m16.605s
user    0m0.114s
sys     0m0.071s

I wasn't finding a consistent difference in response times, but there is certainly less bandwidth being used. Though my example might be a bit extreme as my ASG rules were all duplicates of each other and maybe more easily compressed.

@stephanme @philippthun @FloThinksPi any thoughts on how effective a change like this would be in helping with CCNG scaling issues?

[FloThinksPi]

@sethboyles i think it wont drastically reduce load or execution times on the CC since the CC still outputs 40MBs of content. It may save some time for a client(depending on its internet connection. that otherwise needs to download 40MB.

Client<--- 900K ----> Routing Layer + Nginx<---- 40MB ---->Ruby Process

We never saw bandwidth saturations server side or in the routing layer due to CC responses. Yet sending less data trough public networks can never hurt and also the compression is done by nginx which is highly optimised.

This will increase nginx cpu load a bit then. However currently the CC Ruby process does not use all available cores. And when nginx would use all available cores in a load scenario, the CC Ruby process would be dead a 10x times anyway.

As the cient has to send the gzip header for compression(Accept-Encoding: gzip) does the CF CLI support that as the majority of api calls have the CLI as client ?