Make ingressgateway access logs visible via `cf logs` [DaemonSet version]

[christianang]

Summary of changes

This change is to propagate ingressgateway access logs (RTR logs) to the cf logging system so the access logs show up in cf logs.

This change introduces a second fluentd daemonset, which is used to capture logs from each ingressgateway, transform those logs, and send them to the cf logging fluentd.

We aren't 100% sure this is the direction we want to go with regards to the second fluentd daemonset decision. We want to hear people's opinions or thoughts on this decision.

We see the following tradeoffs with the approach:

Pros:

• A second fluentd daemonset keeps the configuration out of the main fluentd config and keeps that clean abstraction boundary that exists. I.e the main fluentd daemonset focuses on generically capturing app logs and there isn't any specific configuration for individual components.

Cons:

• Keeping our fluentd image in sync with the cf-logging fluentd will cause toil. Currently our implementation has this image hardcoded to a random image digest of cf-k8s-logging.
• A second fluentd daemonset increases our overall resource requirements for the nodes running cf-for-k8s.

Additional Context

https://www.pivotaltracker.com/story/show/173568724

Acceptance Steps

As an App Developer When I run cf logs Then I can expect to see the ingress gateway-access logs in the following format: 2020-06-25T23:42:19.00+0000 [RTR/0] OUT <log>

cc @jenspinney

[ndhanushkodi]

One suggestion in slack that was brought up was to use fluentd sidecars for the ingressgateway pods, but as @tcdowney mentions here it would be difficult to tweak every Pod definition with overlays and maintain that longterm, especially if we allow folks to bring their own Istio.

[ndhanushkodi]

@mike1808 and I explored using a fluent-bit daemonset rather than a fluentd daemonset. The config for that exploration is here.

Since fluent-bit consumes less resources we wanted to see if it was an option to use that rather than a fluentd daemonset.

Here are the results for resource usage for fluent-bit vs fluentd under light load. Fluent-bit uses significantly less memory, but it still uses a significant enough amount of memory that its worth considering not deploying an extra daemonset.

NAME                                        CPU(cores)   MEMORY(bytes)
ingressgateway-fluent-bit-forwarder-6s8b6   3m           38Mi
ingressgateway-fluent-bit-forwarder-g4f5q   4m           39Mi
ingressgateway-fluent-bit-forwarder-r89dr   4m           39Mi
ingressgateway-fluent-bit-forwarder-t4xqf   4m           38Mi
ingressgateway-fluent-bit-forwarder-tdz9c   4m           40Mi
ingressgateway-fluentd-forwarder-5x28q      5m           114Mi
ingressgateway-fluentd-forwarder-jgqsg      5m           130Mi
ingressgateway-fluentd-forwarder-pltbr      5m           125Mi
ingressgateway-fluentd-forwarder-pm5bd      5m           104Mi
ingressgateway-fluentd-forwarder-wws6r      5m           117Mi

[ndhanushkodi]

We are currently waiting for results from an exploration by the logging team so we can have consensus on a path forward for this PR. They will contact the team via slack when they are ready :)

[kauana]

After talking with the logging folks, we decided to go with the sidecar version which was merged yesterday so we are closing this draft PR.