Fine-grained logging configuration and consistency

[HT154]

One of the most important considerations in running production services is logging. It's common practice to aggregate Kubernetes container logs (either the entire container log from the host system or specific files via a sidecar container) for analysis and search with a tool like Splunk or ElasticSearch. Ambassador's logging is currently pretty sad: only logs to stdout/stderr, different components have different log formats, and some processes log without a datestamp altogether. All of this makes it more difficult than it should be to index Ambassador logs.

Ambassador should add configuration options for logging:

• Envoy access log format (#701) https://www.envoyproxy.io/docs/envoy/latest/configuration/access_log
• Log format for Python components (hot-restarter, kubewatch, diagd) using logging module (#441) https://docs.python.org/3/library/logging.html
• Prefix for all logs in entrypoint.sh (including date(1) formatting for timestamps)
• Log destination for all components, should allow writing each component's logs to separate files (at specified paths) in addition to stdout/stderr
• Ideal: a new LoggingService manifest for configuring direct log forwarding to Splunk HEC/ElasticSearch/etc.

Ambassador's default configuration should log everything with the same format (at very least with a consistent timestamp prefix, including time zone).

I've attempted to use the sidecar method with a Splunk forwarder container. I had to write my own Dockerfile that depends on Ambassador and adds a custom entrypoint to tee the logs into files:

Dockerfile

ARG VERSION
FROM quay.io/datawire/ambassador:${VERSION}

RUN apk --no-cache add bash

WORKDIR /ambassador
COPY my_entrypoint.sh .
RUN chmod 755 my_entrypoint.sh

ENTRYPOINT [ "./my_entrypoint.sh" ]

my_entrypoint.sh

#!/bin/bash

mkdir -p /logs

exec > >(tee -a /logs/ambassador_stdout.log)
exec 2> >(tee -a /logs/ambassador_stderr.log)

# variants with added timestamps, but this causes double timestamps on some logs???
# exec > >(awk '{ print strftime("%Y-%m-%d %H:%M:%S %z"), $0; fflush(); }' | tee -a /logs/ambassador_stdout.log)
# exec 2> >(awk '{ print strftime("%Y-%m-%d %H:%M:%S %z"), $0; fflush(); }' | tee -a /logs/ambassador_stderr.log)

exec ./entrypoint.sh

(Installing Bash is required to get >() process redirection)

[richarddli]

Thanks for the detailed feedback! Definitely something that makes sense and we need to work towards this in future releases.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[HT154]

Please don’t close this

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[HT154]

Don’t close this. Bots that mark issues stale are a terrible idea. The only thing they improve is the number of open issues, which is not a metric projects should be optimizing for.

[leosunmo]

I really need this as well in order for Ambassador to be "production ready" for us.

Ideally Ambassador would expose the Envoy config for Access Logs (JSON and Plain) as well as (or perhaps to start with) enable the gRPC Access Logging (https://www.envoyproxy.io/docs/envoy/latest/api-v2/config/accesslog/v2/als.proto) service from Envoy.

My ideal setup (and how Gloo approaches this) is to log the "control plane" logs to stdout (for kubectl logs etc.) and then stream the Access Logs separately to a file (that can be picked up by Filebeat or Fluent Bit etc.) or using Envoys gRPC ALS service.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[HT154]

Bump so this doesn’t get closed >:[

[slively]

pro-actively bumping this to keep it from going stale. I use ambassador for my day job and side job because it's served me so well. The logging is so far the only glaring issue I've encountered. This may even be something I could work on as a contribution. Thank you for the awesome work on this project.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[HT154]

boooo

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[HT154]

nope

[kthommandra]

definitely a priority item

[sebarys]

@richarddli Is currently possible to filter somehow access logs logged by envoy? I would like e.g. log only 5xx requests on production env, logging all of them would be not a good idea.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[HT154]

nope

[linbingdouzhe]

need this feature here..

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[HT154]

nooope

[krish512]

Infact, instead of envoy generating its own RequestID, can we use existing TraceID as generated by AWS ALBs so that it is easier to trace requests from cloud resources to applications

[clguo-tw]

definitely a priority item

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[slively]

Bump

[ronnathaniel]

stay open.

[cindymullins-dw]

A couple of updates on this issue: 1) the custom Envoy log format is already supported but not well addressed in our docs. Probably needs an update here. 2) ability to change logs from text to json is available. please note the Module must be named 'ambassador'. 3) to allow writing each component's logs to separate files could cause additional disk usage in Ambassador pods, so this is unlikely. May be enough if logs can be categorized by introducing labels and still push them to stdout/stderr 4) a logging service by Ambassador is currently achieved by this logging plugin written by @krish512 Krishna Modi, an Ambassador Community Advocate. Leaving in-built logging open as a feature request

[devoxel]

I ran into a few issues with logging recently with Emissary

• No support for Envoy log filters. Fine grained filters would be broadly useful.
• I've been working on a LogService to sidestep this issue, but there's no way to send the hostname of ambassador pod (that I can see). You can get the IP via the grpc stream context, so I'm stuck with that for now.

to allow writing each component's logs to separate files could cause additional disk usage in Ambassador pods, so this is unlikely

File based logging would probably require a manually defined sidecar or a shared volume with a DaemonSet, both of which likely have a high bar to configuration.