Enable envoy access logging per pod

[rvennam]

This is a feature/doc request to enable envoy access logging per pod. For example, enabling access logs for ingress gateway pod or user pod is vital for debugging many issues.

The current tasks requires users to enable it for the entire mesh and requires a rerunning the istio installer with an additional flag. https://istio.io/latest/docs/tasks/observability/logs/access-log/#enable-envoy-s-access-logging

Users do not necessarily have the ability or want to enable mesh wide settings to debug a single application issue.

[GregHanson]

Would it be worth posting an example EnvoyFilter for gateway and sidecar? For instance the following EnvoyFilter works for ingress gateway:

kubectl apply -n istio-system -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: test-filter
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
    patch:
      operation: MERGE
      value:
        config:
          access_log:
          - name: envoy.file_access_log
            typed_config:
              "@type": "type.googleapis.com/envoy.config.accesslog.v2.FileAccessLog"
              path: /dev/stdout
              format: "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% \"%DYNAMIC_METADATA(istio.mixer:status)%\" \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n"
EOF

[rvennam]

@esnible istioctl proxy-config log seems close but it only sets envoy logging level..not access log. Should this be extended to support access as well? https://istio.io/latest/docs/reference/commands/istioctl/#istioctl-proxy-config-log

[rvennam]

@GregHanson Your EnvoyFilter example works great and I think it's worth adding to the Istio documentation.

[uc-dang-tiki]

It dose not work on istio 1.7.3. Our cluster config disable accesslog, and only want to open accesslog for some workload. I try this envoyfilter (base on @GregHanson example), but it dose not work for me

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: test-enable-accesslog
spec:
  workloadSelector:
    labels:
      workload.user.cattle.io/workloadselector: deployment-platform-prod-prod-eris
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      listener:
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
    patch:
      operation: MERGE
      value:
        config:
          access_log:
          - name: envoy.access_loggers.file
            typed_config:
              "@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog"
              path: /dev/stdout
              format: "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% \"%DYNAMIC_METADATA(istio.mixer:status)%\" \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n"

[GregHanson]

@uc-dang-tiki your EnvoyFilter is missing a typed_config in your patch and the filterchain name is incorrect:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: test-enable-accesslog
spec:
  workloadSelector:
    labels:
      workload.user.cattle.io/workloadselector: deployment-platform-prod-prod-eris
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      listener:
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
    patch:
      operation: MERGE
      value:
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
          access_log:
          - name: envoy.access_loggers.file
            typed_config:
              "@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog"
              path: /dev/stdout
              format: "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% \"%DYNAMIC_METADATA(istio.mixer:status)%\" \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n"

Also, make sure the EnvoyFilter is created in the same namespace as your target workload. Also, the EnvoyFilter is only for HTTP traffic - TCP, TLS, and HTTPS traffic won't appear in the logs.

Always check to make sure istioctl proxy-status does not show STALE for the workload you are attempting to apply the EnvoyFilter to. If nothing is stale, you can check the istioctl proxy-config listener|route|cluster <pod_name>.<pod_namespace> -o json to confirm the right names and structures are used in the EnvoyFilter

[uc-dang-tiki]

Thanks you ! it work like a charm :) . But I also have to remove context

[PatrickDuncan]

Hey, it would great if this capability was added to the the localhost:15000 API in the istio-proxy. Example you can change the log level with it: curl -XPOST http://localhost:15000/logging?level=debug. We'd rather not create/delete Kube resources to change the access logging for a single Pod @GregHanson

[tariq1890]

@PatrickDuncan You would need to file a feature request to Envoy instead of Istio. The link you refer to is the envoy admin endpoint.

@GregHanson @esnible I do, however, think it's reasonable for istioctl to provide a way to enable access-logging. Isn't it supposed to be a wrapper over all the envoy internal workings? Right now istioctl seems to be 1:1 mapped with the envoy admin endpoint. I think it would be useful for the EnvoyFilter abstraction to be added as a feature to istioctl.

[esnible]

@tariq1890 The fact that @GregHanson needed 26 lines of break-glass YAML to show how to do this, but @uc-dang-tiki didn't have success using Greg's example without additional help, shows the current difficulty of this.

Greg's solution requires a WorkloadSelector, rather than a pod ID. Perhaps an istioctl syntax like this:

kubectl label pod httpbin-5b5bdcdf6b-vrdn4 use-custom-logformat-123=true
istioctl pc access-log set-format --workload use-custom-logformat-123=true --format "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% \"%DYNAMIC_METADATA(istio.mixer:status)%\" \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n"

The implementation of istioctl pc access-log set-format would be to create an EnvoyFilter.

Thoughts?

[tariq1890]

@esnible Sounds good.

I think we should have a default format. Ideally, I wouldn't want to specify a wordy format when I look to enable access logging

I was thinking something like this

istioctl pc access-log -l app=foo

We can just use the existing labels instead of adding a new one

[esnible]

@tariq1890 Would it make sense to have both a default format, or default short and long formats, plus custom?

Users could say istioctl pc access log ... -verbosity=high or istioctl pc access log ... -custom="..."?

[olliezhu]

Sorry in advance for jumping in this thread with another troubleshooting question. I tried to implement @GregHanson's suggestion but was not able to reproduce per-pod access logging. (I'm running Istio 1.5.10.)

$ kubectl apply --namespace myspace -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: test-filter
spec:
  workloadSelector:
    labels:
      myspace-service: ollies-service
      myspace-service-env: sandbox
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
    patch:
      operation: MERGE
      value:
        config:
          access_log:
          - name: envoy.file_access_log
            typed_config:
              "@type": "type.googleapis.com/envoy.config.accesslog.v2.FileAccessLog"
              path: /dev/stdout
              format: "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% \"%DYNAMIC_METADATA(istio.mixer:status)%\" \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n"
EOF

For context, the pod I'd like to target contains metadata labels that include

metadata:
  labels:
    myspace-role: service-instance
    myspace-service: ollies-service
    myspace-service-env: sandbox
    myspace-version: git_196a3545
    security.istio.io/tlsMode: istio
    service.istio.io/canonical-name: ollies-service
    ...

I ran wget from this pod

$ kubectl exec --namespace myspace "$OS_POD" --container ollies-service -- wget example.com

and looked for any GET requests to example.com in the logs but could not find any.

$ kubectl logs --namespace myspace --selector myspace-service=ollies-service --container=ollies-service
$ kubectl logs --namespace myspace --selector myspace-service=ollies-service --container=istio-proxy

Also, a meta-ask: As a newbie to Istio, I'd appreciate some guidance on how to try to troubleshoot these kinds of issues myself. Currently my best bet is to create the EnvoyFilter as described and cross my fingers that it works as I expect it, and I have no visibility into what exactly is not working as expected and why.

[esnible]

@olliezhu for troubleshooting, the first thing I do is istioctl ps to see if any of the pods aren't SYNCED. If they are not, it usually means Envoy rejected what Pilot gave it.

The next thing is istioctl dashboard envoy <mypod>, and click on config_dump. That brings up Envoy's config. I then use the browser's "Find" to search that giant JSON config for something in my EnvoyFilter added, to see if Pilot pushed the change to Envoy.

[GregHanson]

@olliezhu wget should generate an access log entry. For example, just using the provided samples/sleep/sleep.yaml I was able to produce the following when targeting httpbin.org:

$ kubectl exec "$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})" -c sleep -- wget httpbin.org
...
$ kubectl logs "$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})" -c istio-proxy | tail
... 
[2021-01-22T14:03:58.713Z] "GET / HTTP/1.1" 200 - "-" 0 9593 71 70 "-" "Wget" "cfeb011b-67e4-966b-befd-566678c31c14" "httpbin.org" "3.211.1.78:80" PassthroughCluster 172.30.251.151:34732 3.211.1.78:80 172.30.251.151:34730 - allow_any

If you are not seeing the desired EnvoyFilter taking effect, first check istioctl proxy-status to ensure there are no STALE LDS/RDS/CDS results. If everything shows SYNCED, next step is to check the config envoy has received. Access logs are configured at the listener level in Envoy

$ istioctl proxy-config listener <ollies_service_pod_name>.<ollies_service_ns> -o json | grep path -B 1
...
                    "@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog",
                    "path": "/dev/stdout",

[olliezhu]

@esnible @GregHanson Thanks for the suggestions. I searched my pod's config_dump and could not find any configs merged from the EnvoyFilter (e.g. "/dev/stdout"). Also ran istioctl proxy-config listener ollies-service-748d4749d5-vmv7l.myspace -o json | grep path -B 1 (same as the pod name appears in istioctl proxy-status) but there was no output.

My environment has a large number of pods, and everything was synced except for a handful with RDS NOT SENT (unrelated to ollies-service).

[esnible]

@olliezhu There are two likely causes for your EnvoyFilter failure. It could be the Istio workloadSelector is not correct, or the configPatches is not correct.

Your EnvoyFilter should select the same pods as kubectl get pods --selector myspace-service=ollies-service --selector myspace-service-env=sandbox. Try that interactively, see the pod you are testing is part of the result. Also, can you confirm your EnvoyFilter and pods are in the same Kubernetes namespace?

I tried the same thing, and it also didn't work for me. @GregHanson can you troubleshoot this? I notice that @olliezhu used context: GATEWAY and I tried with and without that? I tested this with Istio Master, samples/httpbin/httpbin.yaml, and

  workloadSelector:
    labels:
      app: httpbin
      version: v1

I verified that I couldn't see @olliezhu 's format on the config_dump. I verified that Istiod logs a push to both my IngressGateway and httpbin pods. I verified my httpbin pod logs nothing when the push happens. The following produced no output:

$ istioctl pc listener deployment/httpbin -o json > /tmp/before.json
$ kubectl apply -f /tmp/issue7613.yaml                     
envoyfilter.networking.istio.io/test-filter created
$ istioctl pc listener deployment/httpbin -o json > /tmp/after.json 
$ diff /tmp/before.json /tmp/after.json

@istio/wg-user-experience-maintainers @istio/wg-policies-and-telemetry-maintainers Perhaps istioctl needs a lint-like tool that does the above steps non-destructively?

[olliezhu]

@esnible I amended the selector to just myspace-service=ollies-service to make this simpler, and because

$ kubectl get pods --namespace myspace --selector myspace-service=ollies-service --selector myspace-service-env=sandbox | wc -l
     707
$ kubectl get pods --namespace myspace --selector myspace-service=ollies-service
NAME                              READY   STATUS    RESTARTS   AGE
ollies-service-748d4749d5-vmv7l   3/3     Running   0          4d18h

I also confirmed that the pod and the EnvoyFilter are in the same k8s namespace myspace.

[GregHanson]

@olliezhu are what type of workload/pod are you trying to enable access logs for? Currently you have context: GATEWAY - did you want to enable access logs on your ingress gateway or a service pod with istio proxy sidecar injected on it? Your provided EnvoyFilter works as expected if I use context: ANY (acceptable contexts are defined here)

[olliezhu]

@GregHanson I'm trying to enable access logs on a service pod with an Istio proxy sidecar injected on it. I updated the EnvoyFilter to use context: ANY and now it seem to be logging additional stuff (yes!), but not what I expected to see -- mostly config warnings.

$ kubectl logs --namespace myspace "$OS_POD" --container=istio-proxy | tail
[Envoy (Epoch 0)] [2021-01-25 18:50:10.640][22][warning][config] [external/envoy/source/server/listener_impl.cc:196] Listener connection limit runtime key envoy.resource_limits.listener.10.101.22.111_8080.connection_limit is empty. There are currently no limitations on the number of accepted connections for listener 10.101.22.111_8080.
...
[Envoy (Epoch 0)] [2021-01-25 18:50:10.647][22][warning][config] [external/envoy/source/server/listener_impl.cc:196] Listener connection limit runtime key envoy.resource_limits.listener.virtualInbound.connection_limit is empty. There are currently no limitations on the number of accepted connections for listener virtualInbound.
[Envoy (Epoch 0)] [2021-01-25 18:50:10.720][22][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:83] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
...

I can confirm that Envoy has received the config:

$ istioctl proxy-config listener ollies-service-748d4749d5-vmv7l.myspace -o json | grep path -B 1
                                        "@type": "type.googleapis.com/envoy.config.accesslog.v2.FileAccessLog",
                                        "path": "/dev/stdout",
...

[GregHanson]

@olliezhu in the Istio-proxy container, there can be 3 different types of logs. There are

1. the access logs which you have configured with the EnvoyFilter
2. the logs from the Envoy process itself
3. the logs from the istio pilot-agent process

Envoy process log verbosity can be controlled at install time (.Values.global.proxy.logLevel) or dynamically via istioctl proxy-config log <pod_name>.<pod> --level <level> (the istioctl method will not persist after pod restart). The pilot-agent log level can be configured at install/inject by setting the appropriate flag (--log_output_level in doc here) or with install option Values.global.logging.level

[esnible]

@howardjohn suggests a --dry-run type command that shows the changes an EnvoyFilter would make to an Envoy config.

[olliezhu]

@GregHanson Not able to check log verbosity at the moment (for reasons outside the scope of this thread) but will do so when I can.

On a related note, could I set up access logging scoped to a single namespace rather than globally or per pod? Could this also be done with an EnvoyFilter, but with the namespace specified and no workloadSelector? e.g., to set up access logging on all pods in the myspace namespace:

$ kubectl apply --namespace myspace -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: test-filter
spec:
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
    patch:
      operation: MERGE
      value:
        config:
          access_log:
          - name: envoy.file_access_log
            typed_config:
              "@type": "type.googleapis.com/envoy.config.accesslog.v2.FileAccessLog"
              path: /dev/stdout
              format: "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% \"%DYNAMIC_METADATA(istio.mixer:status)%\" \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n"
EOF

[GregHanson]

@olliezhu you should be able to configure based on which namespace the EnvoyFilter is created in. Per the docs:

workloadSelector : Criteria used to select the specific set of pods/VMs on which this patch configuration should be applied. If omitted, the set of patches in this configuration will be applied to all workload instances in the same namespace. If omitted, the EnvoyFilter patches will be applied to all workloads in the same namespace. If the EnvoyFilter is present in the config root namespace, it will be applied to all applicable workloads in any namespace.

[Gokul-Radhakrishnan]

@GregHanson I am trying other way around, We have enabled envoy access logs globally and I want to disable envoy access log for selected pods. I am using Istio version 1.8.3.

I can merge the new fields to the access logs using below config (using patch operation MERGE)

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: test-enable-accesslog
  namespace: demo
spec:
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      listener:
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
    patch:
      operation: MERGE
      value:
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
          access_log:
          - name: envoy.access_loggers.file
            typed_config:
              "@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog"
              path: "/dev/stdout"
              format: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %DURATION%\n"

I tried using REMOVE and REPLACE (to disable access logging) operations but it is not working

Please let me know if you have any idea on this

[craigbox]

Hit this problem today. I personally think if you have to know what an EnvoyFilter is, Istio has failed you.

I love the idea of istioctl proxy-config access-log --enable pod/name-ab12c as target complexity.

[saltbo]

Would it be worth posting an example EnvoyFilter for gateway and sidecar? For instance the following EnvoyFilter works for ingress gateway:

kubectl apply -n istio-system -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: test-filter
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
    patch:
      operation: MERGE
      value:
        config:
          access_log:
          - name: envoy.file_access_log
            typed_config:
              "@type": "type.googleapis.com/envoy.config.accesslog.v2.FileAccessLog"
              path: /dev/stdout
              format: "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% \"%DYNAMIC_METADATA(istio.mixer:status)%\" \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n"
EOF

Warning: Envoy filter: unknown field "config" in envoy.config.listener.v3.Filter

client version: 1.12.1 control plane version: 1.12.1 data plane version: 1.12.1 (4 proxies)

[saltbo]

Hit this problem today. I personally think if you have to know what an EnvoyFilter is, Istio has failed you.

I love the idea of istioctl proxy-config access-log --enable pod/name-ab12c as target complexity.

Error: unknown flag: --enable

client version: 1.12.1 control plane version: 1.12.1 data plane version: 1.12.1 (4 proxies)

[GregHanson]

@saltbo the EnvoyFilter you referenced is no longer valid for istio version 1.8 and newer after moving to Envoy's V3 API's. You can find an up-to-date example here: https://github.com/istio/istio/wiki/EnvoyFilter-Samples#tracing-and-access-logging

[douglas-reid]

Starting with 1.12, enabling access logging per pod should be possible via the Telemetry API. It is early days for the API, but the goal is to cover use cases like the original request here. Some initial documentation is provided: https://istio.io/latest/docs/reference/config/telemetry/#AccessLogging.

Example:

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: frontend-logging
  namespace: foo
spec:
  selector:
    matchLabels:
      service.istio.io/canonical-name: frontend
  accessLogging:
  - providers:
     - name: envoy

[saltbo]

@douglas-reid can wei set the accessLogFile by Telemetry?

[zirain]

@douglas-reid can wei set the accessLogFile by Telemetry?

you can not override accessLogFile directly by Telemetry, so far you can setup multi provider in MeshConfig, use Telemetry API apply to different pods.

[saltbo]

Starting with 1.12, set access logfile per pod:

accessLogFile: /dev/stdout
defaultConfig:
  discoveryAddress: istiod.istio-system.svc:15012
  proxyMetadata: {}
  tracing:
    zipkin:
      address: zipkin.istio-system:9411
extensionProviders:
  - name: logfile
    envoyFileAccessLog:
      path: "/data/logs/istio.log"
enablePrometheusMerge: true
rootNamespace: istio-system
trustDomain: cluster.local

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: frontend-logging
spec:
  selector:
    matchLabels:
      app: httpbin
  accessLogging:
    - providers:
        - name: logfile

[pitiwari]

@saltbo i am trying to do similar change as above where i am using provider for access logging but its not working. Is this feature supported on istio version below

istioctl version client version: 1.11.4 control plane version: 1.12.2 data plane version: 1.12.2 (7 proxies), 1.11.4 (8 proxies)

below are the changes

meshconfig change:

      extensionProviders:
      - name: "log"
        envoyFileAccessLog:
          path: /dev/stdout
          logFormat:
          text: "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS% \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME% \"%REQ(test-header)%\"\n"

kind: Telemetry
metadata:
  name:test
  namespace: test
spec:
  accessLogging:
  - providers:
     - name: "log"```

[saltbo]

logFormat is supported on 1.13, you can retry

[gudipudipradeep]

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  selector:
    matchLabels:
      app: sps-www-ms-sss-app
  accessLogging:
    - providers:
      - name: envoy

This configuration is not giving the envoy logs of http traffic for specific deployment matchLabels

[manojtr]

@olliezhu in the Istio-proxy container, there can be 3 different types of logs. There are

1. the access logs which you have configured with the EnvoyFilter
2. the logs from the Envoy process itself
3. the logs from the istio pilot-agent process

Envoy process log verbosity can be controlled at install time (.Values.global.proxy.logLevel) or dynamically via istioctl proxy-config log <pod_name>.<pod> --level <level> (the istioctl method will not persist after pod restart). The pilot-agent log level can be configured at install/inject by setting the appropriate flag (--log_output_level in doc here) or with install option Values.global.logging.level

@GregHanson how do we enable/change pilot-agent log level in a running pod with envoy proxy as a sidecar? What I am trying to get is to log the rewritten liveness and readiness probes.