search

aws / aws-app-mesh-controller-for-k8s

A controller to help manage App Mesh resources for a Kubernetes cluster.
Apache License 2.0
187 stars 109 forks source link

[Feature Request] customize injected envoy container #473

Closed fincd-aws closed 3 years ago

fincd-aws commented 3 years ago

Is your feature request related to a problem?

  1. If I'm running my tracing agent as a DaemonSet or on the nodes, I want each envoy to send tracing to the local host, not a tracing.address=<k8s svc> that is round-robin-ed around the cluster.
  2. generic solution to requests like issues/289 with its specific annotation pull/326

Describe the solution you'd like It would be convenient to directly support --set tracing.address='$NODE_IP' in the helm chart (inventing a variable name here). This would imply that the tracing agent is already running as a node service or DaemonSet on the port provided, and set two parts:

  1. in the controller args:
        args:
...
        - --jaeger-address=$NODE_IP
  1. in the injected init container + envoy container:
        env:
        - name: NODE_IP
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP

This may also need an edit to the init container to ensure it resolves the variable rather than insert the literal $NODE_IP that it does today (see below).

Describe alternatives you've considered

Instead, extending the injector CRD proposal to include a free-form container Spec that replaces the envoy container that would normally be injected. This generic solution would be error-prone if it doesn't have a way to validate these container Spec before use.

What happens today

--set tracing.address='$NODE_IP' in the controller helm values makes the init container write an unusable/invalid envoy config file (with the $NODE_IP literal), and the envoy fails to start.

lydell-manganti-blake commented 3 years ago

It seems to be related to this https://github.com/aws/aws-app-mesh-controller-for-k8s/pull/426

You can use ref:status.hostIP

fincd-aws commented 3 years ago

Yes, that would solve the customer's problem! Please document that special-case of putting ref:status.hostIP in the Helm chart!

Solved by #426

lydell-manganti-blake commented 3 years ago

I've raised a PR to document this. I hope this gives it more visibility.

kalyanjangam-vungle commented 3 years ago

Hi all,

we have tried implementing this and is causing the below error.

helm command used to upgrade: helm upgrade -i appmesh-controller eks/appmesh-controller \ --namespace appmesh-system \ --set region=$AWS_REGION \ --set serviceAccount.create=false \ --set serviceAccount.name=appmesh-controller \ --set tracing.enabled=true \ --set tracing.provider=jaeger \ --set tracing.port=9080 \ --set tracing.address=ref:status.hostIP

ENVOY_TRACING_CFG_FILE is defined, skipping generating envoy tracing config. Starting Envoy. [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:305] initializing epoch 0 (base id=0, hot restart version=11.104) [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:307] statically linked extensions: [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.compression.compressor: envoy.compression.gzip.compressor [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.health_checkers: envoy.health_checkers.redis [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.filters.network: envoy.client_ssl_auth, envoy.echo, envoy.ext_authz, envoy.filters.network.client_ssl_auth, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.http_connection_manager, envoy.filters.network.kafka_broker, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.mysql_proxy, envoy.filters.network.postgres_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.rocketmq_proxy, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.thrift_proxy.transports: auto, framed, header, unframed [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.resolvers: envoy.ip [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.thrift_proxy.filters: envoy.filters.thrift.rate_limit, envoy.filters.thrift.router [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.resource_monitors: envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.udp_packet_writers: udp_default_writer, udp_gso_batch_writer [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.upstreams: envoy.filters.connection_pools.http.generic, envoy.filters.connection_pools.http.http, envoy.filters.connection_pools.http.tcp [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.retry_priorities: envoy.retry_priorities.previous_priorities [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.compression.decompressor: envoy.compression.gzip.decompressor [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.grpc_credentials: envoy.grpc_credentials.aws_iam, envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.tracers: envoy.dynamic.ot, envoy.lightstep, envoy.tracers.datadog, envoy.tracers.dynamic_ot, envoy.tracers.lightstep, envoy.tracers.opencensus, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.dubbo_proxy.protocols: dubbo [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.filters.http: envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.fault, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cdn_loop, envoy.filters.http.compressor, envoy.filters.http.cors, envoy.filters.http.csrf, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.dynamo, envoy.filters.http.ext_authz, envoy.filters.http.fault, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.gzip, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.jwt_authn, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.oauth, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.squash, envoy.filters.http.tap, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.gzip, envoy.health_check, envoy.http_dynamo_filter, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router, envoy.squash [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.udp_listeners: quiche_quic_listener, raw_udp_listener [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.udp_proxy [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.stats_sinks: envoy.dog_statsd, envoy.metrics_service, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.statsd, envoy.statsd [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.dubbo_proxy.filters: envoy.filters.dubbo.router [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.dynamic_forward_proxy, envoy.clusters.redis [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.tap, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, tls [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.http.cache: envoy.extensions.http.cache.simple [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.filters.listener: envoy.filters.listener.http_inspector, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.quic_server_codec: quiche [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.tap, envoy.transport_sockets.tls, raw_buffer, tls [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.quic_client_codec: quiche [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.dubbo_proxy.route_matchers: default [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.bootstrap: envoy.extensions.network.socket_interface.default_socket_interface [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.access_loggers: envoy.access_loggers.file, envoy.access_loggers.http_grpc, envoy.access_loggers.tcp_grpc, envoy.file_access_log, envoy.http_grpc_access_log, envoy.tcp_grpc_access_log [2021-06-11 02:16:08.252][1][info][main] [source/server/server.cc:309] envoy.dubbo_proxy.serializers: dubbo.hessian2 [2021-06-11 02:16:08.260][1][warning][misc] [source/common/protobuf/utility.cc:294] Configuration does not parse cleanly as v3. v2 configuration is deprecated and will be removed from Envoy at the start of Q1 2021: Unknown field in: {"admin":{"accessLogPath":"/tmp/envoy_admin_access.log","address":{"socketAddress":{"address":"0.0.0.0","portValue":9901}}},"layeredRuntime":{"layers":[{"name":"static_layer_0","staticLayer":{"envoy.deprecated_features:envoy.api.v2.Cluster.tls_context":true,"envoy.deprecated_features:envoy.api.v2.core.GrpcService.GoogleGrpc.CallCredentials.MetadataCredentialsFromPlugin.config":true,"envoy.deprecated_features:envoy.api.v2.auth.CertificateValidationContext.verify_subject_alt_name":true,"envoy.deprecated_features:envoy.config.metrics.v2.StatsSink.config":true,"envoy.deprecated_features:envoy.api.v2.route.RouteMatch.regex":true,"envoy.deprecated_features:envoy.api.v2.route.HeaderMatcher.regex_match":true,"envoy.deprecated_features:envoy.api.v2.Cluster.ORIGINAL_DST_LB":true,"envoy.deprecated_features:envoy.listener.tls_inspector":true,"envoy.deprecated_features:envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager.Tracing.operation_name":true,"envoy.deprecated_features:envoy.api.v2.core.HealthCheck.HttpHealthCheck.use_http2":true,"envoy.deprecated_features:envoy.api.v2.listener.Filter.config":true,"envoy.deprecated_features:envoy.reloadable_features.enable_deprecated_v2_api_warning":true,"envoy.deprecated_features:envoy.listener.original_dst":true,"envoy.deprecated_features:envoy.api.v2.Cluster.hosts":true}}]},"node":{"id":"mesh/xrp-test/virtualNode/meister-test_xrp","cluster":"mesh/xrp-test/virtualNode/meister-test_xrp"},"clusterManager":{"outlierDetection":{"eventLogPath":"/dev/stdout"}},"staticResources":{"clusters":[{"name":"jaeger","loadAssignment":{"endpoints":[{"lbEndpoints":[{"endpoint":{"address":{"socketAddress":{"address":"ref:status.hostIP","portValue":9080}}}}]}],"clusterName":"jaeger"},"connectTimeout":"1s","type":"STRICT_DNS"}]},"tracing":{"http":{"typedConfig":{"sharedSpanContext":false,"@type":"type.googleapis.com/envoy.config.trace.v2.ZipkinConfig","collectorEndpoint":"/api/v2/spans","collectorEndpointVersion":"HTTP_JSON","collectorCluster":"jaeger"},"name":"envoy.tracers.zipkin"}},"dynamicResources":{"adsConfig":{"apiType":"GRPC","grpcServices":[{"googleGrpc":{"statPrefix":"ads","callCredentials":[{"fromPlugin":{"name":"envoy.grpc_credentials.aws_iam","config":{"service_name":"appmesh","region":"us-west-2"}}}],"credentialsFactoryName":"envoy.grpc_credentials.aws_iam","channelCredentials":{"sslCredentials":{"rootCerts":{"filename":"/etc/pki/tls/cert.pem"}}},"targetUri":"appmesh-envoy-management.us-west-2.amazonaws.com:443"}}]},"cdsConfig":{"initialFetchTimeout":"0s","ads":{}},"ldsConfig":{"initialFetchTimeout":"0s","ads":{}}}} [2021-06-11 02:16:08.260][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.api.v2.core.GrpcService.GoogleGrpc.CallCredentials.MetadataCredentialsFromPlugin Using deprecated option 'envoy.api.v2.core.GrpcService.GoogleGrpc.CallCredentials.MetadataCredentialsFromPlugin.config' from file grpc_service.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override. [2021-06-11 02:16:08.261][1][info][main] [source/server/server.cc:325] HTTP header map info: [2021-06-11 02:16:08.261][1][warning][runtime] [source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size [2021-06-11 02:16:08.262][1][warning][runtime] [source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size [2021-06-11 02:16:08.262][1][warning][runtime] [source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size [2021-06-11 02:16:08.262][1][warning][runtime] [source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size [2021-06-11 02:16:08.262][1][info][main] [source/server/server.cc:328] request header map: 608 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-method,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-max-retries,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-proto,x-ot-span-context,x-request-id [2021-06-11 02:16:08.262][1][info][main] [source/server/server.cc:328] request trailer map: 128 bytes: [2021-06-11 02:16:08.262][1][info][main] [source/server/server.cc:328] response header map: 424 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id [2021-06-11 02:16:08.262][1][info][main] [source/server/server.cc:328] response trailer map: 152 bytes: grpc-message,grpc-status [2021-06-11 02:16:08.280][1][info][main] [source/server/server.cc:448] admin address: 0.0.0.0:9901 [2021-06-11 02:16:08.282][1][info][main] [source/server/server.cc:583] runtime: layers:

cc @fincd-aws