search

TritonDataCenter / smartos-live

For more information, please see http://smartos.org/ For any questions that aren't answered there, please join the SmartOS discussion list: https://smartos.topicbox.com/groups/smartos-discuss
1.57k stars 245 forks source link

Envoy Proxy Seg Faults in LX #898

Open Smithx10 opened 4 years ago

Smithx10 commented 4 years ago

Running Void Linux: Platform:

bash-5.0# /native/sbin/uname -a
SunOS b50830bb-c147-499e-de19-f3ac0a4673f6 5.11 joyent_20190718T005708Z i86pc i386 i86pc

While running consul connect the envoy sidecar dumps.

Here is a link to the failure: (line 239) https://github.com/envoyproxy/envoy/blob/master/source/common/network/connection_impl.cc#L228

bash-5.0# consul connect envoy -sidecar-for socat
    Defaulting to grpc port = 8502
E0129 04:07:44.707285664  677633 cpu_linux.cc:75]            Cannot handle hot-plugged CPUs
E0129 04:07:44.708231559  677641 cpu_linux.cc:75]            Cannot handle hot-plugged CPUs
E0129 04:07:44.709539203  677642 cpu_linux.cc:75]            Cannot handle hot-plugged CPUs
E0129 04:07:44.709750642  677633 cpu_linux.cc:75]            Cannot handle hot-plugged CPUs
E0129 04:07:44.710061546  677643 cpu_linux.cc:75]            Cannot handle hot-plugged CPUs
[2020-01-29 04:07:44.710][677633][info][main] [external/envoy/source/server/server.cc:251] initializing epoch 0 (hot restart version=disabled)
[2020-01-29 04:07:44.710][677633][info][main] [external/envoy/source/server/server.cc:253] statically linked extensions:
[2020-01-29 04:07:44.710][677633][info][main] [external/envoy/source/server/server.cc:255]   envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.tap, envoy.transport_sockets.tls, raw_buffer, tls
[2020-01-29 04:07:44.710][677633][info][main] [external/envoy/source/server/server.cc:255]   envoy.access_loggers: envoy.file_access_log, envoy.http_grpc_access_log, envoy.tcp_grpc_access_log
[2020-01-29 04:07:44.710][677633][info][main] [external/envoy/source/server/server.cc:255]   envoy.filters.udp_listener: envoy.filters.udp_listener.udp_proxy
[2020-01-29 04:07:44.710][677633][info][main] [external/envoy/source/server/server.cc:255]   envoy.filters.http: envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.fault, envoy.filters.http.adaptive_concurrency, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_stats, envoy.filters.http.header_to_metadata, envoy.filters.http.jwt_authn, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.rbac, envoy.filters.http.tap, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.gzip, envoy.health_check, envoy.http_dynamo_filter, envoy.ip_tagging, envoy.lua, envoy.rate_limit, envoy.router, envoy.squash
[2020-01-29 04:07:44.710][677633][info][main] [external/envoy/source/server/server.cc:255]   envoy.grpc_credentials: envoy.grpc_credentials.aws_iam, envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata
[2020-01-29 04:07:44.710][677633][info][main] [external/envoy/source/server/server.cc:255]   envoy.dubbo_proxy.route_matchers: default
[2020-01-29 04:07:44.710][677633][info][main] [external/envoy/source/server/server.cc:255]   envoy.health_checkers: envoy.health_checkers.redis
[2020-01-29 04:07:44.710][677633][info][main] [external/envoy/source/server/server.cc:255]   envoy.resource_monitors: envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource
[2020-01-29 04:07:44.710][677633][info][main] [external/envoy/source/server/server.cc:255]   envoy.filters.listener: envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector
[2020-01-29 04:07:44.710][677633][info][main] [external/envoy/source/server/server.cc:255]   envoy.dubbo_proxy.protocols: dubbo
[2020-01-29 04:07:44.710][677633][info][main] [external/envoy/source/server/server.cc:255]   envoy.clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.dynamic_forward_proxy, envoy.clusters.redis
[2020-01-29 04:07:44.710][677633][info][main] [external/envoy/source/server/server.cc:255]   envoy.stats_sinks: envoy.dog_statsd, envoy.metrics_service, envoy.stat_sinks.hystrix, envoy.statsd
[2020-01-29 04:07:44.710][677633][info][main] [external/envoy/source/server/server.cc:255]   envoy.tracers: envoy.dynamic.ot, envoy.lightstep, envoy.tracers.datadog, envoy.tracers.opencensus, envoy.tracers.xray, envoy.zipkin
[2020-01-29 04:07:44.710][677633][info][main] [external/envoy/source/server/server.cc:255]   envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.previous_hosts
[2020-01-29 04:07:44.710][677633][info][main] [external/envoy/source/server/server.cc:255]   envoy.retry_priorities: envoy.retry_priorities.previous_priorities
[2020-01-29 04:07:44.710][677633][info][main] [external/envoy/source/server/server.cc:255]   envoy.udp_listeners: raw_udp_listener
[2020-01-29 04:07:44.710][677633][info][main] [external/envoy/source/server/server.cc:255]   envoy.resolvers: envoy.ip
[2020-01-29 04:07:44.710][677633][info][main] [external/envoy/source/server/server.cc:255]   envoy.thrift_proxy.filters: envoy.filters.thrift.rate_limit, envoy.filters.thrift.router
[2020-01-29 04:07:44.710][677633][info][main] [external/envoy/source/server/server.cc:255]   envoy.filters.network: envoy.client_ssl_auth, envoy.echo, envoy.ext_authz, envoy.filters.network.dubbo_proxy, envoy.filters.network.kafka_broker, envoy.filters.network.local_ratelimit, envoy.filters.network.mysql_proxy, envoy.filters.network.rbac, envoy.filters.network.sni_cluster, envoy.filters.network.thrift_proxy, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy
[2020-01-29 04:07:44.710][677633][info][main] [external/envoy/source/server/server.cc:255]   envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.tap, envoy.transport_sockets.tls, raw_buffer, tls
[2020-01-29 04:07:44.710][677633][info][main] [external/envoy/source/server/server.cc:255]   envoy.dubbo_proxy.filters: envoy.filters.dubbo.router
[2020-01-29 04:07:44.710][677633][info][main] [external/envoy/source/server/server.cc:255]   envoy.dubbo_proxy.serializers: dubbo.hessian2
[2020-01-29 04:07:44.710][677633][info][main] [external/envoy/source/server/server.cc:255]   envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter
[2020-01-29 04:07:44.710][677633][info][main] [external/envoy/source/server/server.cc:255]   envoy.thrift_proxy.transports: auto, framed, header, unframed
[2020-01-29 04:07:44.833][677633][warning][misc] [external/envoy/source/common/protobuf/utility.cc:441] Using deprecated option 'envoy.api.v2.Cluster.hosts' from file cluster.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details.
[2020-01-29 04:07:44.833][677633][info][main] [external/envoy/source/server/server.cc:336] admin address: 127.0.0.1:19000
[2020-01-29 04:07:44.837][677633][info][main] [external/envoy/source/server/server.cc:455] runtime: layers:
  - name: base
    static_layer:
      {}
  - name: admin
    admin_layer:
      {}
[2020-01-29 04:07:44.838][677633][info][config] [external/envoy/source/server/configuration_impl.cc:62] loading 0 static secret(s)
[2020-01-29 04:07:44.838][677633][info][config] [external/envoy/source/server/configuration_impl.cc:68] loading 1 cluster(s)
E0129 04:07:44.838127732  677633 cpu_linux.cc:75]            Cannot handle hot-plugged CPUs
E0129 04:07:44.838155249  677633 cpu_linux.cc:75]            Cannot handle hot-plugged CPUs
[2020-01-29 04:07:44.847][677633][info][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:167] cm init: initializing cds
[2020-01-29 04:07:44.849][677633][critical][assert] [external/envoy/source/common/network/connection_impl.cc:239] assert failure: 0 == rc.
[2020-01-29 04:07:44.850][677633][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:83] Caught Segmentation fault, suspect faulting address 0x7fffec9bf62f
[2020-01-29 04:07:44.850][677633][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:70] Backtrace (use tools/stack_decode.py to get line numbers):
[2020-01-29 04:07:44.850][677633][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:71] Envoy version: bb7ceff4c3c5bd4555dff28b6e56d27f2f8be0a7/1.13.0/clean-getenvoy-4bd7718-envoy/RELEASE/BoringSSL
[2020-01-29 04:07:44.852][677633][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #0: __restore_rt [0x7fffecb72120]
Segmentation fault (core dumped)
Smithx10 commented 4 years ago

looks like it is https://smartos.org/bugview/OS-6312

danmcd commented 4 years ago

Also see https://github.com/envoyproxy/envoy/issues/1446

I suspect this may mean OS-6312 isn't a bug at all.

danmcd commented 4 years ago

Also see envoyproxy/envoy#1446

I suspect this may mean OS-6312 isn't a bug at all.

Dammit. I keep forgetting this is an LX issue, where we might have to be bug-for-bug compatible. :-/

danmcd commented 4 years ago

As a workaround see Envoy-in-LX works after applying this workaround (clears after reboot) in the global zone:

root@global# echo "xnet_skip_checks/W1" | mdb -kw

Change the '1' to '0' to restore default system behavior.