search

istio / istio

Connect, secure, control, and observe services.
https://istio.io
Apache License 2.0
35.94k stars 7.76k forks source link

Envoy exited with error: signal: segmentation fault #49777

Closed netkey closed 4 months ago

netkey commented 8 months ago

Is this the right place to submit this?

Bug Description

Hi,I have created an Envoy OAuth2 filter at the Istio Ingress Gateway level,It looks work fine.However, the ingressgateway pod will restart after running serveral days.here is some logs like this:

2024-03-04T17:13:37.618178Z error envoy wasm no context context_id: 62343883 2024-03-04T17:13:37.618226Z critical envoy backtrace Caught Segmentation fault, suspect faulting address 0x0 2024-03-04T17:13:37.618231Z critical envoy backtrace Backtrace (use tools/stack_decode.py to get line numbers): 2024-03-04T17:13:37.618236Z critical envoy backtrace Envoy version: ca3e0c6ebdfe634348776db633b7509cf097e76f/1.24.10-dev/Clean/RELEASE/BoringSSL 2024-03-04T17:13:37.618497Z critical envoy backtrace #0: [0x7f46a1c4f520] 2024-03-04T17:13:37.729990Z critical envoy backtrace #1: proxy_wasm::ContextBase::onResponseHeaders() [0x55b667d9a154] 2024-03-04T17:13:37.771006Z critical envoy backtrace #2: Envoy::Extensions::Common::Wasm::Context::encodeHeaders() [0x55b667b9a4ef] 2024-03-04T17:13:37.814968Z critical envoy backtrace #3: Envoy::Http::FilterManager::encodeHeaders() [0x55b6692444f4] 2024-03-04T17:13:37.853972Z critical envoy backtrace #4: Envoy::Extensions::HttpFilters::Oauth2::OAuth2Filter::finishFlow() [0x55b66797a583] 2024-03-04T17:13:37.888044Z critical envoy backtrace #5: Envoy::Extensions::HttpFilters::Oauth2::OAuth2Filter::onGetAccessTokenSuccess() [0x55b6679797dc] 2024-03-04T17:13:37.943976Z critical envoy backtrace #6: Envoy::Extensions::HttpFilters::Oauth2::OAuth2ClientImpl::onSuccess() [0x55b66797d009] 2024-03-04T17:13:37.982972Z critical envoy backtrace #7: Envoy::Http::AsyncRequestImpl::onComplete() [0x55b66919879e] 2024-03-04T17:13:38.026981Z critical envoy backtrace #8: Envoy::Http::AsyncStreamImpl::encodeData() [0x55b669197cd6] 2024-03-04T17:13:38.062358Z critical envoy backtrace #9: Envoy::Router::UpstreamRequest::decodeData() [0x55b6691da5f5] 2024-03-04T17:13:38.106022Z critical envoy backtrace #10: Envoy::Http::FilterManager::encodeData() [0x55b669245748] 2024-03-04T17:13:38.155973Z critical envoy backtrace #11: Envoy::Http::ResponseDecoderWrapper::decodeData() [0x55b668ef180b] 2024-03-04T17:13:38.156210Z critical envoy backtrace #12: Envoy::Http::ResponseDecoderWrapper::decodeData() [0x55b668ef180b] 2024-03-04T17:13:38.194765Z critical envoy backtrace #13: Envoy::Http::Http1::ClientConnectionImpl::onMessageCompleteBase() [0x55b6690fb631] 2024-03-04T17:13:38.237995Z critical envoy backtrace #14: Envoy::Http::Http1::ConnectionImpl::onMessageCompleteImpl() [0x55b6690f643d] 2024-03-04T17:13:38.274968Z critical envoy backtrace #15: Envoy::Http::Http1::ConnectionImpl::onMessageComplete() [0x55b6690f6168] 2024-03-04T17:13:38.324970Z critical envoy backtrace #16: http_parser_execute [0x55b66951fc39] 2024-03-04T17:13:38.325181Z critical envoy backtrace #17: Envoy::Http::Http1::ConnectionImpl::dispatchSlice() [0x55b6690f44a6] 2024-03-04T17:13:38.325282Z critical envoy backtrace #18: Envoy::Http::Http1::ConnectionImpl::dispatch() [0x55b6690f3c9f] 2024-03-04T17:13:38.361456Z critical envoy backtrace #19: Envoy::Http::Http1::ClientConnectionImpl::dispatch() [0x55b6690f382d] 2024-03-04T17:13:38.415971Z critical envoy backtrace #20: Envoy::Http::CodecClient::onData() [0x55b668fa5e5d] 2024-03-04T17:13:38.447869Z critical envoy backtrace #21: Envoy::Http::CodecClient::CodecReadFilter::onData() [0x55b668fa76d5] 2024-03-04T17:13:38.450461Z critical envoy backtrace #22: Envoy::Network::FilterManagerImpl::onContinueReading() [0x55b66940e2f6] 2024-03-04T17:13:38.450600Z critical envoy backtrace #23: Envoy::Network::ConnectionImpl::onReadReady() [0x55b6694071e8] 2024-03-04T17:13:38.450736Z critical envoy backtrace #24: Envoy::Network::ConnectionImpl::onFileEvent() [0x55b669404d13] 2024-03-04T17:13:38.450804Z critical envoy backtrace #25: std::1::function::__func<>::operator()() [0x55b6693f9bd1] 2024-03-04T17:13:38.450874Z critical envoy backtrace #26: Envoy::Event::FileEventImpl::assignEvents()::$_1::invoke() [0x55b6693fb0ec] 2024-03-04T17:13:38.450966Z critical envoy backtrace #27: event_process_active_single_queue [0x55b669519c30] 2024-03-04T17:13:38.500744Z critical envoy backtrace #28: event_base_loop [0x55b669518641] 2024-03-04T17:13:38.548972Z critical envoy backtrace #29: Envoy::Server::WorkerImpl::threadRoutine() [0x55b668be7efd] 2024-03-04T17:13:38.585977Z critical envoy backtrace #30: Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::{lambda()#1}::invoke() [0x55b6697249c3] 2024-03-04T17:13:38.586359Z critical envoy backtrace #31: [0x7f46a1ca1b43] AsyncClient 0x55b6a34ca410, streamid: 18343960974272027037 &streaminfo: StreamInfoImpl 0x55b6a34ca660, protocol_: 1, responsecode: 200, response_codedetails: via_upstream, attemptcount: 1, health_checkrequest: 0, routename: upstreaminfo: UpstreamInfoImpl 0x55b670c7cf18, upstream_connectionid: 9144619 Http1::ConnectionImpl 0x55b66f256008, dispatching_: 1, dispatching_slice_alreadydrained: 0, reset_streamcalled: 0, handlingupgrade: 0, deferred_end_streamheaders: 0, processingtrailers: 0, bufferedbody.length(): 0, header_parsingstate: Done, current_headerfield: , current_headervalue: absl::get(headers_ortrailers): null Dumping corresponding downstream request: decoder: UpstreamRequest 0x55b68a0a9b00 requestheaders: ':path', '/oauth/token' ':authority', 'myapp-oauth2.myapp.svc.cluster.local' ':method', 'POST' ':scheme', 'http' 'content-type', 'application/x-www-form-urlencoded' 'accept', 'application/json' 'content-length', '228' 'x-envoy-internal', 'true' 'x-forwarded-for', '172.17.30.5' 'x-envoy-expected-rq-timeout-ms', '3000' FilterManager 0x55b670c7c640, state.has_1xxheaders: 0 filter_managercallbacks.requestHeaders(): ':path', '/oauth/token' ':authority', 'myapp-oauth2.myapp.svc.cluster.local' ':method', 'POST' ':scheme', 'http' 'content-type', 'application/x-www-form-urlencoded' 'accept', 'application/json' 'content-length', '228' 'x-envoy-internal', 'true' 'x-forwarded-for', '172.17.30.5' 'x-envoy-expected-rq-timeout-ms', '3000' filter_managercallbacks.requestTrailers(): null filter_managercallbacks.responseHeaders(): null filter_managercallbacks.responseTrailers(): null &streamInfo(): StreamInfoImpl 0x55b6a34ca660, protocol_: 1, responsecode: 200, response_codedetails: via_upstream, attemptcount: 1, health_checkrequest: 0, routename: upstreaminfo: UpstreamInfoImpl 0x55b670c7cf18, upstream_connectionid: 9144619 current_dispatchingbuffer front_slice length: 580 contents: "HTTP/1.1 200 OK\r\nAccess-Control-Allow-Credentials: true\r\nAccess-Control-Allow-Headers: sk,sn,Content-Type, Content-Length, Accept-Encoding, X-CSRF-Token, Authorization, accept, origin, Cache-Control, X-Requested-With\r\nAccess-Control-Allow-Methods: OPTIONS, GET, POST, PUT, DELETE, UPDATE\r\nAccess-Control-Allow-Origin: *\r\nAccess-Control-Max-Age: 86400\r\nContent-Type: application/json; charset=utf-8\r\nDate: Mon, 04 Mar 2024 17:13:35 GMT\r\nContent-Length: 121\r\n\r\n{\"access_token\":\"5d9abc34f17658dbb049a45ec5eb7107e65fb1a8-3da9-406f-96db-5830b88f28ab\",\"expiresin\":604800,\"scope\":\"all\"}" ConnectionImpl 0x55b68bbdbb00, connecting: 0, binderror: 0, state(): Open, read_bufferlimit: 1048576 socket_: ListenSocketImpl 0x55b693405780, transportprotocol: connection_infoprovider: ConnectionInfoSetterImpl 0x55b6fb90d7a8, remoteaddress: 172.17.27.132:80, direct_remoteaddress: 172.17.27.132:80, localaddress: 172.17.30.5:46374, servername: 2024-03-04T17:13:39.018316Z info ads ADS: "@" istio-ingressgateway-oauth2-6c694d7557-jphm8.istio-system-2 terminated 2024-03-04T17:13:39.018420Z info ads ADS: "@" istio-ingressgateway-oauth2-6c694d7557-jphm8.istio-system-1 terminated 2024-03-04T17:13:39.057971Z error Envoy exited with error: signal: segmentation fault

Version

$istioctl version                                                                                                             
client version: 1.16.7
control plane version: 1.16.7
data plane version: 1.14.6 (6 proxies), 1.16.7 (67 proxies)
$kubectl version --short                                                                                                      
Client Version: v1.21.3
Server Version: v1.22.10-aliyun.1

Additional Information

No response

istio-policy-bot commented 4 months ago

🚧 This issue or pull request has been closed due to not having had activity from an Istio team member since 2024-03-07. If you feel this issue or pull request deserves attention, please reopen the issue. Please see this wiki page for more information. Thank you for your contributions.

Created by the issue and PR lifecycle manager.