Closed knm3000 closed 2 years ago
After we had introduced big-endian support in proxy-wasm https://github.com/proxy-wasm/proxy-wasm-cpp-host/pull/198 and fixed https://github.com/proxy-wasm/proxy-wasm-cpp-host/issues/197 the following new error started happening (regression of https://github.com/proxy-wasm/proxy-wasm-cpp-host/pull/198): tcp_metadata_exchange test from proxy repo fails on big-endian platform (envoy binary crash) https://github.com/istio/proxy/blob/master/test/envoye2e/tcp_metadata_exchange/tcp_metadata_exchange_test.go
go test ./test/envoye2e/tcp_metadata_exchange -v ... [2022-07-25 11:05:29.028][240769][critical][assert] [external/envoy/source/exe/main_common.cc:84] panic: out of memory [2022-07-25 11:05:29.091][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:104] Caught Aborted, suspect faulting address 0x3ac8100000000 [2022-07-25 11:05:29.226][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers): [2022-07-25 11:05:29.226][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:92] Envoy version: b977767d8be6ce88e1956cd1f6165242aab2a572/1.20.6/Modified/RELEASE/OpenSSL [2022-07-25 11:05:29.256][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #0: [0x20025f33490] [2022-07-25 11:05:29.257][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #1: __pthread_kill_implementation [0x2002679eb86] [2022-07-25 11:05:29.258][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #2: gsignal [0x200267500e0] [2022-07-25 11:05:29.258][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #3: abort [0x2002672b100] [2022-07-25 11:05:29.362][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #4: Envoy::MainCommonBase::MainCommonBase()::$_1::__invoke() [0x2aa3cea39bc] [2022-07-25 11:05:29.372][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #5: tcmalloc::cpp_throw_oom() [0x2aa4076b0a0] [2022-07-25 11:05:29.381][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #6: tcmalloc::allocate_full_cpp_throw_oom() [0x2aa4076bd1c] [2022-07-25 11:05:29.399][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #7: flatbuffers::FlatBufferBuilder::CreateString() [0x2aa3ce7140a] [2022-07-25 11:05:29.408][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #8: Wasm::Common::extractLocalNodeFlatBuffer() [0x2aa3ce6aa66] [2022-07-25 11:05:29.417][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #9: proxy_wasm::null_plugin::Stats::PluginRootContext::configure() [0x2aa3cdbbcc0] [2022-07-25 11:05:29.426][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #10: proxy_wasm::null_plugin::Stats::PluginRootContext::onConfigure() [0x2aa3cdbbc18] [2022-07-25 11:05:29.435][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #11: std::_Function_handler<>::_M_invoke() [0x2aa3d9a2898] [2022-07-25 11:05:29.445][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #12: proxy_wasm::ContextBase::onConfigure() [0x2aa3d9a83c8] [2022-07-25 11:05:29.463][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #13: proxy_wasm::WasmBase::configure() [0x2aa3d9c42b4] [2022-07-25 11:05:29.474][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #14: proxy_wasm::createWasm() [0x2aa3d9c66da] [2022-07-25 11:05:29.484][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #15: Envoy::Extensions::Common::Wasm::createWasm()::$_8::operator()() [0x2aa3d6d05f4] [2022-07-25 11:05:29.512][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #16: Envoy::Extensions::Common::Wasm::createWasm() [0x2aa3d6cd924] [2022-07-25 11:05:29.527][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #17: Envoy::Extensions::NetworkFilters::Wasm::FilterConfig::FilterConfig() [0x2aa3d5ecf9a] [2022-07-25 11:05:29.537][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #18: Envoy::Extensions::NetworkFilters::Wasm::WasmFilterConfig::createFilterFactoryFromProtoTyped() [0x2aa3d5eab6c] [2022-07-25 11:05:29.547][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #19: Envoy::Extensions::NetworkFilters::Common::FactoryBase<>::createFilterFactoryFromProto() [0x2aa3d5eb976] [2022-07-25 11:05:29.557][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #20: Envoy::Server::ProdListenerComponentFactory::createNetworkFilterFactoryList_() [0x2aa3f96c45a] [2022-07-25 11:05:29.575][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #21: Envoy::Server::ListenerFilterChainFactoryBuilder::buildFilterChainInternal() [0x2aa3f97985a] [2022-07-25 11:05:29.586][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #22: Envoy::Server::ListenerFilterChainFactoryBuilder::buildFilterChain() [0x2aa3f979526] [2022-07-25 11:05:29.599][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #23: Envoy::Server::FilterChainManagerImpl::addFilterChains() [0x2aa3f996c4c] [2022-07-25 11:05:29.609][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #24: Envoy::Server::ListenerImpl::ListenerImpl() [0x2aa3f95f46a] [2022-07-25 11:05:29.620][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #25: Envoy::Server::ListenerManagerImpl::addOrUpdateListenerInternal() [0x2aa3f974652] [2022-07-25 11:05:29.630][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #26: Envoy::Server::ListenerManagerImpl::addOrUpdateListener() [0x2aa3f9735d8] [2022-07-25 11:05:29.658][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #27: Envoy::Server::LdsApiImpl::onConfigUpdate() [0x2aa3f9e8e16] [2022-07-25 11:05:29.691][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #28: Envoy::Server::LdsApiImpl::onConfigUpdate() [0x2aa3f9ea7f8] [2022-07-25 11:05:29.718][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #29: Envoy::Config::GrpcSubscriptionImpl::onConfigUpdate() [0x2aa3fb27e70] [2022-07-25 11:05:29.739][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #30: Envoy::Config::GrpcMuxImpl::onDiscoveryResponse() [0x2aa3fb309a8] [2022-07-25 11:05:29.748][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #31: Envoy::Grpc::AsyncStreamCallbacks<>::onReceiveMessageRaw() [0x2aa3fb34a3c] [2022-07-25 11:05:29.763][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #32: Envoy::Grpc::AsyncStreamImpl::onData() [0x2aa3fb570c6] [2022-07-25 11:05:29.780][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #33: Envoy::Http::AsyncStreamImpl::encodeData() [0x2aa3fb60b4c] [2022-07-25 11:05:29.789][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #34: Envoy::Router::UpstreamRequest::decodeData() [0x2aa3fbcb9f0] [2022-07-25 11:05:29.798][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #35: Envoy::Http::ResponseDecoderWrapper::decodeData() [0x2aa3f81b63e] [2022-07-25 11:05:29.807][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #36: Envoy::Http::Http2::ConnectionImpl::onFrameReceived() [0x2aa3faa37a0] [2022-07-25 11:05:29.816][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #37: Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::$_22::__invoke() [0x2aa3fab0824] [2022-07-25 11:05:29.836][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #38: nghttp2_session_on_data_received [0x2aa3fd088e4] [2022-07-25 11:05:29.846][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #39: nghttp2_session_mem_recv [0x2aa3fd0ab0a] [2022-07-25 11:05:29.855][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #40: Envoy::Http::Http2::ConnectionImpl::dispatch() [0x2aa3faa1a00] [2022-07-25 11:05:29.864][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #41: Envoy::Http::CodecClient::onData() [0x2aa3f8cb478] [2022-07-25 11:05:29.873][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #42: Envoy::Http::CodecClient::CodecReadFilter::onData() [0x2aa3f8cd5e2] [2022-07-25 11:05:29.882][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #43: Envoy::Network::FilterManagerImpl::onRead() [0x2aa3fb19de2] [2022-07-25 11:05:29.891][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #44: Envoy::Network::ConnectionImpl::onReadReady() [0x2aa3fb0f666] [2022-07-25 11:05:29.900][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #45: Envoy::Network::ConnectionImpl::onFileEvent() [0x2aa3fb0ccc8] [2022-07-25 11:05:29.908][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #46: std::_Function_handler<>::_M_invoke() [0x2aa3fae79da] [2022-07-25 11:05:29.917][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #47: Envoy::Event::FileEventImpl::assignEvents()::$_1::__invoke() [0x2aa3fae8d84] [2022-07-25 11:05:29.926][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #48: event_process_active_single_queue [0x2aa3fcf8b3e] [2022-07-25 11:05:29.936][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #49: event_base_loop [0x2aa3fcf757e] [2022-07-25 11:05:29.945][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #50: Envoy::Event::DispatcherImpl::run() [0x2aa3fae512e] [2022-07-25 11:05:29.973][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #51: Envoy::Server::InstanceImpl::run() [0x2aa3f4cadf6] [2022-07-25 11:05:29.983][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #52: Envoy::MainCommonBase::run() [0x2aa3cea1312] [2022-07-25 11:05:29.992][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #53: Envoy::MainCommon::main() [0x2aa3cea1c76] [2022-07-25 11:05:30.001][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #54: main [0x2aa3ce9d13e] [2022-07-25 11:05:30.001][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #55: __libc_start_call_main [0x200267317f2] [2022-07-25 11:05:30.002][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #56: __libc_start_main@GLIBC_2.2 [0x200267318d0] [2022-07-25 11:05:30.010][240769][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #57: [0x2aa3cc9ee90] AsyncClient 0x2aa43ecb080, stream_id_: 8385127285791150716
The root cause is that tcp_metadata_exchange test uses null-vm, and in https://github.com/proxy-wasm/proxy-wasm-cpp-host/pull/198 we introduced byte swapping on big-endian for all wasm runtimes (including null-vm). Since null-vm is not using wasm binary, we should not reverse bytes for null-vm (for other wasm runtimes we still need to reverse bytes on big-endian). Opened https://github.com/proxy-wasm/proxy-wasm-cpp-host/pull/282 to fix it.
After we had introduced big-endian support in proxy-wasm https://github.com/proxy-wasm/proxy-wasm-cpp-host/pull/198 and fixed https://github.com/proxy-wasm/proxy-wasm-cpp-host/issues/197 the following new error started happening (regression of https://github.com/proxy-wasm/proxy-wasm-cpp-host/pull/198): tcp_metadata_exchange test from proxy repo fails on big-endian platform (envoy binary crash) https://github.com/istio/proxy/blob/master/test/envoye2e/tcp_metadata_exchange/tcp_metadata_exchange_test.go
The root cause is that tcp_metadata_exchange test uses null-vm, and in https://github.com/proxy-wasm/proxy-wasm-cpp-host/pull/198 we introduced byte swapping on big-endian for all wasm runtimes (including null-vm). Since null-vm is not using wasm binary, we should not reverse bytes for null-vm (for other wasm runtimes we still need to reverse bytes on big-endian). Opened https://github.com/proxy-wasm/proxy-wasm-cpp-host/pull/282 to fix it.