search

googleapis / gax-php

Google API Extensions for PHP
http://googleapis.github.io/gax-php/
BSD 3-Clause "New" or "Revised" License
237 stars 52 forks source link

Intermittent segfault in 1.34.1 #581

Open vvoicehovics opened 2 months ago

vvoicehovics commented 2 months ago

Hi,

I have noticed occasional segfault crashes and was able to pin them to gax update 1.34.0 --> 1.34.1, this change https://github.com/googleapis/gax-php/commit/a47a469d9ef76613c5d320539646323a5e7b978d . The crash happens intermittently, once in 5-10 runs. I dont know how changes from 1.34.1 can break it, but it never crashes in 1.34.0.

image

I use Guzzle request pool to execute up to 200 requests concurrently. When request has succeeded a message is published to pubsub in Guzzle middleware ---> possibly crashes have to do with concurrency of the pool (i was only able to reproduce the issue when using request pool and dont know why the 1.34.1 change would otherwise break anything). One consequence of the 1.34.1 is that hasCheckedUniverse remains false and shouldCheckUniverseDomain is called with each request.

Additionally, as can see from the error, it starts with DEADLINE_EXCEEDED and then as i understand next/later request results in segfault. I never get DEADLINE_EXCEEDED in 1.34.0 or without segfault.

This doesnt seem to be linked to memory/cpu resource exhaustion.

@bshaffer

Environment details

Core dump

Program terminated with signal SIGSEGV, Segmentation fault.

#0  0x00005816f63f3cea in zend_array_dup ()

[Current thread is 1 (LWP 419)]

(gdb) bt

#0  0x00005816f63f3cea in zend_array_dup ()
#1  0x00005816f640a491 in ?? ()
#2  0x00005816f64486e3 in execute_ex ()
#3  0x00005816f63d246f in zend_call_function ()
#4  0x00007d14db813f81 in plugin_get_metadata () from /usr/lib/php81/modules/grpc.so
#5  0x00007d14db59fb69 in grpc_plugin_credentials::GetRequestMetadata(std::unique_ptr<grpc_metadata_batch, grpc_core::Arena::PooledDeleter>, grpc_call_credentials::GetRequestMetadataArgs const*) ()
   from /usr/lib/php81/modules/grpc.so
#6  0x00007d14db5cd513 in grpc_core::ClientAuthFilter::GetCallCredsMetadata(grpc_core::CallArgs) () from /usr/lib/php81/modules/grpc.so
#7  0x00007d14db5ced2e in grpc_core::promise_detail::SeqState<grpc_core::promise_detail::TrySeqTraits, grpc_core::ArenaPromise<absl::lts_20240116::Status>, grpc_core::ClientAuthFilter::MakeCallPromise(grpc_core::CallArgs, std::function<grpc_core::ArenaPromise<std::unique_ptr<grpc_metadata_batch, grpc_core::Arena::PooledDeleter> > (grpc_core::CallArgs)>)::{lambda()#1}, std::function<grpc_core::ArenaPromise<std::unique_ptr<grpc_metadata_batch, grpc_core::Arena::PooledDeleter> > (grpc_core::CallArgs)> >::PollOnce() () from /usr/lib/php81/modules/grpc.so
#8  0x00007d14db5cf443 in grpc_core::arena_promise_detail::AllocatedCallable<std::unique_ptr<grpc_metadata_batch, grpc_core::Arena::PooledDeleter>, grpc_core::promise_detail::TrySeq<grpc_core::ArenaPromise<absl::lts_20240116::Status>, grpc_core::ClientAuthFilter::MakeCallPromise(grpc_core::CallArgs, std::function<grpc_core::ArenaPromise<std::unique_ptr<grpc_metadata_batch, grpc_core::Arena::PooledDeleter> > (grpc_core::CallArgs)>)::{lambda()#1}, std::function<grpc_core::ArenaPromise<std::unique_ptr<grpc_metadata_batch, grpc_core::Arena::PooledDeleter> > (grpc_core::CallArgs)> > >::PollOnce(grpc_core::arena_promise_detail::ArgType*) () from /usr/lib/php81/modules/grpc.so
#9  0x00007d14db4b46da in grpc_core::promise_filter_detail::ClientCallData::PollContext::Run() () from /usr/lib/php81/modules/grpc.so
#10 0x00007d14db4a803a in grpc_core::promise_filter_detail::ClientCallData::StartPromise(grpc_core::promise_filter_detail::BaseCallData::Flusher*) () from /usr/lib/php81/modules/grpc.so
#11 0x00007d14db4a8452 in grpc_core::promise_filter_detail::ClientCallData::StartBatch(grpc_transport_stream_op_batch*) () from /usr/lib/php81/modules/grpc.so
#12 0x00007d14db4a3d9d in grpc_core::promise_filter_detail::BaseCallData::Flusher::~Flusher() () from /usr/lib/php81/modules/grpc.so
#13 0x00007d14db4a8609 in grpc_core::promise_filter_detail::ClientCallData::StartBatch(grpc_transport_stream_op_batch*) () from /usr/lib/php81/modules/grpc.so
#14 0x00007d14db512666 in grpc_core::ExecCtx::Flush() () from /usr/lib/php81/modules/grpc.so
#15 0x00007d14db508003 in pollset_work(grpc_pollset*, grpc_pollset_worker**, grpc_core::Timestamp) () from /usr/lib/php81/modules/grpc.so
#16 0x00007d14db50d181 in pollset_work(grpc_pollset*, grpc_pollset_worker**, grpc_core::Timestamp) () from /usr/lib/php81/modules/grpc.so
#17 0x00007d14db516810 in grpc_pollset_work(grpc_pollset*, grpc_pollset_worker**, grpc_core::Timestamp) () from /usr/lib/php81/modules/grpc.so
#18 0x00007d14db5fe34a in cq_pluck(grpc_completion_queue*, void*, gpr_timespec, void*) () from /usr/lib/php81/modules/grpc.so
#19 0x00007d14db8135e7 in zim_Call_startBatch () from /usr/lib/php81/modules/grpc.so
#20 0x00005816f644e9e2 in execute_ex ()
#21 0x00005816f63d246f in zend_call_function ()
#22 0x00005816f6306bef in ?? ()
#23 0x00005816f644d8c0 in execute_ex ()
#24 0x00005816f644fd0f in zend_execute ()
#25 0x00005816f63e0f38 in zend_execute_scripts ()
#26 0x00005816f6379483 in php_execute_script ()
#27 0x00005816f64c8c72 in ?? ()
#28 0x00005816f6238347 in ?? ()
#29 0x00007d14debf99ca in ?? () from /lib/ld-musl-x86_64.so.1
#30 0x00007d14debf99a2 in ?? () from /lib/ld-musl-x86_64.so.1
#31 0x00007ffe4412c760 in ?? ()
#32 0x0000000000000000 in ?? ()