search

DataDog / dd-trace-php

Datadog PHP Clients
https://docs.datadoghq.com/tracing/setup/php
Other
483 stars 149 forks source link

[Bug]: dd-trace-php causes php-fpm to SEGFAULT (likely due to use of die()) #2276

Closed zaneclaes closed 9 months ago

zaneclaes commented 10 months ago

Bug report

Since upgrading nginx, PHP, and dd-trace-php, I have began experiencing segfaults with php-fpm. I finally tracked it down to this library and validated the fix by removing dd-trace-php from the docker container.

The error presents itself in the php-fpm logs with a mysterious SEGFAULT:

 exited on signal 11 (SIGSEGV - core dumped) after 43.414095 seconds from start

I was unable to get a core dump in the Kubernetes pod, but eventually I found this issue in the opentracing-php repository which perfectly explained the problem: https://github.com/open-telemetry/opentelemetry-php/issues/951

PHP version

8.2

Tracer or profiler version

0.91.2

Installed extensions

No response

Output of phpinfo()

Datadog PHP tracer extension For help, check out the documentation at https://docs.datadoghq.com/tracing/languages/php/ (c) Datadog 2020

Datadog tracing support => disabled Version => 0.91.2 DATADOG TRACER CONFIGURATION => { "date": "2023-09-18T23:16:16Z", "os_name": "Linux multipress-54fdbb55c6-zg84l 5.15.0-83-generic https://github.com/DataDog/dd-trace-php/pull/92~20.04.1-Ubuntu SMP Mon Aug 21 14:00:49 UTC 2023 x86_64", "os_version": "5.15.0-83-generic", "version": "0.91.2", "lang": "php", "lang_version": "8.2.8", "env": null, "enabled": true, "service": null, "enabled_cli": false, "agent_url": "http://10.0.0.15:8126", "debug": false, "analytics_enabled": false, "sample_rate": -1, "sampling_rules": [], "tags": [], "service_mapping": [], "distributed_tracing_enabled": true, "priority_sampling_enabled": true, "dd_version": null, "architecture": "x86_64", "sapi": "cli", "datadog.trace.request_init_hook": "/opt/datadog/dd-library/0.91.2/dd-trace-sources/bridge/dd_wrap_autoloader.php", "open_basedir_configured": false, "uri_fragment_regex": null, "uri_mapping_incoming": null, "uri_mapping_outgoing": null, "auto_flush_enabled": false, "generate_root_span": true, "http_client_split_by_domain": false, "measure_compile_time": true, "report_hostname_on_root_span": false, "traced_internal_functions": null, "auto_prepend_file_configured": false, "integrations_disabled": "default", "enabled_from_env": false, "opcache.file_cache": null }

Directive => Local Value => Master Value datadog.agent_host => 10.0.0.15 => 10.0.0.15 datadog.amqp_analytics_enabled => Off => Off datadog.amqp_analytics_sample_rate => 1 => 1 datadog.autofinish_spans => Off => Off datadog.cakephp_analytics_enabled => Off => Off datadog.cakephp_analytics_sample_rate => 1 => 1 datadog.codeigniter_analytics_enabled => Off => Off datadog.codeigniter_analytics_sample_rate => 1 => 1 datadog.curl_analytics_enabled => Off => Off datadog.curl_analytics_sample_rate => 1 => 1 datadog.dbm_propagation_mode => disabled => disabled datadog.distributed_tracing => On => On datadog.dogstatsd_port => 8125 => 8125 datadog.dogstatsd_url => no value => no value datadog.elasticsearch_analytics_enabled => Off => Off datadog.elasticsearch_analytics_sample_rate => 1 => 1 datadog.eloquent_analytics_enabled => Off => Off datadog.eloquent_analytics_sample_rate => 1 => 1 datadog.env => no value => no value datadog.guzzle_analytics_enabled => Off => Off datadog.guzzle_analytics_sample_rate => 1 => 1 datadog.http_server_route_based_naming => On => On datadog.instrumentation_telemetry_enabled => On => On datadog.integrations_disabled => default => default datadog.laminas_analytics_enabled => Off => Off datadog.laminas_analytics_sample_rate => 1 => 1 datadog.laravel_analytics_enabled => Off => Off datadog.laravel_analytics_sample_rate => 1 => 1 datadog.laravelqueue_analytics_enabled => Off => Off datadog.laravelqueue_analytics_sample_rate => 1 => 1 datadog.log_backtrace => Off => Off datadog.logs_analytics_enabled => Off => Off datadog.logs_analytics_sample_rate => 1 => 1 datadog.logs_injection => On => On datadog.lumen_analytics_enabled => Off => Off datadog.lumen_analytics_sample_rate => 1 => 1 datadog.memcache_analytics_enabled => Off => Off datadog.memcache_analytics_sample_rate => 1 => 1 datadog.memcached_analytics_enabled => Off => Off datadog.memcached_analytics_sample_rate => 1 => 1 datadog.mongo_analytics_enabled => Off => Off datadog.mongo_analytics_sample_rate => 1 => 1 datadog.mongodb_analytics_enabled => Off => Off datadog.mongodb_analytics_sample_rate => 1 => 1 datadog.mysqli_analytics_enabled => Off => Off datadog.mysqli_analytics_sample_rate => 1 => 1 datadog.nette_analytics_enabled => Off => Off datadog.nette_analytics_sample_rate => 1 => 1 datadog.pcntl_analytics_enabled => Off => Off datadog.pcntl_analytics_sample_rate => 1 => 1 datadog.pdo_analytics_enabled => Off => Off datadog.pdo_analytics_sample_rate => 1 => 1 datadog.phpredis_analytics_enabled => Off => Off datadog.phpredis_analytics_sample_rate => 1 => 1 datadog.predis_analytics_enabled => Off => Off datadog.predis_analytics_sample_rate => 1 => 1 datadog.priority_sampling => On => On datadog.propagation_style_extract => tracecontext,Datadog,B3,B3 single header => tracecontext,Datadog,B3,B3 single header datadog.propagation_style_inject => tracecontext,Datadog => tracecontext,Datadog datadog.psr18_analytics_enabled => Off => Off datadog.psr18_analytics_sample_rate => 1 => 1 datadog.roadrunner_analytics_enabled => Off => Off datadog.roadrunner_analytics_sample_rate => 1 => 1 datadog.sampling_rate => -1 => -1 datadog.service => no value => no value datadog.service_mapping => no value => no value datadog.service_name => no value => no value datadog.slim_analytics_enabled => Off => Off datadog.slim_analytics_sample_rate => 1 => 1 datadog.span_sampling_rules => [] => [] datadog.span_sampling_rules_file => no value => no value datadog.sqlsrv_analytics_enabled => Off => Off datadog.sqlsrv_analytics_sample_rate => 1 => 1 datadog.symfony_analytics_enabled => Off => Off datadog.symfony_analytics_sample_rate => 1 => 1 datadog.tags => no value => no value datadog.trace.128_bit_traceid_generation_enabled => Off => Off datadog.trace.128_bit_traceid_logging_enabled => Off => Off datadog.trace.agent_attempt_retry_time_msec => 5000 => 5000 datadog.trace.agent_connect_timeout => 100 => 100 datadog.trace.agent_debug_verbose_curl => Off => Off datadog.trace.agent_flush_after_n_requests => 10 => 10 datadog.trace.agent_flush_interval => 5000 => 5000 datadog.trace.agent_max_consecutive_failures => 3 => 3 datadog.trace.agent_max_payload_size => 52428800 => 52428800 datadog.trace.agent_port => 0 => 0 datadog.trace.agent_stack_backlog => 12 => 12 datadog.trace.agent_stack_initial_size => 131072 => 131072 datadog.trace.agent_timeout => 500 => 500 datadog.trace.agent_url => no value => no value datadog.trace.amqp_analytics_enabled => Off => Off datadog.trace.amqp_analytics_sample_rate => 1 => 1 datadog.trace.amqp_enabled => On => On datadog.trace.analytics_enabled => Off => Off datadog.trace.append_trace_ids_to_logs => Off => Off datadog.trace.auto_flush_enabled => Off => Off datadog.trace.beta_high_memory_pressure_percent => 80 => 80 datadog.trace.bgs_connect_timeout => 2000 => 2000 datadog.trace.bgs_timeout => 5000 => 5000 datadog.trace.cakephp_analytics_enabled => Off => Off datadog.trace.cakephp_analytics_sample_rate => 1 => 1 datadog.trace.cakephp_enabled => On => On datadog.trace.cli_enabled => Off => Off datadog.trace.client_ip_enabled => Off => Off datadog.trace.client_ip_header => no value => no value datadog.trace.codeigniter_analytics_enabled => Off => Off datadog.trace.codeigniter_analytics_sample_rate => 1 => 1 datadog.trace.codeigniter_enabled => On => On datadog.trace.curl_analytics_enabled => Off => Off datadog.trace.curl_analytics_sample_rate => 1 => 1 datadog.trace.curl_enabled => On => On datadog.trace.db_client_split_by_instance => Off => Off datadog.trace.debug => Off => Off datadog.trace.debug_curl_output => Off => Off datadog.trace.debug_prng_seed => -1 => -1 datadog.trace.elasticsearch_analytics_enabled => Off => Off datadog.trace.elasticsearch_analytics_sample_rate => 1 => 1 datadog.trace.elasticsearch_enabled => On => On datadog.trace.eloquent_analytics_enabled => Off => Off datadog.trace.eloquent_analytics_sample_rate => 1 => 1 datadog.trace.eloquent_enabled => On => On datadog.trace.enabled => Off => On datadog.trace.flush_collect_cycles => Off => Off datadog.trace.forked_process => On => On datadog.trace.generate_root_span => On => On datadog.trace.global_tags => no value => no value datadog.trace.guzzle_analytics_enabled => Off => Off datadog.trace.guzzle_analytics_sample_rate => 1 => 1 datadog.trace.guzzle_enabled => On => On datadog.trace.header_tags => no value => no value datadog.trace.health_metrics_enabled => Off => Off datadog.trace.health_metrics_heartbeat_sample_rate => 0.001 => 0.001 datadog.trace.hook_limit => 100 => 100 datadog.trace.http_client_split_by_domain => Off => Off datadog.trace.http_post_data_param_allowed => no value => no value datadog.trace.http_url_query_param_allowed => => datadog.trace.laminas_analytics_enabled => Off => Off datadog.trace.laminas_analytics_sample_rate => 1 => 1 datadog.trace.laminas_enabled => On => On datadog.trace.laravel_analytics_enabled => Off => Off datadog.trace.laravel_analytics_sample_rate => 1 => 1 datadog.trace.laravel_enabled => On => On datadog.trace.laravelqueue_analytics_enabled => Off => Off datadog.trace.laravelqueue_analytics_sample_rate => 1 => 1 datadog.trace.laravelqueue_enabled => On => On datadog.trace.logs_analytics_enabled => Off => Off datadog.trace.logs_analytics_sample_rate => 1 => 1 datadog.trace.logs_enabled => On => On datadog.trace.lumen_analytics_enabled => Off => Off datadog.trace.lumen_analytics_sample_rate => 1 => 1 datadog.trace.lumen_enabled => On => On datadog.trace.measure_compile_time => On => On datadog.trace.memcache_analytics_enabled => Off => Off datadog.trace.memcache_analytics_sample_rate => 1 => 1 datadog.trace.memcache_enabled => On => On datadog.trace.memcached_analytics_enabled => Off => Off datadog.trace.memcached_analytics_sample_rate => 1 => 1 datadog.trace.memcached_enabled => On => On datadog.trace.memory_limit => no value => no value datadog.trace.mongo_analytics_enabled => Off => Off datadog.trace.mongo_analytics_sample_rate => 1 => 1 datadog.trace.mongo_enabled => On => On datadog.trace.mongodb_analytics_enabled => Off => Off datadog.trace.mongodb_analytics_sample_rate => 1 => 1 datadog.trace.mongodb_enabled => On => On datadog.trace.mysqli_analytics_enabled => Off => Off datadog.trace.mysqli_analytics_sample_rate => 1 => 1 datadog.trace.mysqli_enabled => On => On datadog.trace.nette_analytics_enabled => Off => Off datadog.trace.nette_analytics_sample_rate => 1 => 1 datadog.trace.nette_enabled => On => On datadog.trace.obfuscation_query_stringregexp => (?i)(?:p(?:ass)?w(?:or)?d|pass(?:?phrase)?|secret|(?:api?|private?|public?|access?|secret?)key(?:?id)?|token|consumer?(?:id|key|secret)|sign(?:ed|ature)?|auth(?:entication|orization)?)(?:(?:\s|%20)(?:=|%3D)[^&]+|(?:"|%22)(?:\s|%20)(?::|%3A)(?:\s|%20)(?:"|%22)(?:%2[^2]|%[^2]|[^"%])+(?:"|%22))|bearer(?:\s|%20)+[a-z0-9.-]|token(?::|%3A)[a-z0-9]{13}|gh[opsu][0-9a-zA-Z]{36}|eyI-L+.eyI-L+(?:.(?:[\w.+/=-]|%3D|%2F|%2B)+)?|[-]{5}BEGIN(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY[-]{5}[^-]+[-]{5}END(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY|ssh-rsa(?:\s|%20)(?:[a-z0-9/.+]|%2F|%5C|%2B){100,} => (?i)(?:p(?:ass)?w(?:or)?d|pass(?:?phrase)?|secret|(?:api?|private?|public?|access?|secret_?)key(?:?id)?|token|consumer?(?:id|key|secret)|sign(?:ed|ature)?|auth(?:entication|orization)?)(?:(?:\s|%20)(?:=|%3D)[^&]+|(?:"|%22)(?:\s|%20)(?::|%3A)(?:\s|%20)(?:"|%22)(?:%2[^2]|%[^2]|[^"%])+(?:"|%22))|bearer(?:\s|%20)+[a-z0-9.-]|token(?::|%3A)[a-z0-9]{13}|gh[opsu][0-9a-zA-Z]{36}|eyI-L+.eyI-L+(?:.(?:[\w.+/=-]|%3D|%2F|%2B)+)?|[-]{5}BEGIN(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY[-]{5}[^-]+[-]{5}END(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY|ssh-rsa(?:\s|%20)(?:[a-z0-9/.+]|%2F|%5C|%2B){100,} datadog.trace.once_logs => On => On datadog.trace.pcntl_analytics_enabled => Off => Off datadog.trace.pcntl_analytics_sample_rate => 1 => 1 datadog.trace.pcntl_enabled => On => On datadog.trace.pdo_analytics_enabled => Off => Off datadog.trace.pdo_analytics_sample_rate => 1 => 1 datadog.trace.pdo_enabled => On => On datadog.trace.peer_service_defaults_enabled => Off => Off datadog.trace.peer_service_mapping => no value => no value datadog.trace.phpredis_analytics_enabled => Off => Off datadog.trace.phpredis_analytics_sample_rate => 1 => 1 datadog.trace.phpredis_enabled => On => On datadog.trace.predis_analytics_enabled => Off => Off datadog.trace.predis_analytics_sample_rate => 1 => 1 datadog.trace.predis_enabled => On => On datadog.trace.propagate_service => Off => Off datadog.trace.propagate_user_id_default => Off => Off datadog.trace.propagation_style => tracecontext,Datadog => tracecontext,Datadog datadog.trace.propagation_style_extract => tracecontext,Datadog,B3,B3 single header => tracecontext,Datadog,B3,B3 single header datadog.trace.propagation_style_inject => tracecontext,Datadog => tracecontext,Datadog datadog.trace.psr18_analytics_enabled => Off => Off datadog.trace.psr18_analytics_sample_rate => 1 => 1 datadog.trace.psr18_enabled => On => On datadog.trace.rate_limit => 0 => 0 datadog.trace.redis_client_split_by_host => Off => Off datadog.trace.remove_autoinstrumentation_orphans => Off => Off datadog.trace.remove_integration_service_names_enabled => Off => Off datadog.trace.remove_root_span_laravel_queue => On => On datadog.trace.report_hostname => Off => Off datadog.trace.request_init_hook => /opt/datadog/dd-library/0.91.2/dd-trace-sources/bridge/dd_wrap_autoloader.php => /opt/datadog/dd-library/0.91.2/dd-trace-sources/bridge/dd_wrap_autoloader.php datadog.trace.resource_uri_fragment_regex => no value => no value datadog.trace.resource_uri_mapping_incoming => no value => no value datadog.trace.resource_uri_mapping_outgoing => no value => no value datadog.trace.resource_uri_query_param_allowed => no value => no value datadog.trace.retain_thread_capabilities => Off => Off datadog.trace.roadrunner_analytics_enabled => Off => Off datadog.trace.roadrunner_analytics_sample_rate => 1 => 1 datadog.trace.roadrunner_enabled => On => On datadog.trace.sample_rate => -1 => -1 datadog.trace.sampling_rules => [] => [] datadog.trace.shutdown_timeout => 5000 => 5000 datadog.trace.sidecar_trace_sender => Off => Off datadog.trace.slim_analytics_enabled => Off => Off datadog.trace.slim_analytics_sample_rate => 1 => 1 datadog.trace.slim_enabled => On => On datadog.trace.spans_limit => 1000 => 1000 datadog.trace.sqlsrv_analytics_enabled => Off => Off datadog.trace.sqlsrv_analytics_sample_rate => 1 => 1 datadog.trace.sqlsrv_enabled => On => On datadog.trace.startup_logs => On => On datadog.trace.symfony_analytics_enabled => Off => Off datadog.trace.symfony_analytics_sample_rate => 1 => 1 datadog.trace.symfony_enabled => On => On datadog.trace.traced_internal_functions => no value => no value datadog.trace.url_as_resource_names_enabled => On => On datadog.trace.warn_legacy_dd_trace => On => On datadog.trace.web_analytics_enabled => Off => Off datadog.trace.web_analytics_sample_rate => 1 => 1 datadog.trace.web_enabled => On => On datadog.trace.wordpress_additional_actions => no value => no value datadog.trace.wordpress_analytics_enabled => Off => Off datadog.trace.wordpress_analytics_sample_rate => 1 => 1 datadog.trace.wordpress_callbacks => Off => Off datadog.trace.wordpress_enabled => On => On datadog.trace.wordpress_enhanced_integration => Off => Off datadog.trace.x_datadog_tags_max_length => 512 => 512 datadog.trace.yii_analytics_enabled => Off => Off datadog.trace.yii_analytics_sample_rate => 1 => 1 datadog.trace.yii_enabled => On => On datadog.trace.zendframework_analytics_enabled => Off => Off datadog.trace.zendframework_analytics_sample_rate => 1 => 1 datadog.trace.zendframework_enabled => On => On datadog.version => no value => no value datadog.web_analytics_enabled => Off => Off datadog.web_analytics_sample_rate => 1 => 1 datadog.wordpress_analytics_enabled => Off => Off datadog.wordpress_analytics_sample_rate => 1 => 1 datadog.yii_analytics_enabled => Off => Off datadog.yii_analytics_sample_rate => 1 => 1 datadog.zendframework_analytics_enabled => Off => Off datadog.zendframework_analytics_sample_rate => 1 => 1 ddtrace.cgroup_file => /proc/self/cgroup => /proc/self/cgroup ddtrace.disable => Off => Off ddtrace.request_init_hook => /opt/datadog/dd-library/0.91.2/dd-trace-sources/bridge/dd_wrap_autoloader.php => /opt/datadog/dd-library/0.91.2/dd-trace-sources/bridge/dd_wrap_autoloader.php

Upgrading from

No response

bwoebi commented 10 months ago

Hey @zaneclaes,

Is there a specific way to reproduce this? Is it possible for you to somehow get a core dump and show me the backtrace? With just the information at hand there's little I can do :-/

Thanks!

bwoebi commented 10 months ago

For our purposes, the UnwindExit can be trivially caught without problems and does not crash:

$ DD_TRACE_CLI_ENABLED=1 php -r 'function x() { die(); } \DDTrace\trace_function("x", function($s) { var_dump($s->exception); }); x();'
object(UnwindExit)#7 (0) {
}
zaneclaes commented 10 months ago

@bwoebi unfortunately, per the OP, I failed to get a core dump on Kuberentes (despite running privileged mode and changing the core_pattern). I'm not entirely sure why none was generated; there simply were no dumps in /tmp.

That said, AFAICT your test case does not accurately reproduce the scenario described in the issue report. Note that the segfault is from php-fpm, and not from PHP itself. The PHP process exits just fine, and is not part of the bug report.

In fact, the following comment contains a rich discussion on the various problems and ways to fix the UnwindExit function (and the fact that the function is named identically reinforces my original suspicion that this is just a fix that needs to be ported over): https://github.com/open-telemetry/opentelemetry-php/issues/951#issuecomment-1590407907

bwoebi commented 10 months ago

@zaneclaes We do not have the same problem than OpenTelemetry here as their problem was that they have a specific way to restore exceptions, which is different here.

Hence, I'm pretty sure that we do have a different issue at hand, but only a core dump will be able to tell what exactly the issue is.

I'm not familiar with kubernetes, so I cannot help you with the core dump probably, but given that you have SIGSEGV - core dumped in your error log, it has been dumped somewhere. But I don't know where.

bwoebi commented 10 months ago

I actually found an issue, but it is only reproducible with datadog.trace.debug=1 (because it tries to print the UnwindExit then).

$ DD_TRACE_DEBUG=1 DD_TRACE_CLI_ENABLED=1 php -r 'function x() { die(); } \DDTrace\trace_function("x", function($s) { die(); }); x();'
AddressSanitizer:DEADLYSIGNAL
=================================================================
==18968==ERROR: AddressSanitizer: SEGV on unknown address 0x000fdf231f75 (pc 0xaaaab752b9c4 bp 0xfffff04c8b30 sp 0xfffff04c8b30 T0)
==18968==The signal is caused by a READ memory access.
    #0 0xaaaab752b9c3 in zend_hash_str_find_bucket /usr/local/src/php/Zend/zend_hash.c:715
    #1 0xaaaab753d4ef in zend_hash_str_find /usr/local/src/php/Zend/zend_hash.c:2582
    #2 0xffff8ce4704f in zai_symbol_lookup_table /home/circleci/app/tmp/build_extension/zend_abstract_interface/symbols/lookup.c:44
    #3 0xffff8ce487fb in zai_symbol_lookup_property_impl /home/circleci/app/tmp/build_extension/zend_abstract_interface/symbols/lookup.c:272
    #4 0xffff8ce492d3 in zai_symbol_lookup /home/circleci/app/tmp/build_extension/zend_abstract_interface/symbols/lookup.c:380
    #5 0xffff8ce35b5f in zai_symbol_lookup_property /home/circleci/app/tmp/build_extension/zend_abstract_interface/symbols/api/property.h:8
    #6 0xffff8ce35d8b in zai_exception_read_property /home/circleci/app/tmp/build_extension/zend_abstract_interface/exceptions/exceptions.h:38
    #7 0xffff8ce365c3 in zai_exception_message /home/circleci/app/tmp/build_extension/zend_abstract_interface/exceptions/exceptions.c:33
    #8 0xffff8cefac33 in dd_uhook_report_sandbox_error /home/circleci/app/tmp/build_extension/ext/hook/uhook.c:118
    #9 0xffff8cf05487 in dd_uhook_call /home/circleci/app/tmp/build_extension/ext/hook/uhook_legacy.c:81
    #10 0xffff8cf06dc7 in dd_uhook_end /home/circleci/app/tmp/build_extension/ext/hook/uhook_legacy.c:225
    #11 0xffff8ce411b3 in zai_hook_finish /home/circleci/app/tmp/build_extension/zend_abstract_interface/hook/hook.c:987
    #12 0xffff8ce217eb in zai_hook_safe_finish /home/circleci/app/tmp/build_extension/zend_abstract_interface/interceptor/php8/interceptor.c:49
    #13 0xffff8ce21b9f in zai_interceptor_observer_end_handler /home/circleci/app/tmp/build_extension/zend_abstract_interface/interceptor/php8/interceptor.c:141
    #14 0xaaaab7829c0f in call_end_observers /usr/local/src/php/Zend/zend_observer.c:274
    #15 0xaaaab7829ca3 in zend_observer_fcall_end /usr/local/src/php/Zend/zend_observer.c:283
    #16 0xaaaab75be2b7 in zend_dispatch_try_catch_finally_helper_SPEC /usr/local/src/php/Zend/zend_vm_execute.h:3149
    #17 0xaaaab75befcf in ZEND_HANDLE_EXCEPTION_SPEC_HANDLER /usr/local/src/php/Zend/zend_vm_execute.h:3230
    #18 0xaaaab77521cb in execute_ex /usr/local/src/php/Zend/zend_vm_execute.h:56156
    #19 0xaaaab77661f3 in zend_execute /usr/local/src/php/Zend/zend_vm_execute.h:60408
    #20 0xaaaab749ee9b in zend_eval_stringl /usr/local/src/php/Zend/zend_execute_API.c:1296
    #21 0xaaaab749f37b in zend_eval_stringl_ex /usr/local/src/php/Zend/zend_execute_API.c:1338
    #22 0xaaaab749f4af in zend_eval_string_ex /usr/local/src/php/Zend/zend_execute_API.c:1348
    #23 0xaaaab7995a6b in do_cli /usr/local/src/php/sapi/cli/php_cli.c:995
    #24 0xaaaab7997c6b in main /usr/local/src/php/sapi/cli/php_cli.c:1333
    #25 0xffff95889da3 in __libc_start_main ../csu/libc-start.c:308
    #26 0xaaaab60dd473  (/opt/php/debug-zts-asan/bin/php+0x5ad473)

I am, however, strongly suspecting that this is not related to your issue (or do you have debug enabled and do a die() in a trace hook callback?).

This is the only place I could spot, where we miss checking instanceof zend_ce_throwable before inspecting the exception.

zaneclaes commented 10 months ago

Again, I have to point out that you are not testing php-fpm @bwoebi ... unless I am seriously misunderstanding something. I'm not sure how you could expect to get a good reproduction of the error without using php-fpm, since the error is coming from php-fpm and there are NO php errors.

AFAIK, I do not have debug enabled (I checked my ENV variables, my pod annotations, and the config files).

zaneclaes commented 10 months ago

If you want to try to reproduce the error inside the docker container, here's a repository: https://github.com/zaneclaes/multipress/blob/master/Dockerfile#L60-L64

The commented lines would need to be uncommented. Then you'd run the docker container with the env variable SITES set to the domain name of the site you are hosting. However, it may be easier to setup your own php/nginx config, depending on your test case.

bwoebi commented 10 months ago

@zaneclaes Thanks for the dockerfile, is there a specific URL I need to hit within that container to reproduce the crash?

zaneclaes commented 9 months ago

Hmm. I know it was happening when I tried to access /wp-admin/, and that there were some warnings appearing in the PHP (a result of the 7->8 upgrade). The health-check endpoints worked fine though.

bwoebi commented 9 months ago

I've been happily clicking around in your containers /wp-admin, but not getting crashes. Can you please try to figure out when exactly it happens?

zaneclaes commented 9 months ago

Well, this is odd. I started the container using the same image that did not work before, with the same site that did not work before... and now it worked fine. I can only conclude that something about the warnings/errors generated by the PHP8 upgrade somehow triggered this problem. The only thing I did was to fix those upgrade problems and then reintroduce the tracer 🤷 I guess I'll go ahead and close this.