search

DataDog / dd-trace-php

Datadog PHP Clients
https://docs.datadoghq.com/tracing/setup/php
Other
497 stars 155 forks source link

[BUG] dd-trace causes segfault when used with opcache + preload w/ autoload on PHP 8.1 (Laravel) #1795

Closed phroggyy closed 1 year ago

phroggyy commented 2 years ago

Bug description

When having ddtrace enabled, Laravel applications running on Alpine Linux 3.16, on the linux/amd64 platform in Docker, with PHP 8.1, will segfault upon executing a preload.php (as referenced in opcache config) which requires the vendor/autoload.php.

I have a complete reproduction including steps at my reproduction repo: https://github.com/phroggyy/dd-trace-preload-segfault-repro.

In particular, see the preload.php, opcache config and Dockerfile for the most relevant parts.

Relates to https://github.com/DataDog/dd-trace-php/issues/1042.

I have also concluded that the extension works as expected in PHP 8.0.

Context (why autoload.php)

We rely on Composer's autoloader to be able to preload our entire application into memory. Since we require all application files (with a few exceptions that cause issues due to missing files in prod envs), we need the autoloader to be registered in order to use require_once properly, as that will also properly require chained dependencies.

Our entire preload script that we use in production looks as follows, but note that reproduction occurs with just the first line:

preload.php paths = $paths; // We'll use composer's classmap // to easily find which classes to autoload, // based on their filename $classMap = require __DIR__ . '/vendor/composer/autoload_classmap.php'; $this->fileMap = array_flip($classMap); } public function paths(string ...$paths): Preloader { $this->paths = array_merge( $this->paths, $paths ); return $this; } public function ignoreClasses(string ...$names): Preloader { $this->ignoredClasses = array_merge( $this->ignoredClasses, $names ); return $this; } public function ignorePaths(string ...$paths): Preloader { $this->ignoredPaths = array_merge( $this->ignoredPaths, $paths ); return $this; } public function load(): void { // We'll loop over all registered paths // and load them one by one foreach ($this->paths as $path) { $this->loadPath(rtrim($path, '/')); } $count = self::$count; } private function loadPath(string $path): void { // If the current path is a directory, // we'll load all files in it if (is_dir($path)) { $this->loadDir($path); return; } // Otherwise we'll just load this one file $this->loadFile($path); } private function loadDir(string $path): void { $handle = opendir($path); // We'll loop over all files and directories // in the current path, // and load them one by one while ($file = readdir($handle)) { if (in_array($file, ['.', '..'])) { continue; } $this->loadPath("{$path}/{$file}"); } closedir($handle); } private function loadFile(string $path): void { // We resolve the classname from composer's autoload mapping $class = $this->fileMap[$path] ?? null; // And use it to make sure the class, or the file pattern, shouldn't be ignored if ($this->shouldIgnoreClass($class) || $this->shouldIgnorePath($path)) { return; } // Finally we require the path, // causing all its dependencies to be loaded as well require_once($path); self::$count++; } private function shouldIgnoreClass(?string $name): bool { if ($name === null) { return true; } foreach ($this->ignoredClasses as $ignore) { if (strpos($name, $ignore) === 0) { return true; } } return false; } private function shouldIgnorePath(string $path): bool { foreach ($this->ignoredPaths as $ignoredPath) { $ignoredPath = preg_quote($ignoredPath, '/'); if (preg_match("/$ignoredPath/", $path) === 1) { return true; } } return false; } } (new Preloader()) ->paths( __DIR__ . '/vendor/laravel/framework', __DIR__ . '/app/Providers', __DIR__ . '/app/Http/Middleware', __DIR__ . '/app/Http/Controllers' ) ->ignorePaths( 'Illuminate/Foundation/Testing', 'Illuminate/Database/PDO', 'Illuminate/Database/DBAL', 'Illuminate/Testing', ) ->ignoreClasses( \Illuminate\Filesystem\Cache::class, \Illuminate\Log\LogManager::class, \Illuminate\Http\Testing\File::class, \Illuminate\Http\UploadedFile::class, \Illuminate\Support\Carbon::class ) ->load(); ``` ### PHP version 8.1 ### Tracer version 0.81.1 ### Installed extensions ``` [PHP Modules] Core ctype curl date ddtrace dom fileinfo filter ftp hash iconv json libxml mbstring mysqlnd openssl pcre PDO pdo_sqlite Phar posix readline Reflection session SimpleXML sodium SPL sqlite3 standard tokenizer xml xmlreader xmlwriter Zend OPcache zlib [Zend Modules] Zend OPcache ddtrace ``` ### OS info ``` NAME="Alpine Linux" VERSION_ID=3.16.3 PRETTY_NAME="Alpine Linux v3.16" ``` ### Diagnostics and configuration #### Output of phpinfo() (ddtrace >= 0.47.0) ``` ddtrace Datadog PHP tracer extension For help, check out the documentation at https://docs.datadoghq.com/tracing/languages/php/ (c) Datadog 2020 Datadog tracing support => disabled Version => 0.81.1 DATADOG TRACER CONFIGURATION => { "date": "2022-11-14T18:37:33Z", "os_name": "Linux fccd33590a92 5.15.49-linuxkit #1 SMP PREEMPT Tue Sep 13 07:51:32 UTC 2022 aarch64", "os_version": "5.15.49-linuxkit", "version": "0.81.1", "lang": "php", "lang_version": "8.1.8", "env": null, "enabled": true, "service": null, "enabled_cli": false, "agent_url": "http:\/\/host.docker.internal:8126", "debug": true, "analytics_enabled": false, "sample_rate": 1, "sampling_rules": [], "tags": [], "service_mapping": [], "distributed_tracing_enabled": true, "priority_sampling_enabled": true, "dd_version": null, "architecture": "aarch64", "sapi": "cli", "datadog.trace.request_init_hook": "\/opt\/datadog\/dd-library\/0.81.1\/dd-trace-sources\/bridge\/dd_wrap_autoloader.php", "open_basedir_configured": false, "uri_fragment_regex": null, "uri_mapping_incoming": null, "uri_mapping_outgoing": null, "auto_flush_enabled": false, "generate_root_span": true, "http_client_split_by_domain": false, "measure_compile_time": true, "report_hostname_on_root_span": false, "traced_internal_functions": null, "auto_prepend_file_configured": false, "integrations_disabled": "default", "enabled_from_env": false, "opcache.file_cache": null } Directive => Local Value => Master Value ddtrace.disable => Off => Off ddtrace.cgroup_file => /proc/self/cgroup => /proc/self/cgroup datadog.trace.request_init_hook => /opt/datadog/dd-library/0.81.1/dd-trace-sources/bridge/dd_wrap_autoloader.php => /opt/datadog/dd-library/0.81.1/dd-trace-sources/bridge/dd_wrap_autoloader.php ddtrace.request_init_hook => /opt/datadog/dd-library/0.81.1/dd-trace-sources/bridge/dd_wrap_autoloader.php => /opt/datadog/dd-library/0.81.1/dd-trace-sources/bridge/dd_wrap_autoloader.php datadog.trace.agent_url => no value => no value datadog.agent_host => host.docker.internal => host.docker.internal datadog.dogstatsd_url => no value => no value datadog.distributed_tracing => On => On datadog.dogstatsd_port => 8125 => 8125 datadog.env => no value => no value datadog.autofinish_spans => Off => Off datadog.trace.url_as_resource_names_enabled => On => On datadog.integrations_disabled => default => default datadog.priority_sampling => On => On datadog.service => no value => no value datadog.service_name => no value => no value datadog.service_mapping => no value => no value datadog.tags => no value => no value datadog.trace.global_tags => no value => no value datadog.trace.agent_port => 8126 => 8126 datadog.trace.analytics_enabled => Off => Off datadog.trace.auto_flush_enabled => Off => Off datadog.trace.cli_enabled => Off => Off datadog.trace.measure_compile_time => On => On datadog.trace.debug => On => On datadog.trace.enabled => Off => On datadog.trace.health_metrics_enabled => Off => Off datadog.trace.health_metrics_heartbeat_sample_rate => 0.001 => 0.001 datadog.trace.db_client_split_by_instance => Off => Off datadog.trace.http_client_split_by_domain => Off => Off datadog.trace.redis_client_split_by_host => Off => Off datadog.trace.memory_limit => no value => no value datadog.trace.report_hostname => Off => Off datadog.trace.resource_uri_fragment_regex => no value => no value datadog.trace.resource_uri_mapping_incoming => no value => no value datadog.trace.resource_uri_mapping_outgoing => no value => no value datadog.trace.resource_uri_query_param_allowed => no value => no value datadog.trace.http_url_query_param_allowed => * => * datadog.trace.rate_limit => 0 => 0 datadog.trace.sample_rate => 1 => 1 datadog.sampling_rate => 1 => 1 datadog.trace.sampling_rules => [] => [] datadog.span_sampling_rules => [] => [] datadog.span_sampling_rules_file => no value => no value datadog.trace.header_tags => no value => no value datadog.trace.x_datadog_tags_max_length => 512 => 512 datadog.trace.propagate_service => Off => Off datadog.propagation_style_extract => Datadog,B3,B3 single header => Datadog,B3,B3 single header datadog.propagation_style_inject => Datadog => Datadog datadog.trace.traced_internal_functions => no value => no value datadog.trace.agent_timeout => 500 => 500 datadog.trace.agent_connect_timeout => 100 => 100 datadog.trace.debug_prng_seed => -1 => -1 datadog.log_backtrace => Off => Off datadog.trace.generate_root_span => On => On datadog.trace.spans_limit => 1000 => 1000 datadog.trace.agent_max_consecutive_failures => 3 => 3 datadog.trace.agent_attempt_retry_time_msec => 5000 => 5000 datadog.trace.bgs_connect_timeout => 2000 => 2000 datadog.trace.bgs_timeout => 5000 => 5000 datadog.trace.agent_flush_interval => 5000 => 5000 datadog.trace.agent_flush_after_n_requests => 10 => 10 datadog.trace.shutdown_timeout => 5000 => 5000 datadog.trace.startup_logs => On => On datadog.trace.agent_debug_verbose_curl => Off => Off datadog.trace.debug_curl_output => Off => Off datadog.trace.beta_high_memory_pressure_percent => 80 => 80 datadog.trace.warn_legacy_dd_trace => On => On datadog.trace.retain_thread_capabilities => Off => Off datadog.version => no value => no value datadog.trace.obfuscation_query_string_regexp => (?i)(?:p(?:ass)?w(?:or)?d|pass(?:_?phrase)?|secret|(?:api_?|private_?|public_?|access_?|secret_?)key(?:_?id)?|token|consumer_?(?:id|key|secret)|sign(?:ed|ature)?|auth(?:entication|orization)?)(?:(?:\s|%20)*(?:=|%3D)[^&]+|(?:"|%22)(?:\s|%20)*(?::|%3A)(?:\s|%20)*(?:"|%22)(?:%2[^2]|%[^2]|[^"%])+(?:"|%22))|bearer(?:\s|%20)+[a-z0-9\._\-]|token(?::|%3A)[a-z0-9]{13}|gh[opsu]_[0-9a-zA-Z]{36}|ey[I-L](?:[\w=-]|%3D)+\.ey[I-L](?:[\w=-]|%3D)+(?:\.(?:[\w.+\/=-]|%3D|%2F|%2B)+)?|[\-]{5}BEGIN(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY[\-]{5}[^\-]+[\-]{5}END(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY|ssh-rsa(?:\s|%20)*(?:[a-z0-9\/\.+]|%2F|%5C|%2B){100,} => (?i)(?:p(?:ass)?w(?:or)?d|pass(?:_?phrase)?|secret|(?:api_?|private_?|public_?|access_?|secret_?)key(?:_?id)?|token|consumer_?(?:id|key|secret)|sign(?:ed|ature)?|auth(?:entication|orization)?)(?:(?:\s|%20)*(?:=|%3D)[^&]+|(?:"|%22)(?:\s|%20)*(?::|%3A)(?:\s|%20)*(?:"|%22)(?:%2[^2]|%[^2]|[^"%])+(?:"|%22))|bearer(?:\s|%20)+[a-z0-9\._\-]|token(?::|%3A)[a-z0-9]{13}|gh[opsu]_[0-9a-zA-Z]{36}|ey[I-L](?:[\w=-]|%3D)+\.ey[I-L](?:[\w=-]|%3D)+(?:\.(?:[\w.+\/=-]|%3D|%2F|%2B)+)?|[\-]{5}BEGIN(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY[\-]{5}[^\-]+[\-]{5}END(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY|ssh-rsa(?:\s|%20)*(?:[a-z0-9\/\.+]|%2F|%5C|%2B){100,} datadog.trace.forked_process => On => On datadog.trace.agent_max_payload_size => 52428800 => 52428800 datadog.trace.agent_stack_initial_size => 131072 => 131072 datadog.trace.agent_stack_backlog => 12 => 12 datadog.trace.cakephp_enabled => On => On datadog.trace.cakephp_analytics_enabled => Off => Off datadog.cakephp_analytics_enabled => Off => Off datadog.trace.cakephp_analytics_sample_rate => 1 => 1 datadog.cakephp_analytics_sample_rate => 1 => 1 datadog.trace.codeigniter_enabled => On => On datadog.trace.codeigniter_analytics_enabled => Off => Off datadog.codeigniter_analytics_enabled => Off => Off datadog.trace.codeigniter_analytics_sample_rate => 1 => 1 datadog.codeigniter_analytics_sample_rate => 1 => 1 datadog.trace.curl_enabled => On => On datadog.trace.curl_analytics_enabled => Off => Off datadog.curl_analytics_enabled => Off => Off datadog.trace.curl_analytics_sample_rate => 1 => 1 datadog.curl_analytics_sample_rate => 1 => 1 datadog.trace.elasticsearch_enabled => On => On datadog.trace.elasticsearch_analytics_enabled => Off => Off datadog.elasticsearch_analytics_enabled => Off => Off datadog.trace.elasticsearch_analytics_sample_rate => 1 => 1 datadog.elasticsearch_analytics_sample_rate => 1 => 1 datadog.trace.eloquent_enabled => On => On datadog.trace.eloquent_analytics_enabled => Off => Off datadog.eloquent_analytics_enabled => Off => Off datadog.trace.eloquent_analytics_sample_rate => 1 => 1 datadog.eloquent_analytics_sample_rate => 1 => 1 datadog.trace.guzzle_enabled => On => On datadog.trace.guzzle_analytics_enabled => Off => Off datadog.guzzle_analytics_enabled => Off => Off datadog.trace.guzzle_analytics_sample_rate => 1 => 1 datadog.guzzle_analytics_sample_rate => 1 => 1 datadog.trace.laravel_enabled => On => On datadog.trace.laravel_analytics_enabled => Off => Off datadog.laravel_analytics_enabled => Off => Off datadog.trace.laravel_analytics_sample_rate => 1 => 1 datadog.laravel_analytics_sample_rate => 1 => 1 datadog.trace.lumen_enabled => On => On datadog.trace.lumen_analytics_enabled => Off => Off datadog.lumen_analytics_enabled => Off => Off datadog.trace.lumen_analytics_sample_rate => 1 => 1 datadog.lumen_analytics_sample_rate => 1 => 1 datadog.trace.memcached_enabled => On => On datadog.trace.memcached_analytics_enabled => Off => Off datadog.memcached_analytics_enabled => Off => Off datadog.trace.memcached_analytics_sample_rate => 1 => 1 datadog.memcached_analytics_sample_rate => 1 => 1 datadog.trace.mongo_enabled => On => On datadog.trace.mongo_analytics_enabled => Off => Off datadog.mongo_analytics_enabled => Off => Off datadog.trace.mongo_analytics_sample_rate => 1 => 1 datadog.mongo_analytics_sample_rate => 1 => 1 datadog.trace.mongodb_enabled => On => On datadog.trace.mongodb_analytics_enabled => Off => Off datadog.mongodb_analytics_enabled => Off => Off datadog.trace.mongodb_analytics_sample_rate => 1 => 1 datadog.mongodb_analytics_sample_rate => 1 => 1 datadog.trace.mysqli_enabled => On => On datadog.trace.mysqli_analytics_enabled => Off => Off datadog.mysqli_analytics_enabled => Off => Off datadog.trace.mysqli_analytics_sample_rate => 1 => 1 datadog.mysqli_analytics_sample_rate => 1 => 1 datadog.trace.nette_enabled => On => On datadog.trace.nette_analytics_enabled => Off => Off datadog.nette_analytics_enabled => Off => Off datadog.trace.nette_analytics_sample_rate => 1 => 1 datadog.nette_analytics_sample_rate => 1 => 1 datadog.trace.pcntl_enabled => On => On datadog.trace.pcntl_analytics_enabled => Off => Off datadog.pcntl_analytics_enabled => Off => Off datadog.trace.pcntl_analytics_sample_rate => 1 => 1 datadog.pcntl_analytics_sample_rate => 1 => 1 datadog.trace.pdo_enabled => On => On datadog.trace.pdo_analytics_enabled => Off => Off datadog.pdo_analytics_enabled => Off => Off datadog.trace.pdo_analytics_sample_rate => 1 => 1 datadog.pdo_analytics_sample_rate => 1 => 1 datadog.trace.phpredis_enabled => On => On datadog.trace.phpredis_analytics_enabled => Off => Off datadog.phpredis_analytics_enabled => Off => Off datadog.trace.phpredis_analytics_sample_rate => 1 => 1 datadog.phpredis_analytics_sample_rate => 1 => 1 datadog.trace.predis_enabled => On => On datadog.trace.predis_analytics_enabled => Off => Off datadog.predis_analytics_enabled => Off => Off datadog.trace.predis_analytics_sample_rate => 1 => 1 datadog.predis_analytics_sample_rate => 1 => 1 datadog.trace.slim_enabled => On => On datadog.trace.slim_analytics_enabled => Off => Off datadog.slim_analytics_enabled => Off => Off datadog.trace.slim_analytics_sample_rate => 1 => 1 datadog.slim_analytics_sample_rate => 1 => 1 datadog.trace.symfony_enabled => On => On datadog.trace.symfony_analytics_enabled => Off => Off datadog.symfony_analytics_enabled => Off => Off datadog.trace.symfony_analytics_sample_rate => 1 => 1 datadog.symfony_analytics_sample_rate => 1 => 1 datadog.trace.web_enabled => On => On datadog.trace.web_analytics_enabled => Off => Off datadog.web_analytics_enabled => Off => Off datadog.trace.web_analytics_sample_rate => 1 => 1 datadog.web_analytics_sample_rate => 1 => 1 datadog.trace.wordpress_enabled => On => On datadog.trace.wordpress_analytics_enabled => Off => Off datadog.wordpress_analytics_enabled => Off => Off datadog.trace.wordpress_analytics_sample_rate => 1 => 1 datadog.wordpress_analytics_sample_rate => 1 => 1 datadog.trace.yii_enabled => On => On datadog.trace.yii_analytics_enabled => Off => Off datadog.yii_analytics_enabled => Off => Off datadog.trace.yii_analytics_sample_rate => 1 => 1 datadog.yii_analytics_sample_rate => 1 => 1 datadog.trace.zendframework_enabled => On => On datadog.trace.zendframework_analytics_enabled => Off => Off datadog.zendframework_analytics_enabled => Off => Off datadog.trace.zendframework_analytics_sample_rate => 1 => 1 datadog.zendframework_analytics_sample_rate => 1 => 1 ```
bwoebi commented 2 years ago

Hey @phroggyy,

thanks a lot for the reproducer, I'm able to reproduce it and looking into it :-)

bwoebi commented 2 years ago

There was a change introduced in PHP 8.1, which causes differences in shutdown execution specifically for preloading. Reported at https://github.com/php/php-src/issues/9957.

I'll check whether there's anything reasonable we can do to mitigate this bug on our side.

phroggyy commented 2 years ago

@bwoebi if it's not too much to ask, I'd love to understand how you debugged this - are there some good resources to understanding those internals (in this case, particularly the memory management of custom objects)? Would love to be able to contribute to both this extension and others when I find bugs (even though in this case it's in PHP), so any references would be much appreciated! (both on debugging, and just reference to the diff functions utilised)

bwoebi commented 2 years ago

In this particular instance it's pretty mundane, I just executed USE_ZEND_ALLOC=0 valgrind php-fpm, and from the resulting stacktrace (which is referenced in the php-src issue), I was able to conclude the issue immediately. I just had to infer a couple ??? by looking at the code because these docker containers are provided without debug symbols, sadly.