[BUG] SIGSEGV trying to read DD_DOGSTATSD_URL

[bruecktech]

Bug description

We are experience segfaults caused by the datadog php tracer that seem to be related to the DD_DOGSTATSD_URL env variable. We are not setting this env var anywhere. This is the stacktrace pulled from a coredump

bash-5.1# gdb /usr/local/sbin/php-fpm core.33
GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-alpine-linux-musl".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/sbin/php-fpm...
(No debugging symbols found in /usr/local/sbin/php-fpm)

warning: Can't open file /dev/zero (deleted) during file-backed mapping note processing
[New LWP 33]
[New LWP 51]
[New LWP 50]
[New LWP 53]
[New LWP 173]
[New LWP 52]
[New LWP 54]
[New LWP 55]

Core was generated by `php-fpm: pool www      '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007ff7603ea904 in strncmp (_l=_l@entry=0x7ff75f7e3d30 <zai_config_memoized_entries+1136> "DD_DOGSTATSD_URL",
    _r=0x7ff7464e5960 <error: Cannot access memory at address 0x7ff7464e5960>, n=15, n@entry=16) at src/string/strncmp.c:7
7   src/string/strncmp.c: No such file or directory.
[Current thread is 1 (LWP 33)]
(gdb) bt
#0  0x00007ff7603ea904 in strncmp (_l=_l@entry=0x7ff75f7e3d30 <zai_config_memoized_entries+1136> "DD_DOGSTATSD_URL",
    _r=0x7ff7464e5960 <error: Cannot access memory at address 0x7ff7464e5960>, n=15, n@entry=16) at src/string/strncmp.c:7
#1  0x00007ff7603b6b62 in getenv (name=name@entry=0x7ff75f7e3d30 <zai_config_memoized_entries+1136> "DD_DOGSTATSD_URL") at src/env/getenv.c:10
#2  0x00007ff75f79b818 in zai_getenv_ex (name=..., buf=..., pre_rinit=pre_rinit@entry=false)
    at /home/circleci/datadog/tmp/build_extension/zend_abstract_interface/env/env.c:39
#3  0x00007ff75f79b39e in zai_config_ini_rinit () at /home/circleci/datadog/tmp/build_extension/zend_abstract_interface/config/config_ini.c:313
#4  0x00007ff75f799b14 in zai_config_rinit () at /home/circleci/datadog/tmp/build_extension/zend_abstract_interface/config/config.c:150
#5  0x00007ff75f7b3c8f in zm_activate_ddtrace (type=<optimized out>, module_number=<optimized out>) at /home/circleci/datadog/tmp/build_extension/ext/php7/ddtrace.c:639
#6  0x000055a9f31f1bb0 in zend_activate_modules ()
#7  0x000055a9f318ae76 in php_request_startup ()
#8  0x000055a9f2f2f0b6 in ?? ()
#9  0x00007ff7603b6a03 in libc_start_main_stage2 (main=0x55a9f2f2e2a0, argc=1, argv=0x7fff83513368) at src/env/__libc_start_main.c:94
#10 0x000055a9f2f2fe5e in _start ()

PHP version

bash-5.1# php -v
PHP 7.4.28 (cli) (built: Apr  5 2022 02:36:13) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.28, Copyright (c), by Zend Technologies
    with ddtrace v0.78.0, Copyright Datadog, by Datadog
    with datadog-profiling v0.9.0, Copyright Datadog, by Datadog
    with ddappsec v0.4.0, Copyright Datadog, by Datadog

Tracer version

bash-5.1# php -r "echo phpversion('ddtrace').PHP_EOL;"
0.78.0

Installed extensions

bash-5.1# php -m
[PHP Modules]
apcu
bcmath
calendar
Core
ctype
curl
datadog-profiling
date
ddappsec
ddtrace
dom
fileinfo
filter
ftp
gd
gettext
grpc
hash
iconv
imagick
intl
json
libxml
mbstring
mysqlnd
openssl
pcntl
pcre
PDO
pdo_mysql
pdo_pgsql
pdo_sqlite
pgsql
Phar
posix
protobuf
readline
redis
Reflection
session
SimpleXML
sockets
sodium
SPL
sqlite3
standard
tokenizer
xml
xmlreader
xmlwriter
Zend OPcache
zip
zlib

[Zend Modules]
Zend OPcache
datadog-profiling
ddappsec
ddtrace

OS info

bash-5.1# cat /etc/os-release | grep -E "(NAME)|(VERSION)"
NAME="Alpine Linux"
VERSION_ID=3.14.6
PRETTY_NAME="Alpine Linux v3.14"

Diagnostics and configuration

Output of phpinfo() (ddtrace >= 0.47.0)

bash-5.1# php --ri=ddtrace

ddtrace

Datadog PHP tracer extension
For help, check out the documentation at https://docs.datadoghq.com/tracing/languages/php/
(c) Datadog 2020

Datadog tracing support => disabled
Version => 0.78.0
DATADOG TRACER CONFIGURATION => {"date":"2022-08-29T08:59:22Z","os_name":"Linux 0af80f84f4be 5.10.104-linuxkit #1 SMP PREEMPT Thu Mar 17 17:05:54 UTC 2022 x86_64","os_version":"5.10.104-linuxkit","version":"0.78.0","lang":"php","lang_version":"7.4.30","env":null,"enabled":true,"service":null,"enabled_cli":false,"agent_url":"http:\/\/localhost:8126","debug":false,"analytics_enabled":false,"sample_rate":1,"sampling_rules":[],"tags":[],"service_mapping":[],"distributed_tracing_enabled":true,"priority_sampling_enabled":true,"dd_version":null,"architecture":"x86_64","sapi":"cli","datadog.trace.request_init_hook":"\/opt\/datadog\/dd-library\/0.78.0\/dd-trace-sources\/bridge\/dd_wrap_autoloader.php","open_basedir_configured":false,"uri_fragment_regex":null,"uri_mapping_incoming":null,"uri_mapping_outgoing":null,"auto_flush_enabled":false,"generate_root_span":true,"http_client_split_by_domain":false,"measure_compile_time":true,"report_hostname_on_root_span":false,"traced_internal_functions":null,"auto_prepend_file_configured":false,"integrations_disabled":"default","enabled_from_env":false,"opcache.file_cache":null,"agent_error":"Failed to connect to localhost port 8126 after 4 ms: Connection refused"}

Directive => Local Value => Master Value
ddtrace.disable => Off => Off
ddtrace.cgroup_file => /proc/self/cgroup => /proc/self/cgroup
datadog.trace.request_init_hook => /opt/datadog/dd-library/0.78.0/dd-trace-sources/bridge/dd_wrap_autoloader.php => /opt/datadog/dd-library/0.78.0/dd-trace-sources/bridge/dd_wrap_autoloader.php
ddtrace.request_init_hook => /opt/datadog/dd-library/0.78.0/dd-trace-sources/bridge/dd_wrap_autoloader.php => /opt/datadog/dd-library/0.78.0/dd-trace-sources/bridge/dd_wrap_autoloader.php
datadog.trace.agent_url => no value => no value
datadog.agent_host => no value => no value
datadog.dogstatsd_url => no value => no value
datadog.distributed_tracing => On => On
datadog.dogstatsd_port => 8125 => 8125
datadog.env => no value => no value
datadog.autofinish_spans => Off => Off
datadog.trace.url_as_resource_names_enabled => On => On
datadog.integrations_disabled => default => default
datadog.priority_sampling => On => On
datadog.service => no value => no value
datadog.service_name => no value => no value
datadog.service_mapping => no value => no value
datadog.tags => no value => no value
datadog.trace.global_tags => no value => no value
datadog.trace.agent_port => 8126 => 8126
datadog.trace.analytics_enabled => Off => Off
datadog.trace.auto_flush_enabled => Off => Off
datadog.trace.cli_enabled => Off => Off
datadog.trace.measure_compile_time => On => On
datadog.trace.debug => Off => Off
datadog.trace.enabled => Off => On
datadog.trace.health_metrics_enabled => Off => Off
datadog.trace.health_metrics_heartbeat_sample_rate => 0.001 => 0.001
datadog.trace.db_client_split_by_instance => Off => Off
datadog.trace.http_client_split_by_domain => Off => Off
datadog.trace.redis_client_split_by_host => Off => Off
datadog.trace.memory_limit => no value => no value
datadog.trace.report_hostname => Off => Off
datadog.trace.resource_uri_fragment_regex => no value => no value
datadog.trace.resource_uri_mapping_incoming => no value => no value
datadog.trace.resource_uri_mapping_outgoing => no value => no value
datadog.trace.resource_uri_query_param_allowed => no value => no value
datadog.trace.http_url_query_param_allowed => * => *
datadog.trace.sample_rate => 1 => 1
datadog.sampling_rate => 1 => 1
datadog.trace.sampling_rules => [] => []
datadog.span_sampling_rules => [] => []
datadog.span_sampling_rules_file => no value => no value
datadog.trace.header_tags => no value => no value
datadog.trace.x_datadog_tags_max_length => 512 => 512
datadog.trace.propagate_service => Off => Off
datadog.propagation_style_extract => Datadog,B3,B3 single header => Datadog,B3,B3 single header
datadog.propagation_style_inject => Datadog => Datadog
datadog.trace.traced_internal_functions => no value => no value
datadog.trace.agent_timeout => 500 => 500
datadog.trace.agent_connect_timeout => 100 => 100
datadog.trace.debug_prng_seed => -1 => -1
datadog.log_backtrace => Off => Off
datadog.trace.generate_root_span => On => On
datadog.trace.spans_limit => 1000 => 1000
datadog.trace.agent_max_consecutive_failures => 3 => 3
datadog.trace.agent_attempt_retry_time_msec => 5000 => 5000
datadog.trace.bgs_connect_timeout => 2000 => 2000
datadog.trace.bgs_timeout => 5000 => 5000
datadog.trace.agent_flush_interval => 5000 => 5000
datadog.trace.agent_flush_after_n_requests => 10 => 10
datadog.trace.shutdown_timeout => 5000 => 5000
datadog.trace.startup_logs => On => On
datadog.trace.agent_debug_verbose_curl => Off => Off
datadog.trace.debug_curl_output => Off => Off
datadog.trace.beta_high_memory_pressure_percent => 80 => 80
datadog.trace.warn_legacy_dd_trace => On => On
datadog.trace.retain_thread_capabilities => Off => Off
datadog.version => no value => no value
datadog.trace.obfuscation_query_string_regexp => (?i)(?:p(?:ass)?w(?:or)?d|pass(?:_?phrase)?|secret|(?:api_?|private_?|public_?|access_?|secret_?)key(?:_?id)?|token|consumer_?(?:id|key|secret)|sign(?:ed|ature)?|auth(?:entication|orization)?)(?:(?:\s|%20)*(?:=|%3D)[^&]+|(?:"|%22)(?:\s|%20)*(?::|%3A)(?:\s|%20)*(?:"|%22)(?:%2[^2]|%[^2]|[^"%])+(?:"|%22))|bearer(?:\s|%20)+[a-z0-9\._\-]|token(?::|%3A)[a-z0-9]{13}|gh[opsu]_[0-9a-zA-Z]{36}|ey[I-L](?:[\w=-]|%3D)+\.ey[I-L](?:[\w=-]|%3D)+(?:\.(?:[\w.+\/=-]|%3D|%2F|%2B)+)?|[\-]{5}BEGIN(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY[\-]{5}[^\-]+[\-]{5}END(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY|ssh-rsa(?:\s|%20)*(?:[a-z0-9\/\.+]|%2F|%5C|%2B){100,} => (?i)(?:p(?:ass)?w(?:or)?d|pass(?:_?phrase)?|secret|(?:api_?|private_?|public_?|access_?|secret_?)key(?:_?id)?|token|consumer_?(?:id|key|secret)|sign(?:ed|ature)?|auth(?:entication|orization)?)(?:(?:\s|%20)*(?:=|%3D)[^&]+|(?:"|%22)(?:\s|%20)*(?::|%3A)(?:\s|%20)*(?:"|%22)(?:%2[^2]|%[^2]|[^"%])+(?:"|%22))|bearer(?:\s|%20)+[a-z0-9\._\-]|token(?::|%3A)[a-z0-9]{13}|gh[opsu]_[0-9a-zA-Z]{36}|ey[I-L](?:[\w=-]|%3D)+\.ey[I-L](?:[\w=-]|%3D)+(?:\.(?:[\w.+\/=-]|%3D|%2F|%2B)+)?|[\-]{5}BEGIN(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY[\-]{5}[^\-]+[\-]{5}END(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY|ssh-rsa(?:\s|%20)*(?:[a-z0-9\/\.+]|%2F|%5C|%2B){100,}
datadog.trace.client_ip_header_disabled => Off => Off
datadog.trace.client_ip_header => no value => no value
datadog.trace.forked_process => On => On
datadog.trace.cakephp_enabled => On => On
datadog.trace.cakephp_analytics_enabled => Off => Off
datadog.cakephp_analytics_enabled => Off => Off
datadog.trace.cakephp_analytics_sample_rate => 1 => 1
datadog.cakephp_analytics_sample_rate => 1 => 1
datadog.trace.codeigniter_enabled => On => On
datadog.trace.codeigniter_analytics_enabled => Off => Off
datadog.codeigniter_analytics_enabled => Off => Off
datadog.trace.codeigniter_analytics_sample_rate => 1 => 1
datadog.codeigniter_analytics_sample_rate => 1 => 1
datadog.trace.curl_enabled => On => On
datadog.trace.curl_analytics_enabled => Off => Off
datadog.curl_analytics_enabled => Off => Off
datadog.trace.curl_analytics_sample_rate => 1 => 1
datadog.curl_analytics_sample_rate => 1 => 1
datadog.trace.elasticsearch_enabled => On => On
datadog.trace.elasticsearch_analytics_enabled => Off => Off
datadog.elasticsearch_analytics_enabled => Off => Off
datadog.trace.elasticsearch_analytics_sample_rate => 1 => 1
datadog.elasticsearch_analytics_sample_rate => 1 => 1
datadog.trace.eloquent_enabled => On => On
datadog.trace.eloquent_analytics_enabled => Off => Off
datadog.eloquent_analytics_enabled => Off => Off
datadog.trace.eloquent_analytics_sample_rate => 1 => 1
datadog.eloquent_analytics_sample_rate => 1 => 1
datadog.trace.guzzle_enabled => On => On
datadog.trace.guzzle_analytics_enabled => Off => Off
datadog.guzzle_analytics_enabled => Off => Off
datadog.trace.guzzle_analytics_sample_rate => 1 => 1
datadog.guzzle_analytics_sample_rate => 1 => 1
datadog.trace.laravel_enabled => On => On
datadog.trace.laravel_analytics_enabled => Off => Off
datadog.laravel_analytics_enabled => Off => Off
datadog.trace.laravel_analytics_sample_rate => 1 => 1
datadog.laravel_analytics_sample_rate => 1 => 1
datadog.trace.lumen_enabled => On => On
datadog.trace.lumen_analytics_enabled => Off => Off
datadog.lumen_analytics_enabled => Off => Off
datadog.trace.lumen_analytics_sample_rate => 1 => 1
datadog.lumen_analytics_sample_rate => 1 => 1
datadog.trace.memcached_enabled => On => On
datadog.trace.memcached_analytics_enabled => Off => Off
datadog.memcached_analytics_enabled => Off => Off
datadog.trace.memcached_analytics_sample_rate => 1 => 1
datadog.memcached_analytics_sample_rate => 1 => 1
datadog.trace.mongo_enabled => On => On
datadog.trace.mongo_analytics_enabled => Off => Off
datadog.mongo_analytics_enabled => Off => Off
datadog.trace.mongo_analytics_sample_rate => 1 => 1
datadog.mongo_analytics_sample_rate => 1 => 1
datadog.trace.mongodb_enabled => On => On
datadog.trace.mongodb_analytics_enabled => Off => Off
datadog.mongodb_analytics_enabled => Off => Off
datadog.trace.mongodb_analytics_sample_rate => 1 => 1
datadog.mongodb_analytics_sample_rate => 1 => 1
datadog.trace.mysqli_enabled => On => On
datadog.trace.mysqli_analytics_enabled => Off => Off
datadog.mysqli_analytics_enabled => Off => Off
datadog.trace.mysqli_analytics_sample_rate => 1 => 1
datadog.mysqli_analytics_sample_rate => 1 => 1
datadog.trace.nette_enabled => On => On
datadog.trace.nette_analytics_enabled => Off => Off
datadog.nette_analytics_enabled => Off => Off
datadog.trace.nette_analytics_sample_rate => 1 => 1
datadog.nette_analytics_sample_rate => 1 => 1
datadog.trace.pcntl_enabled => On => On
datadog.trace.pcntl_analytics_enabled => Off => Off
datadog.pcntl_analytics_enabled => Off => Off
datadog.trace.pcntl_analytics_sample_rate => 1 => 1
datadog.pcntl_analytics_sample_rate => 1 => 1
datadog.trace.pdo_enabled => On => On
datadog.trace.pdo_analytics_enabled => Off => Off
datadog.pdo_analytics_enabled => Off => Off
datadog.trace.pdo_analytics_sample_rate => 1 => 1
datadog.pdo_analytics_sample_rate => 1 => 1
datadog.trace.phpredis_enabled => On => On
datadog.trace.phpredis_analytics_enabled => Off => Off
datadog.phpredis_analytics_enabled => Off => Off
datadog.trace.phpredis_analytics_sample_rate => 1 => 1
datadog.phpredis_analytics_sample_rate => 1 => 1
datadog.trace.predis_enabled => On => On
datadog.trace.predis_analytics_enabled => Off => Off
datadog.predis_analytics_enabled => Off => Off
datadog.trace.predis_analytics_sample_rate => 1 => 1
datadog.predis_analytics_sample_rate => 1 => 1
datadog.trace.slim_enabled => On => On
datadog.trace.slim_analytics_enabled => Off => Off
datadog.slim_analytics_enabled => Off => Off
datadog.trace.slim_analytics_sample_rate => 1 => 1
datadog.slim_analytics_sample_rate => 1 => 1
datadog.trace.symfony_enabled => On => On
datadog.trace.symfony_analytics_enabled => Off => Off
datadog.symfony_analytics_enabled => Off => Off
datadog.trace.symfony_analytics_sample_rate => 1 => 1
datadog.symfony_analytics_sample_rate => 1 => 1
datadog.trace.web_enabled => On => On
datadog.trace.web_analytics_enabled => Off => Off
datadog.web_analytics_enabled => Off => Off
datadog.trace.web_analytics_sample_rate => 1 => 1
datadog.web_analytics_sample_rate => 1 => 1
datadog.trace.wordpress_enabled => On => On
datadog.trace.wordpress_analytics_enabled => Off => Off
datadog.wordpress_analytics_enabled => Off => Off
datadog.trace.wordpress_analytics_sample_rate => 1 => 1
datadog.wordpress_analytics_sample_rate => 1 => 1
datadog.trace.yii_enabled => On => On
datadog.trace.yii_analytics_enabled => Off => Off
datadog.yii_analytics_enabled => Off => Off
datadog.trace.yii_analytics_sample_rate => 1 => 1
datadog.yii_analytics_sample_rate => 1 => 1
datadog.trace.zendframework_enabled => On => On
datadog.trace.zendframework_analytics_enabled => Off => Off
datadog.zendframework_analytics_enabled => Off => Off
datadog.trace.zendframework_analytics_sample_rate => 1 => 1
datadog.zendframework_analytics_sample_rate => 1 => 1

[bwoebi]

Hey @bruecktech,

This has nothing to do with DD_DOGSTATSD_URL itself, but only the fact that it happens to be the first env variable which is re-read on every request. I.e. the reading of the environment fails.

This error happens when the environment is (temporaily at least) in a corrupted state. I'm suspecting this is not an issue caused by ddtrace itself, but something different, but the issue manifesting with ddtrace, because it happens to reliably and often re-read some parts of the environment. At least, ddtrace never directly writes to the environment, it only reads it. As such it should not be responsible for the possible environment corruption.

If this error happens reasonably often, you could try using valgrind to figure out what freed the memory the environment was residing in. (run fpm with: USE_ZEND_ALLOC=0 valgrind --trace-children=yes -- php-fpm -F --fpm-config <CONFIG_FILE_PATH> <MORE_OPTIONS>)

[krakjoe]

Hi @bruecktech did you manage to get any more information for us ?

[jlesueur]

We are experiencing a similar segfault, but it's not predictable, happens only in production (maybe a race condition only occurs in busy processes). This is with php 8.0.25, an ddtrace 0.81.1:

(gdb) bt
#0  __GI_getenv (name=0x7fc99b2c85e2 <zai_config_memoized_entries+1186> "_DOGSTATSD_URL", name@entry=0x7fc99b2c85e0 <zai_config_memoized_entries+1184> "DD_DOGSTATSD_URL") at getenv.c:84
#1  0x00007fc99b088998 in zai_getenv_ex (name=..., buf=..., pre_rinit=pre_rinit@entry=false) at /home/circleci/datadog/tmp/build_extension/zend_abstract_interface/env/env.c:39
#2  0x00007fc99b088445 in zai_config_ini_rinit () at /home/circleci/datadog/tmp/build_extension/zend_abstract_interface/config/config_ini.c:329
#3  0x00007fc99b086c44 in zai_config_rinit () at /home/circleci/datadog/tmp/build_extension/zend_abstract_interface/config/config.c:151
#4  0x00007fc99b09c33f in ddtrace_activate () at /home/circleci/datadog/tmp/build_extension/ext/ddtrace.c:194
#5  0x0000556bdde9cd8e in zend_llist_apply (l=<optimized out>, func=func@entry=0x556bdde99260 <zend_extension_activator>) at ./Zend/zend_llist.c:182
#6  0x0000556bdde994aa in init_executor () at ./Zend/zend_execute_API.c:152
#7  0x0000556bddea8f47 in zend_activate () at ./Zend/zend.c:1212
#8  0x0000556bdde45309 in php_request_startup () at ./main/main.c:1725
#9  0x0000556bddcfa66a in main (argc=<optimized out>, argv=<optimized out>) at ./sapi/fpm/fpm/fpm_main.c:1852

Not sure how we can use valgrind in production, so this might not be much use to you. We're seeing another segfault as well, but the backtrace doesn't point to datadog. Just including here in case a coincidence is more than random.

(gdb) bt
#0  ps_files_close (data=0x0) at ./ext/session/mod_files.c:448
#1  ps_close_files (mod_data=0x5607000b0c60 <ps_globals+96>) at ./ext/session/mod_files.c:448
#2  0x00005606ffd7e032 in php_rshutdown_session_globals () at ./ext/session/session.c:133
#3  0x00005606ffd84a22 in zm_deactivate_session (type=<optimized out>, module_number=<optimized out>) at ./ext/session/session.c:2748
#4  0x00005606ffe9e614 in zend_deactivate_modules () at ./Zend/zend_API.c:2704
#5  0x00005606ffe35925 in php_request_shutdown (dummy=<optimized out>) at ./main/main.c:1830
#6  0x00005606ffcea74a in main (argc=<optimized out>, argv=<optimized out>) at ./sapi/fpm/fpm/fpm_main.c:1942