DataDog / dd-trace-php

Datadog PHP Clients
https://docs.datadoghq.com/tracing/setup/php
Other
488 stars 152 forks source link

[Bug]: CLI PHP Processes do not stop and with Apache (mod_php) the connection is closed unexpectedly #2629

Open elzekool opened 5 months ago

elzekool commented 5 months ago

Bug report

On our platform we see two behaviors that seem related to the DataDog PHP tracing agent:

Both issues only arrise sporadicly and we did not find an exact trigger that makes it hard to provide details on the second issue. We could connect with one stuck process with GDB and extracted the following stacktrace:

root@juju-machine-7:/home/ubuntu# strace -p 1333711 strace: Process 1333711 attached futex(0x7f9fa43899d0, FUTEX_WAIT, 1333735, NULL

0 __pthread_clockjoin_ex (threadid=140323631699712, thread_return=thread_return@entry=0x0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, block=block@entry=true) at pthread_join_common.c:145

1 0x00007f9fb0d1e9f3 in __pthread_join (threadid=, thread_return=thread_return@entry=0x0) at pthread_join.c:24

2 0x00007f9face1f39a in ddtrace_coms_flush_shutdown_writer_synchronous () at /home/circleci/datadog/tmp/build_extension/ext/coms.c:1237

3 0x00007f9face1f42f in ddtrace_coms_flush_shutdown_writer_synchronous () at /home/circleci/datadog/tmp/build_extension/ext/coms.c:1197

4 0x00007f9face157b2 in zm_shutdown_ddtrace (type=, module_number=25) at /home/circleci/datadog/tmp/build_extension/ext/ddtrace.c:897

5 0x00005566fde21e1f in ?? ()

6 0x00005566fde1ba50 in ?? ()

7 0x00005566fde2d639 in zend_hash_graceful_reverse_destroy ()

8 0x00005566fde1c9e7 in ?? ()

9 0x00005566fddbc6e2 in php_module_shutdown ()

10 0x00005566fdc82a09 in ?? ()

11 0x00007f9fb0d5c083 in __libc_start_main (main=0x5566fdc825e0, argc=3, argv=0x7fff3af15a38, init=, fini=, rtld_fini=, stack_end=0x7fff3af15a28) at ../csu/libc-start.c:308

12 0x00005566fdc831be in _start ()

I could trace this to the following function call: https://github.com/DataDog/dd-trace-php/blob/0.96.0/ext/coms.c#L1237

Additional information:

Agent version: 7.50.3 dd-trace: 0.96 PHP version: 7.4.3-4ubuntu2.20 Linux/Ubuntu version: Ubuntu 20.04.6 LTS

PHP version

7.4.3-4ubuntu2.20

Tracer or profiler version

0.96

Installed extensions

[PHP Modules] amqp apcu bcmath bz2 calendar Core ctype curl date dom enchant exif fileinfo filter ftp gd gearman geoip gettext gmp hash iconv igbinary imagick imap intl json ldap libxml mbstring memcache memcached mongodb msgpack mysqli mysqlnd oci8 odbc openssl pcntl pcre PDO pdo_mysql PDO_OCI PDO_ODBC pdo_pgsql pdo_sqlsrv pgsql Phar posix readline redis Reflection session shmop SimpleXML soap sockets sodium SPL sqlsrv ssh2 standard tidy tokenizer xml xmlreader xmlrpc xmlwriter xsl Zend OPcache zip zlib

[Zend Modules] Zend OPcache

Output of phpinfo()

ddtrace

Datadog PHP tracer extension For help, check out the documentation at https://docs.datadoghq.com/tracing/languages/php/ (c) Datadog 2020

Datadog tracing support => enabled Version => 0.96.0 DATADOG TRACER CONFIGURATION => { "date": "2024-04-17T07:37:59Z", "os_name": "Linux juju-machine-7 5.4.0-173-generic #191-Ubuntu SMP Fri Feb 2 13:55:07 UTC 2024 x86_64", "os_version": "5.4.0-173-generic", "version": "0.96.0", "lang": "php", "lang_version": "7.4.33", "env": "postnl.1c.eu-central.aws", "enabled": true, "service": "staging.shop.postnl.nl", "enabled_cli": true, "agent_url": "http:\/\/127.0.0.1:1400", "debug": false, "analytics_enabled": false, "sample_rate": -1, "sampling_rules": [], "tags": { "hosting.domain.main": "staging.shop.postnl.nl", "hosting.domain.alternate": "staging.shop.postnl.nl", "hosting.context": "acceptance", "hosting.source": "cli" }, "service_mapping": [], "distributed_tracing_enabled": true, "priority_sampling_enabled": true, "dd_version": "20240320105200", "architecture": "x86_64", "sapi": "cli", "datadog.trace.request_init_hook": "\/opt\/datadog\/dd-library\/0.96.0\/dd-trace-sources\/bridge\/dd_wrap_autoloader.php", "open_basedir_configured": false, "uri_fragment_regex": null, "uri_mapping_incoming": null, "uri_mapping_outgoing": null, "auto_flush_enabled": false, "generate_root_span": true, "http_client_split_by_domain": false, "measure_compile_time": true, "report_hostname_on_root_span": false, "traced_internal_functions": null, "auto_prepend_file_configured": true, "integrations_disabled": "default", "enabled_from_env": true, "opcache.file_cache": null }

                           Diagnostics                               

Diagnostic checks => passed

Directive => Local Value => Master Value ddtrace.disable => Off => Off ddtrace.cgroup_file => /proc/self/cgroup => /proc/self/cgroup datadog.trace.request_init_hook => /opt/datadog/dd-library/0.96.0/dd-trace-sources/bridge/dd_wrap_autoloader.php => /opt/datadog/dd-library/0.96.0/dd-trace-sources/bridge/dd_wrap_autoloader.php ddtrace.request_init_hook => /opt/datadog/dd-library/0.96.0/dd-trace-sources/bridge/dd_wrap_autoloader.php => /opt/datadog/dd-library/0.96.0/dd-trace-sources/bridge/dd_wrap_autoloader.php datadog.trace.agent_url => no value => no value datadog.agent_host => 127.0.0.1 => 127.0.0.1 datadog.dogstatsd_url => no value => no value datadog.api_key => no value => no value datadog.distributed_tracing => On => On datadog.dogstatsd_port => 1401 => 1401 datadog.env => postnl.1c.eu-central.aws => postnl.1c.eu-central.aws datadog.autofinish_spans => Off => Off datadog.trace.url_as_resource_names_enabled => On => On datadog.http_server_route_based_naming => On => On datadog.integrations_disabled => default => default datadog.priority_sampling => On => On datadog.service => staging.shop.postnl.nl => staging.shop.postnl.nl datadog.service_name => staging.shop.postnl.nl => staging.shop.postnl.nl datadog.service_mapping => no value => no value datadog.tags => hosting.domain.main:staging.shop.postnl.nl,hosting.domain.alternate:staging.shop.postnl.nl,hosting.context:acceptance,hosting.source:cli => hosting.domain.main:staging.shop.postnl.nl,hosting.domain.alternate:staging.shop.postnl.nl,hosting.context:acceptance,hosting.source:cli datadog.trace.global_tags => hosting.domain.main:staging.shop.postnl.nl,hosting.domain.alternate:staging.shop.postnl.nl,hosting.context:acceptance,hosting.source:cli => hosting.domain.main:staging.shop.postnl.nl,hosting.domain.alternate:staging.shop.postnl.nl,hosting.context:acceptance,hosting.source:cli datadog.trace.agent_port => 1400 => 1400 datadog.trace.analytics_enabled => Off => Off datadog.trace.append_trace_ids_to_logs => Off => Off datadog.trace.auto_flush_enabled => Off => Off datadog.trace.cli_enabled => On => On datadog.trace.measure_compile_time => On => On datadog.trace.debug => Off => Off datadog.trace.enabled => On => On datadog.instrumentation_telemetry_enabled => On => On datadog.trace.health_metrics_enabled => Off => Off datadog.trace.health_metrics_heartbeat_sample_rate => 0.001 => 0.001 datadog.trace.db_client_split_by_instance => Off => Off datadog.trace.http_client_split_by_domain => Off => Off datadog.trace.redis_client_split_by_host => Off => Off datadog.trace.memory_limit => no value => no value datadog.trace.report_hostname => Off => Off datadog.trace.flush_collect_cycles => Off => Off datadog.trace.laravel_queue_distributed_tracing => On => On datadog.trace.remove_root_span_laravel_queue => On => On datadog.trace.remove_autoinstrumentation_orphans => Off => Off datadog.trace.resource_uri_fragment_regex => no value => no value datadog.trace.resource_uri_mapping_incoming => no value => no value datadog.trace.resource_uri_mapping_outgoing => no value => no value datadog.trace.resource_uri_query_param_allowed => no value => no value datadog.trace.http_url_query_param_allowed => => datadog.trace.http_post_data_param_allowed => no value => no value datadog.trace.rate_limit => 0 => 0 datadog.trace.sample_rate => -1 => -1 datadog.sampling_rate => -1 => -1 datadog.trace.sampling_rules => [] => [] datadog.trace.sampling_rules_format => regex => regex datadog.span_sampling_rules => [] => [] datadog.span_sampling_rules_file => no value => no value datadog.trace.header_tags => no value => no value datadog.trace.x_datadog_tags_max_length => 512 => 512 datadog.trace.peer_service_mapping => no value => no value datadog.trace.peer_service_defaults_enabled => Off => Off datadog.trace.remove_integration_service_names_enabled => Off => Off datadog.trace.propagate_service => Off => Off datadog.trace.propagation_style_extract => tracecontext,Datadog,B3,B3 single header => tracecontext,Datadog,B3,B3 single header datadog.propagation_style_extract => tracecontext,Datadog,B3,B3 single header => tracecontext,Datadog,B3,B3 single header datadog.trace.propagation_style_inject => tracecontext,Datadog => tracecontext,Datadog datadog.propagation_style_inject => tracecontext,Datadog => tracecontext,Datadog datadog.trace.propagation_style => tracecontext,Datadog => tracecontext,Datadog datadog.trace.traced_internal_functions => no value => no value datadog.trace.agent_timeout => 500 => 500 datadog.trace.agent_connect_timeout => 100 => 100 datadog.trace.debug_prng_seed => -1 => -1 datadog.log_backtrace => Off => Off datadog.trace.generate_root_span => On => On datadog.trace.spans_limit => 1000 => 1000 datadog.trace.128_bit_traceid_generation_enabled => On => On datadog.trace.128_bit_traceid_logging_enabled => Off => Off datadog.trace.agent_max_consecutive_failures => 3 => 3 datadog.trace.agent_attempt_retry_time_msec => 5000 => 5000 datadog.trace.bgs_connect_timeout => 2000 => 2000 datadog.trace.bgs_timeout => 5000 => 5000 datadog.trace.agent_flush_interval => 5000 => 5000 datadog.trace.agent_flush_after_n_requests => 10 => 10 datadog.trace.shutdown_timeout => 5000 => 5000 datadog.trace.startup_logs => On => On datadog.trace.once_logs => On => On datadog.trace.agent_debug_verbose_curl => Off => Off datadog.trace.debug_curl_output => Off => Off datadog.trace.beta_high_memory_pressure_percent => 80 => 80 datadog.trace.sidecar_trace_sender => Off => Off datadog.trace.agentless => Off => Off datadog.trace.warn_legacy_dd_trace => On => On datadog.trace.retain_thread_capabilities => Off => Off datadog.version => 20240320105200 => 20240320105200 datadog.trace.obfuscation_query_stringregexp => (?i)(?:(?:"|%22)?)(?:(?:old[-]?|new[-]?)?p(?:ass)?w(?:or)?d(?:1|2)?|pass(?:[-]?phrase)?|secret|(?:api[-]?|private[-]?|public[-]?|access[-]?|secret[-]?|app(?:lication)?[-]?)key(?:[-]?id)?|token|consumer[-]?(?:id|key|secret)|sign(?:ed|ature)?|auth(?:entication|orization)?)(?:(?:\s|%20)(?:=|%3D)[^&]+|(?:"|%22)(?:\s|%20)(?::|%3A)(?:\s|%20)(?:"|%22)(?:%2[^2]|%[^2]|[^"%])+(?:"|%22))|(?:bearer(?:\s|%20)+[a-z0-9.-]+|token(?::|%3A)[a-z0-9]{13}|gh[opsu][0-9a-zA-Z]{36}|eyI-L+.eyI-L+(?:.(?:[\w.+/=-]|%3D|%2F|%2B)+)?|-{5}BEGIN(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY-{5}[^-]+-{5}END(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY(?:-{5})?(?:\n|%0A)?|(?:ssh-(?:rsa|dss)|ecdsa-[a-z0-9]+-[a-z0-9]+)(?:\s|%20|%09)+(?:[a-z0-9/.+]|%2F|%5C|%2B){100,}(?:=|%3D)(?:(?:\s|%20|%09)+[a-z0-9.-]+)?) => (?i)(?:(?:"|%22)?)(?:(?:old[-]?|new[-]?)?p(?:ass)?w(?:or)?d(?:1|2)?|pass(?:[-]?phrase)?|secret|(?:api[-]?|private[-]?|public[-]?|access[-]?|secret[-]?|app(?:lication)?[-]?)key(?:[-]?id)?|token|consumer[-]?(?:id|key|secret)|sign(?:ed|ature)?|auth(?:entication|orization)?)(?:(?:\s|%20)(?:=|%3D)[^&]+|(?:"|%22)(?:\s|%20)(?::|%3A)(?:\s|%20)(?:"|%22)(?:%2[^2]|%[^2]|[^"%])+(?:"|%22))|(?:bearer(?:\s|%20)+[a-z0-9.-]+|token(?::|%3A)[a-z0-9]{13}|gh[opsu][0-9a-zA-Z]{36}|eyI-L+.eyI-L+(?:.(?:[\w.+/=-]|%3D|%2F|%2B)+)?|-{5}BEGIN(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY-{5}[^-]+-{5}END(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY(?:-{5})?(?:\n|%0A)?|(?:ssh-(?:rsa|dss)|ecdsa-[a-z0-9]+-[a-z0-9]+)(?:\s|%20|%09)+(?:[a-z0-9/.+]|%2F|%5C|%2B){100,}(?:=|%3D)(?:(?:\s|%20|%09)+[a-z0-9._-]+)?) datadog.trace.client_ip_enabled => Off => Off datadog.trace.client_ip_header => no value => no value datadog.trace.forked_process => On => On datadog.trace.hook_limit => 1000 => 1000 datadog.trace.agent_max_payload_size => 52428800 => 52428800 datadog.trace.agent_stack_initial_size => 131072 => 131072 datadog.trace.agent_stack_backlog => 12 => 12 datadog.trace.propagate_user_id_default => Off => Off datadog.dbm_propagation_mode => disabled => disabled datadog.trace.wordpress_additional_actions => no value => no value datadog.trace.wordpress_callbacks => Off => Off datadog.trace.wordpress_enhanced_integration => Off => Off datadog.trace.otel_enabled => Off => Off datadog.trace.amqp_enabled => On => On datadog.trace.amqp_analytics_enabled => Off => Off datadog.amqp_analytics_enabled => Off => Off datadog.trace.amqp_analytics_sample_rate => 1 => 1 datadog.amqp_analytics_sample_rate => 1 => 1 datadog.trace.cakephp_enabled => On => On datadog.trace.cakephp_analytics_enabled => Off => Off datadog.cakephp_analytics_enabled => Off => Off datadog.trace.cakephp_analytics_sample_rate => 1 => 1 datadog.cakephp_analytics_sample_rate => 1 => 1 datadog.trace.codeigniter_enabled => On => On datadog.trace.codeigniter_analytics_enabled => Off => Off datadog.codeigniter_analytics_enabled => Off => Off datadog.trace.codeigniter_analytics_sample_rate => 1 => 1 datadog.codeigniter_analytics_sample_rate => 1 => 1 datadog.trace.curl_enabled => On => On datadog.trace.curl_analytics_enabled => Off => Off datadog.curl_analytics_enabled => Off => Off datadog.trace.curl_analytics_sample_rate => 1 => 1 datadog.curl_analytics_sample_rate => 1 => 1 datadog.trace.drupal_enabled => On => On datadog.trace.drupal_analytics_enabled => Off => Off datadog.drupal_analytics_enabled => Off => Off datadog.trace.drupal_analytics_sample_rate => 1 => 1 datadog.drupal_analytics_sample_rate => 1 => 1 datadog.trace.elasticsearch_enabled => On => On datadog.trace.elasticsearch_analytics_enabled => Off => Off datadog.elasticsearch_analytics_enabled => Off => Off datadog.trace.elasticsearch_analytics_sample_rate => 1 => 1 datadog.elasticsearch_analytics_sample_rate => 1 => 1 datadog.trace.eloquent_enabled => On => On datadog.trace.eloquent_analytics_enabled => Off => Off datadog.eloquent_analytics_enabled => Off => Off datadog.trace.eloquent_analytics_sample_rate => 1 => 1 datadog.eloquent_analytics_sample_rate => 1 => 1 datadog.trace.guzzle_enabled => On => On datadog.trace.guzzle_analytics_enabled => Off => Off datadog.guzzle_analytics_enabled => Off => Off datadog.trace.guzzle_analytics_sample_rate => 1 => 1 datadog.guzzle_analytics_sample_rate => 1 => 1 datadog.trace.laminas_enabled => On => On datadog.trace.laminas_analytics_enabled => Off => Off datadog.laminas_analytics_enabled => Off => Off datadog.trace.laminas_analytics_sample_rate => 1 => 1 datadog.laminas_analytics_sample_rate => 1 => 1 datadog.trace.laravel_enabled => On => On datadog.trace.laravel_analytics_enabled => Off => Off datadog.laravel_analytics_enabled => Off => Off datadog.trace.laravel_analytics_sample_rate => 1 => 1 datadog.laravel_analytics_sample_rate => 1 => 1 datadog.trace.laravelqueue_enabled => On => On datadog.trace.laravelqueue_analytics_enabled => Off => Off datadog.laravelqueue_analytics_enabled => Off => Off datadog.trace.laravelqueue_analytics_sample_rate => 1 => 1 datadog.laravelqueue_analytics_sample_rate => 1 => 1 datadog.trace.logs_enabled => On => On datadog.logs_injection => On => On datadog.trace.logs_analytics_enabled => Off => Off datadog.logs_analytics_enabled => Off => Off datadog.trace.logs_analytics_sample_rate => 1 => 1 datadog.logs_analytics_sample_rate => 1 => 1 datadog.trace.lumen_enabled => On => On datadog.trace.lumen_analytics_enabled => Off => Off datadog.lumen_analytics_enabled => Off => Off datadog.trace.lumen_analytics_sample_rate => 1 => 1 datadog.lumen_analytics_sample_rate => 1 => 1 datadog.trace.magento_enabled => On => On datadog.trace.magento_analytics_enabled => Off => Off datadog.magento_analytics_enabled => Off => Off datadog.trace.magento_analytics_sample_rate => 1 => 1 datadog.magento_analytics_sample_rate => 1 => 1 datadog.trace.memcache_enabled => On => On datadog.trace.memcache_analytics_enabled => Off => Off datadog.memcache_analytics_enabled => Off => Off datadog.trace.memcache_analytics_sample_rate => 1 => 1 datadog.memcache_analytics_sample_rate => 1 => 1 datadog.trace.memcached_enabled => On => On datadog.trace.memcached_analytics_enabled => Off => Off datadog.memcached_analytics_enabled => Off => Off datadog.trace.memcached_analytics_sample_rate => 1 => 1 datadog.memcached_analytics_sample_rate => 1 => 1 datadog.trace.mongo_enabled => On => On datadog.trace.mongo_analytics_enabled => Off => Off datadog.mongo_analytics_enabled => Off => Off datadog.trace.mongo_analytics_sample_rate => 1 => 1 datadog.mongo_analytics_sample_rate => 1 => 1 datadog.trace.mongodb_enabled => On => On datadog.trace.mongodb_analytics_enabled => Off => Off datadog.mongodb_analytics_enabled => Off => Off datadog.trace.mongodb_analytics_sample_rate => 1 => 1 datadog.mongodb_analytics_sample_rate => 1 => 1 datadog.trace.mysqli_enabled => On => On datadog.trace.mysqli_analytics_enabled => Off => Off datadog.mysqli_analytics_enabled => Off => Off datadog.trace.mysqli_analytics_sample_rate => 1 => 1 datadog.mysqli_analytics_sample_rate => 1 => 1 datadog.trace.nette_enabled => On => On datadog.trace.nette_analytics_enabled => Off => Off datadog.nette_analytics_enabled => Off => Off datadog.trace.nette_analytics_sample_rate => 1 => 1 datadog.nette_analytics_sample_rate => 1 => 1 datadog.trace.pcntl_enabled => On => On datadog.trace.pcntl_analytics_enabled => Off => Off datadog.pcntl_analytics_enabled => Off => Off datadog.trace.pcntl_analytics_sample_rate => 1 => 1 datadog.pcntl_analytics_sample_rate => 1 => 1 datadog.trace.pdo_enabled => On => On datadog.trace.pdo_analytics_enabled => Off => Off datadog.pdo_analytics_enabled => Off => Off datadog.trace.pdo_analytics_sample_rate => 1 => 1 datadog.pdo_analytics_sample_rate => 1 => 1 datadog.trace.phpredis_enabled => On => On datadog.trace.phpredis_analytics_enabled => Off => Off datadog.phpredis_analytics_enabled => Off => Off datadog.trace.phpredis_analytics_sample_rate => 1 => 1 datadog.phpredis_analytics_sample_rate => 1 => 1 datadog.trace.predis_enabled => On => On datadog.trace.predis_analytics_enabled => Off => Off datadog.predis_analytics_enabled => Off => Off datadog.trace.predis_analytics_sample_rate => 1 => 1 datadog.predis_analytics_sample_rate => 1 => 1 datadog.trace.psr18_enabled => On => On datadog.trace.psr18_analytics_enabled => Off => Off datadog.psr18_analytics_enabled => Off => Off datadog.trace.psr18_analytics_sample_rate => 1 => 1 datadog.psr18_analytics_sample_rate => 1 => 1 datadog.trace.roadrunner_enabled => On => On datadog.trace.roadrunner_analytics_enabled => Off => Off datadog.roadrunner_analytics_enabled => Off => Off datadog.trace.roadrunner_analytics_sample_rate => 1 => 1 datadog.roadrunner_analytics_sample_rate => 1 => 1 datadog.trace.sqlsrv_enabled => On => On datadog.trace.sqlsrv_analytics_enabled => Off => Off datadog.sqlsrv_analytics_enabled => Off => Off datadog.trace.sqlsrv_analytics_sample_rate => 1 => 1 datadog.sqlsrv_analytics_sample_rate => 1 => 1 datadog.trace.slim_enabled => On => On datadog.trace.slim_analytics_enabled => Off => Off datadog.slim_analytics_enabled => Off => Off datadog.trace.slim_analytics_sample_rate => 1 => 1 datadog.slim_analytics_sample_rate => 1 => 1 datadog.trace.symfony_enabled => On => On datadog.trace.symfony_analytics_enabled => Off => Off datadog.symfony_analytics_enabled => Off => Off datadog.trace.symfony_analytics_sample_rate => 1 => 1 datadog.symfony_analytics_sample_rate => 1 => 1 datadog.trace.web_enabled => On => On datadog.trace.web_analytics_enabled => Off => Off datadog.web_analytics_enabled => Off => Off datadog.trace.web_analytics_sample_rate => 1 => 1 datadog.web_analytics_sample_rate => 1 => 1 datadog.trace.wordpress_enabled => On => On datadog.trace.wordpress_analytics_enabled => Off => Off datadog.wordpress_analytics_enabled => Off => Off datadog.trace.wordpress_analytics_sample_rate => 1 => 1 datadog.wordpress_analytics_sample_rate => 1 => 1 datadog.trace.yii_enabled => On => On datadog.trace.yii_analytics_enabled => Off => Off datadog.yii_analytics_enabled => Off => Off datadog.trace.yii_analytics_sample_rate => 1 => 1 datadog.yii_analytics_sample_rate => 1 => 1 datadog.trace.zendframework_enabled => On => On datadog.trace.zendframework_analytics_enabled => Off => Off datadog.zendframework_analytics_enabled => Off => Off datadog.trace.zendframework_analytics_sample_rate => 1 => 1 datadog.zendframework_analytics_sample_rate => 1 => 1

datadog-profiling

Version => 0.96.0 Profiling Enabled => false Profiling Experimental Features Enabled => false (profiling disabled) Experimental CPU Time Profiling Enabled => false (profiling disabled) Allocation Profiling Enabled => false (profiling disabled) Experimental Timeline Enabled => false (profiling disabled) Exception Profiling Enabled => false (profiling disabled) Endpoint Collection Enabled => false (profiling disabled) Platform's CPU Time API Works => true Profiling Log Level => off Profiling Agent Endpoint => http://127.0.0.1:1400/ Application's Environment (DD_ENV) => postnl.1c.eu-central.aws Application's Service (DD_SERVICE) => staging.shop.postnl.nl Application's Version (DD_VERSION) => 20240320105200

Directive => Local Value => Master Value datadog.profiling.enabled => Off => Off datadog.profiling.experimental_features_enabled => Off => Off datadog.profiling.endpoint_collection_enabled => On => On datadog.profiling.experimental_cpu_time_enabled => On => On datadog.profiling.experimental_cpu_enabled => On => On datadog.profiling.allocation_enabled => On => On datadog.profiling.experimental_allocation_enabled => On => On datadog.profiling.experimental_timeline_enabled => Off => Off datadog.profiling.exception_enabled => On => On datadog.profiling.experimental_exception_enabled => On => On datadog.profiling.exception_sampling_distance => 100 => 100 datadog.profiling.experimental_exception_sampling_distance => 100 => 100 datadog.profiling.log_level => off => off datadog.profiling.output_pprof => no value => no value

Upgrading from

No response

bwoebi commented 5 months ago

Hey @elzekool,

That should not happen; is the other thread (which we are waiting for) still running? If yes, can you please dump that threads stacktrace too?

Also, does the issue still occur with 0.98.0 or newer? We've modified the retry logic there, maybe that helps.

Lastly, you could try using the sidecar as a trace sender via datadog.trace.sidecar_trace_sender = 1 - then it's not going to wait, but the sending happens in a separate process.