[Bug]: Laravel 10x Horizon Not Showing up in APM

[jhjm32087]

Bug report

We just installed the latest version of dd-trace 0.91.2 to get Laravel 10 Horizon Queue Support in APM. Unfortunately. I still don't see it in APM.

The version of Laravel you are using: Laravel 10.20.0 Laravel 10.20.0

Any exception debug output generated when using DD_TRACE_DEBUG=true None see screenshot below:

Whether you still receive other traces from this service Yes, APM from Laravel API and Scheduler are running normally.

The output of phpinfo See below.

PHP version

8.1.22

Tracer or profiler version

0.91.2

Installed extensions

[PHP Modules] Core ctype curl datadog-profiling date ddappsec ddtrace dom fileinfo filter ftp gd hash iconv imagick intl json libxml mbstring mysqlnd openssl pcntl pcre sqlite3 standard tokenizer xml xmlreader xmlwriter Zend OPcache zlib

[Zend Modules] Zend OPcache datadog-profiling ddappsec ddtrace

Output of phpinfo()

ddtrace

Datadog PHP tracer extension For help, check out the documentation at https://docs.datadoghq.com/tracing/languages/php/ (c) Datadog 2020

Datadog tracing support => disabled Version => 0.91.2 DATADOG TRACER CONFIGURATION => { "date": "2023-09-19T22:47:10Z", "os_name": "", "os_version": "", "version": "0.91.2", "lang": "php", "lang_version": "8.1.22", "env": "production", "enabled": true, "service": "", "enabled_cli": true, "agent_url": "http:\/\/10.0.13.78:8126", "debug": false, "analytics_enabled": false, "sample_rate": -1, "sampling_rules": [], "tags": [], "service_mapping": [], "distributed_tracing_enabled": true, "priority_sampling_enabled": true, "dd_version": null, "architecture": "x86_64", "sapi": "cli", "datadog.trace.request_init_hook": "\/opt\/datadog\/dd-library\/0.91.2\/dd-trace-sources\/bridge\/dd_wrap_autoloader.php", "open_basedir_configured": false, "uri_fragment_regex": null, "uri_mapping_incoming": null, "uri_mapping_outgoing": null, "auto_flush_enabled": false, "generate_root_span": true, "http_client_split_by_domain": false, "measure_compile_time": true, "report_hostname_on_root_span": false, "traced_internal_functions": null, "auto_prepend_file_configured": false, "integrations_disabled": "default", "enabled_from_env": false, "opcache.file_cache": null }

Directive => Local Value => Master Value ddtrace.disable => Off => Off ddtrace.cgroup_file => /proc/self/cgroup => /proc/self/cgroup datadog.trace.request_init_hook => /opt/datadog/dd-library/0.91.2/dd-trace-sources/bridge/dd_wrap_autoloader.php => /opt/datadog/dd-library/0.91.2/dd-trace-sources/bridge/dd_wrap_autoloader.php ddtrace.request_init_hook => /opt/datadog/dd-library/0.91.2/dd-trace-sources/bridge/dd_wrap_autoloader.php => /opt/datadog/dd-library/0.91.2/dd-trace-sources/bridge/dd_wrap_autoloader.php datadog.trace.agent_url => no value => no value datadog.agent_host => 10.0.13.78 => 10.0.13.78 datadog.dogstatsd_url => no value => no value datadog.distributed_tracing => On => On datadog.dogstatsd_port => 8125 => 8125 datadog.env => production => production datadog.autofinish_spans => Off => Off datadog.trace.url_as_resource_names_enabled => On => On datadog.http_server_route_based_naming => On => On datadog.integrations_disabled => default => default datadog.priority_sampling => On => On datadog.service => laravel-api => laravel-api datadog.service_name => laravel-api => laravel-api datadog.service_mapping => no value => no value datadog.tags => no value => no value datadog.trace.global_tags => no value => no value datadog.trace.agent_port => 0 => 0 datadog.trace.analytics_enabled => Off => Off datadog.trace.append_trace_ids_to_logs => Off => Off datadog.trace.auto_flush_enabled => Off => Off datadog.trace.cli_enabled => Off => On datadog.trace.measure_compile_time => On => On datadog.trace.debug => Off => Off datadog.trace.enabled => Off => On datadog.instrumentation_telemetry_enabled => On => On datadog.trace.health_metrics_enabled => Off => Off datadog.trace.health_metrics_heartbeat_sample_rate => 0.001 => 0.001 datadog.trace.db_client_split_by_instance => Off => Off datadog.trace.http_client_split_by_domain => Off => Off datadog.trace.redis_client_split_by_host => Off => Off datadog.trace.memory_limit => no value => no value datadog.trace.report_hostname => Off => Off datadog.trace.flush_collect_cycles => Off => Off datadog.trace.remove_root_span_laravel_queue => On => On datadog.trace.remove_autoinstrumentation_orphans => Off => Off datadog.trace.resource_uri_fragment_regex => no value => no value datadog.trace.resource_uri_mapping_incoming => no value => no value datadog.trace.resource_uri_mapping_outgoing => no value => no value datadog.trace.resource_uri_query_param_allowed => no value => no value datadog.trace.http_url_query_param_allowed => => datadog.trace.http_post_data_param_allowed => no value => no value datadog.trace.rate_limit => 0 => 0 datadog.trace.sample_rate => -1 => -1 datadog.sampling_rate => -1 => -1 datadog.trace.sampling_rules => [] => [] datadog.span_sampling_rules => [] => [] datadog.span_sampling_rules_file => no value => no value datadog.trace.header_tags => no value => no value datadog.trace.x_datadog_tags_max_length => 512 => 512 datadog.trace.peer_service_mapping => no value => no value datadog.trace.peer_service_defaults_enabled => Off => Off datadog.trace.remove_integration_service_names_enabled => Off => Off datadog.trace.propagate_service => Off => Off datadog.trace.propagation_style_extract => tracecontext,Datadog,B3,B3 single header => tracecontext,Datadog,B3,B3 single header datadog.propagation_style_extract => tracecontext,Datadog,B3,B3 single header => tracecontext,Datadog,B3,B3 single header datadog.trace.propagation_style_inject => tracecontext,Datadog => tracecontext,Datadog datadog.propagation_style_inject => tracecontext,Datadog => tracecontext,Datadog datadog.trace.propagation_style => tracecontext,Datadog => tracecontext,Datadog datadog.trace.traced_internal_functions => no value => no value datadog.trace.agent_timeout => 500 => 500 datadog.trace.agent_connect_timeout => 100 => 100 datadog.trace.debug_prng_seed => -1 => -1 datadog.log_backtrace => Off => Off datadog.trace.generate_root_span => On => On datadog.trace.spans_limit => 1000 => 1000 datadog.trace.128_bit_traceid_generation_enabled => Off => Off datadog.trace.128_bit_traceid_logging_enabled => Off => Off datadog.trace.agent_max_consecutive_failures => 3 => 3 datadog.trace.agent_attempt_retry_time_msec => 5000 => 5000 datadog.trace.bgs_connect_timeout => 2000 => 2000 datadog.trace.bgs_timeout => 5000 => 5000 datadog.trace.agent_flush_interval => 5000 => 5000 datadog.trace.agent_flush_after_n_requests => 10 => 10 datadog.trace.shutdown_timeout => 5000 => 5000 datadog.trace.startup_logs => On => On datadog.trace.once_logs => On => On datadog.trace.agent_debug_verbose_curl => Off => Off datadog.trace.debug_curl_output => Off => Off datadog.trace.beta_high_memory_pressure_percent => 80 => 80 datadog.trace.sidecar_trace_sender => Off => Off datadog.trace.warn_legacy_dd_trace => On => On datadog.trace.retain_thread_capabilities => Off => Off datadog.version => no value => no value datadog.trace.obfuscation_query_stringregexp => (?i)(?:p(?:ass)?w(?:or)?d|pass(?:?phrase)?|secret|(?:api?|private?|public?|access?|secret?)key(?:?id)?|token|consumer?(?:id|key|secret)|sign(?:ed|ature)?|auth(?:entication|orization)?)(?:(?:\s|%20)(?:=|%3D)[^&]+|( ?:"|%22)(?:\s|%20)(?::|%3A)(?:\s|%20)*(?:"|%22)(?:%2[^2]|%[^2]|[^"%])+(?:"|%22))|bearer(?:\s|%20)+[a-z0-9.-]|token(?::|%3A)[a-z0-9]{13}|gh[opsu][0-9a-zA-Z]{36}|eyI-L+.eyI-L+(?:.(?:[\w.+\/=-]|%3D|%2F|%2B)+)?|[-]{5}BEGIN(?:[a-z\s]|%2 0)+PRIVATE(?:\s|%20)KEY[-]{5}[^-]+[-]{5}END(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY|ssh-rsa(?:\s|%20)*(?:[a-z0-9\/.+]|%2F|%5C|%2B){100,} => (?i)(?:p(?:ass)?w(?:or)?d|pass(?:?phrase)?|secret|(?:api?|private?|public?|access?|secret?)key(?:?id)?|token|consumer?(?:id |key|secret)|sign(?:ed|ature)?|auth(?:entication|orization)?)(?:(?:\s|%20)(?:=|%3D)[^&]+|(?:"|%22)(?:\s|%20)(?::|%3A)(?:\s|%20)*(?:"|%22)(?:%2[^2]|%[^2]|[^"%])+(?:"|%22))|bearer(?:\s|%20)+[a-z0-9.-]|token(?::|%3A)[a-z0-9]{13}|gh[opsu]_[0-9a-zA-Z]{36}|eyI-L+.eyI-L+(?:.(?:[\w.+\/=-]|%3D|%2F|%2B)+)?|[-]{5}BEGIN(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY[-]{5}[^-]+[-]{5}END(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY|ssh-rsa(?:\s|%20)*(?:[a-z0-9\/.+]|%2F|%5C|%2B){100,} datadog.trace.client_ip_enabled => Off => Off datadog.trace.client_ip_header => no value => no value datadog.trace.forked_process => On => On datadog.trace.hook_limit => 100 => 100 datadog.trace.agent_max_payload_size => 52428800 => 52428800 datadog.trace.agent_stack_initial_size => 131072 => 131072 datadog.trace.agent_stack_backlog => 12 => 12 datadog.trace.propagate_user_id_default => Off => Off datadog.dbm_propagation_mode => full => full datadog.trace.wordpress_additional_actions => no value => no value datadog.trace.wordpress_callbacks => Off => Off datadog.trace.wordpress_enhanced_integration => Off => Off datadog.trace.amqp_enabled => On => On datadog.trace.amqp_analytics_enabled => Off => Off datadog.amqp_analytics_enabled => Off => Off datadog.trace.amqp_analytics_sample_rate => 1 => 1 datadog.amqp_analytics_sample_rate => 1 => 1 datadog.trace.cakephp_enabled => On => On datadog.trace.cakephp_analytics_enabled => Off => Off datadog.cakephp_analytics_enabled => Off => Off datadog.trace.cakephp_analytics_sample_rate => 1 => 1 datadog.cakephp_analytics_sample_rate => 1 => 1 datadog.trace.codeigniter_enabled => On => On datadog.trace.codeigniter_analytics_enabled => Off => Off datadog.codeigniter_analytics_enabled => Off => Off datadog.trace.codeigniter_analytics_sample_rate => 1 => 1 datadog.codeigniter_analytics_sample_rate => 1 => 1 datadog.trace.curl_enabled => On => On datadog.trace.curl_analytics_enabled => Off => Off datadog.curl_analytics_enabled => Off => Off datadog.trace.curl_analytics_sample_rate => 1 => 1 datadog.curl_analytics_sample_rate => 1 => 1 datadog.trace.elasticsearch_enabled => On => On datadog.trace.elasticsearch_analytics_enabled => Off => Off datadog.elasticsearch_analytics_enabled => Off => Off datadog.trace.elasticsearch_analytics_sample_rate => 1 => 1 datadog.elasticsearch_analytics_sample_rate => 1 => 1 datadog.trace.eloquent_enabled => On => On datadog.trace.eloquent_analytics_enabled => Off => Off datadog.eloquent_analytics_enabled => Off => Off datadog.trace.eloquent_analytics_sample_rate => 1 => 1 datadog.eloquent_analytics_sample_rate => 1 => 1 datadog.trace.guzzle_enabled => On => On datadog.trace.guzzle_analytics_enabled => Off => Off datadog.guzzle_analytics_enabled => Off => Off datadog.trace.guzzle_analytics_sample_rate => 1 => 1 datadog.guzzle_analytics_sample_rate => 1 => 1 datadog.trace.laminas_enabled => On => On datadog.trace.laminas_analytics_enabled => Off => Off datadog.laminas_analytics_enabled => Off => Off datadog.trace.laminas_analytics_sample_rate => 1 => 1 datadog.laminas_analytics_sample_rate => 1 => 1 datadog.trace.laravel_enabled => On => On datadog.trace.laravel_analytics_enabled => Off => Off datadog.laravel_analytics_enabled => Off => Off datadog.trace.laravel_analytics_sample_rate => 1 => 1 datadog.laravel_analytics_sample_rate => 1 => 1 datadog.trace.laravelqueue_enabled => On => On datadog.trace.laravelqueue_analytics_enabled => Off => Off datadog.laravelqueue_analytics_enabled => Off => Off datadog.trace.laravelqueue_analytics_sample_rate => 1 => 1 datadog.laravelqueue_analytics_sample_rate => 1 => 1 datadog.trace.logs_enabled => On => On datadog.logs_injection => On => On datadog.trace.logs_analytics_enabled => Off => Off datadog.logs_analytics_enabled => Off => Off datadog.trace.logs_analytics_sample_rate => 1 => 1 datadog.logs_analytics_sample_rate => 1 => 1 datadog.trace.lumen_enabled => On => On datadog.trace.lumen_analytics_enabled => Off => Off datadog.lumen_analytics_enabled => Off => Off datadog.trace.lumen_analytics_sample_rate => 1 => 1 datadog.lumen_analytics_sample_rate => 1 => 1 datadog.trace.memcache_enabled => On => On datadog.trace.memcache_analytics_enabled => Off => Off datadog.memcache_analytics_enabled => Off => Off datadog.trace.memcache_analytics_sample_rate => 1 => 1 datadog.memcache_analytics_sample_rate => 1 => 1 datadog.trace.memcached_enabled => On => On datadog.trace.memcached_analytics_enabled => Off => Off datadog.memcached_analytics_enabled => Off => Off datadog.trace.memcached_analytics_sample_rate => 1 => 1 datadog.memcached_analytics_sample_rate => 1 => 1 datadog.trace.mongo_enabled => On => On datadog.trace.mongo_analytics_enabled => Off => Off datadog.mongo_analytics_enabled => Off => Off datadog.trace.mongo_analytics_sample_rate => 1 => 1 datadog.mongo_analytics_sample_rate => 1 => 1 datadog.trace.mongodb_enabled => On => On datadog.trace.mongodb_analytics_enabled => Off => Off datadog.mongodb_analytics_enabled => Off => Off datadog.trace.mongodb_analytics_sample_rate => 1 => 1 datadog.mongodb_analytics_sample_rate => 1 => 1 datadog.trace.mysqli_enabled => On => On datadog.trace.mysqli_analytics_enabled => Off => Off datadog.mysqli_analytics_enabled => Off => Off datadog.trace.mysqli_analytics_sample_rate => 1 => 1 datadog.mysqli_analytics_sample_rate => 1 => 1 datadog.trace.nette_enabled => On => On datadog.trace.nette_analytics_enabled => Off => Off datadog.nette_analytics_enabled => Off => Off datadog.trace.nette_analytics_sample_rate => 1 => 1 datadog.nette_analytics_sample_rate => 1 => 1 datadog.trace.pcntl_enabled => On => On datadog.trace.pcntl_analytics_enabled => Off => Off datadog.pcntl_analytics_enabled => Off => Off datadog.trace.pcntl_analytics_sample_rate => 1 => 1 datadog.pcntl_analytics_sample_rate => 1 => 1 datadog.trace.pdo_enabled => On => On datadog.trace.pdo_analytics_enabled => Off => Off datadog.pdo_analytics_enabled => Off => Off datadog.trace.pdo_analytics_sample_rate => 1 => 1 datadog.pdo_analytics_sample_rate => 1 => 1 datadog.trace.phpredis_enabled => On => On datadog.trace.phpredis_analytics_enabled => Off => Off datadog.phpredis_analytics_enabled => Off => Off datadog.trace.phpredis_analytics_sample_rate => 1 => 1 datadog.phpredis_analytics_sample_rate => 1 => 1 datadog.trace.predis_enabled => On => On datadog.trace.predis_analytics_enabled => Off => Off datadog.predis_analytics_enabled => Off => Off datadog.trace.predis_analytics_sample_rate => 1 => 1 datadog.predis_analytics_sample_rate => 1 => 1 datadog.trace.psr18_enabled => On => On datadog.trace.psr18_analytics_enabled => Off => Off datadog.psr18_analytics_enabled => Off => Off datadog.trace.psr18_analytics_sample_rate => 1 => 1 datadog.psr18_analytics_sample_rate => 1 => 1 datadog.trace.roadrunner_enabled => On => On datadog.trace.roadrunner_analytics_enabled => Off => Off datadog.roadrunner_analytics_enabled => Off => Off datadog.trace.roadrunner_analytics_sample_rate => 1 => 1 datadog.roadrunner_analytics_sample_rate => 1 => 1 datadog.trace.sqlsrv_enabled => On => On datadog.trace.sqlsrv_analytics_enabled => Off => Off datadog.sqlsrv_analytics_enabled => Off => Off datadog.trace.sqlsrv_analytics_sample_rate => 1 => 1 datadog.sqlsrv_analytics_sample_rate => 1 => 1 datadog.trace.slim_enabled => On => On datadog.trace.slim_analytics_enabled => Off => Off datadog.slim_analytics_enabled => Off => Off datadog.trace.slim_analytics_sample_rate => 1 => 1 datadog.slim_analytics_sample_rate => 1 => 1 datadog.trace.symfony_enabled => On => On datadog.trace.symfony_analytics_enabled => Off => Off datadog.symfony_analytics_enabled => Off => Off datadog.trace.symfony_analytics_sample_rate => 1 => 1 datadog.symfony_analytics_sample_rate => 1 => 1 datadog.trace.web_enabled => On => On datadog.trace.web_analytics_enabled => Off => Off datadog.web_analytics_enabled => Off => Off datadog.trace.web_analytics_sample_rate => 1 => 1 datadog.web_analytics_sample_rate => 1 => 1 datadog.trace.wordpress_enabled => On => On datadog.trace.wordpress_analytics_enabled => Off => Off datadog.wordpress_analytics_enabled => Off => Off datadog.trace.zendframework_analytics_sample_rate => 1 => 1 datadog.zendframework_analytics_sample_rate => 1 => 1

datadog-profiling

Version => 0.91.2 Profiling Enabled => true Experimental CPU Time Profiling Enabled => true Allocation Profiling Enabled => true Experimental Timeline Enabled => false Endpoint Collection Enabled => true Platform's CPU Time API Works => true Profiling Log Level => off Profiling Agent Endpoint => http://10.0.13.78:8126/ Application's Environment (DD_ENV) => production Application's Service (DD_SERVICE) => laravel-api Application's Version (DD_VERSION) =>

Directive => Local Value => Master Value datadog.profiling.enabled => On => On datadog.profiling.endpoint_collection_enabled => On => On datadog.profiling.experimental_cpu_time_enabled => On => On datadog.profiling.experimental_cpu_enabled => On => On datadog.profiling.allocation_enabled => On => On datadog.profiling.experimental_allocation_enabled => On => On datadog.profiling.experimental_timeline_enabled => Off => Off datadog.profiling.log_level => off => off datadog.profiling.output_pprof => no value => no value

Upgrading from

No response

[PROFeNoM]

Hi @jhjm32087, and thanks for opening a separate issue with this screenshot 😃

There was a stupid typo.

Could you please try out this artifact (Related CI job) to confirm it fixes the issue? 🤞

FYI, to install the artifact, if you followed the extension installation doc, simply change the URL of the official installer by the URL of the artifact and you're good to go 👍

[jhjm32087]

@PROFeNoM, I can confirm that the artifact fixed the issue. Another question any chance all the extra noise could be filtered out or trace_method filter those?

[PROFeNoM]

Hi @jhjm32087 :wave: I'm glad you asked!

You can set DD_TRACE_REMOVE_AUTOINSTRUMENTATION_ORPHANS=true to filter them out (or rather, sample them out*) 😃

[jhjm32087]

@PROFeNoM, you rock! I assume that 0.91.3 will fix the typo issue?

[PROFeNoM]

Either 0.91.3 or 0.92.0, depending on how soon we can ship out the remaining PRs to get approved for 0.92.0. At the earliest, it would be released Friday - otherwise, Monday :+1: The sooner, the better

[jhjm32087]

@PROFeNoM Thank you again! No big rush on my side just curious!

[bwoebi]

Hey @jhjm32087 This has been released just now as 0.92.0 :-)

[trevorgehman]

@PROFeNoM @jhjm32087 Not trying to hijack this, but with all the recent changes around support for Laravel Horizon and Queues, I wonder if there is an up-to-date guide somewhere with recommended defaults for getting useful APM set up for Laravel + Horizon.

We would prefer to use Datadog instead of Sentry for APM, but I can't find an easy "getting started" guide that's specific to Laravel.

Thanks in advance!

[jhjm32087]

@trevorgehman, As a baseline, I would try these settings for K8.

• name: DD_AGENT_HOST valueFrom: fieldRef: fieldPath: status.hostIP
• name: DD_TRACE_CLI_ENABLED value: "1"
• name: DD_ENV value: "production"
• name: DD_SERVICE value: "horizon"
• name: DD_TRACE_LARAVELQUEUE_ENABLED value: "1"
• name: DD_TRACE_REMOVE_ROOT_SPAN_LARAVEL_QUEUE value: "true"
• name: DD_TRACE_REMOVE_AUTOINSTRUMENTATION_ORPHANS value: "true"