DataDog / dd-trace-php

Datadog PHP Clients
https://docs.datadoghq.com/tracing/setup/php
Other
497 stars 155 forks source link

[BUG] Segmentation fault on PHP 8.1.14-cli-alpine #1890

Closed iNviNho closed 1 year ago

iNviNho commented 1 year ago

Bug description

Hi,

When calling

GlobalTracer::get()->flush();

in our Laravel commands, the application fails with

Segmentation fault (core dumped)

This is how we install the datadog-php-trace.php

RUN curl -Lo datadog-php-trace.php "https://github.com/DataDog/dd-trace-php/releases/download/0.83.1/datadog-setup.php" \
    && php datadog-php-trace.php --php-bin=all

PHP version

PHP 8.1.14 (cli) (built: Jan  9 2023 19:04:32) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.1.14, Copyright (c) Zend Technologies
    with Zend OPcache v8.1.14, Copyright (c), by Zend Technologies
    with Xdebug v3.2.0, Copyright (c) 2002-2022, by Derick Rethans
    with ddtrace v0.83.1, Copyright Datadog, by Datadog
    with ddappsec v0.4.5, Copyright Datadog, by Datadog

Tracer version

0.83.1

Installed extensions

[PHP Modules]
bcmath
calendar
Core
ctype
curl
date
ddappsec
ddtrace
dom
fileinfo
filter
ftp
hash
iconv
intl
json
libxml
mbstring
mysqlnd
openssl
pcntl
pcre
PDO
pdo_mysql
pdo_sqlite
Phar
posix
rdkafka
readline
redis
Reflection
session
SimpleXML
soap
sockets
sodium
SPL
sqlite3
standard
tokenizer
xdebug
xml
xmlreader
xmlwriter
Zend OPcache
zip
zlib

[Zend Modules]
Xdebug
Zend OPcache
ddappsec
ddtrace

OS info

NAME="Alpine Linux"
VERSION_ID=3.17.1
PRETTY_NAME="Alpine Linux v3.17"

Diagnostics and configuration

Output of phpinfo() (ddtrace >= 0.47.0)

Datadog PHP tracer extension
For help, check out the documentation at https://docs.datadoghq.com/tracing/languages/php/
(c) Datadog 2020

Datadog tracing support => enabled
Version => 0.83.1
DATADOG TRACER CONFIGURATION => {
    "date": "2023-01-30T11:17:17Z",
    "os_name": "Linux 41293e7125fa 5.15.0-58-generic #64-Ubuntu SMP Thu Jan 5 11:43:13 UTC 2023 x86_64",
    "os_version": "5.15.0-58-generic",
    "version": "0.83.1",
    "lang": "php",
    "lang_version": "8.1.8",
    "env": null,
    "enabled": true,
    "service": "broker.offer-service",
    "enabled_cli": true,
    "agent_url": "http:\/\/datadog:8126",
    "debug": false,
    "analytics_enabled": false,
    "sample_rate": 1,
    "sampling_rules": [],
    "tags": [],
    "service_mapping": {
        "pdo": "broker.offer-service.db",
        "phpredis": "broker.offer-service.redis"
    },
    "distributed_tracing_enabled": true,
    "priority_sampling_enabled": true,
    "dd_version": null,
    "architecture": "x86_64",
    "sapi": "cli",
    "datadog.trace.request_init_hook": "\/opt\/datadog\/dd-library\/0.83.1\/dd-trace-sources\/bridge\/dd_wrap_autoloader.php",
    "open_basedir_configured": false,
    "uri_fragment_regex": null,
    "uri_mapping_incoming": null,
    "uri_mapping_outgoing": null,
    "auto_flush_enabled": true,
    "generate_root_span": false,
    "http_client_split_by_domain": false,
    "measure_compile_time": true,
    "report_hostname_on_root_span": false,
    "traced_internal_functions": null,
    "auto_prepend_file_configured": false,
    "integrations_disabled": "default",
    "enabled_from_env": true,
    "opcache.file_cache": null,
    "agent_error": "Resolving timed out after 2000 milliseconds"
}

                               Diagnostics                               
agent_error => Resolving timed out after 2000 milliseconds
Diagnostic checks => failed

Directive => Local Value => Master Value
ddtrace.disable => Off => Off
ddtrace.cgroup_file => /proc/self/cgroup => /proc/self/cgroup
datadog.trace.request_init_hook => /opt/datadog/dd-library/0.83.1/dd-trace-sources/bridge/dd_wrap_autoloader.php => /opt/datadog/dd-library/0.83.1/dd-trace-sources/bridge/dd_wrap_autoloader.php
ddtrace.request_init_hook => /opt/datadog/dd-library/0.83.1/dd-trace-sources/bridge/dd_wrap_autoloader.php => /opt/datadog/dd-library/0.83.1/dd-trace-sources/bridge/dd_wrap_autoloader.php
datadog.trace.agent_url => no value => no value
datadog.agent_host => datadog => datadog
datadog.dogstatsd_url => no value => no value
datadog.distributed_tracing => On => On
datadog.dogstatsd_port => 8125 => 8125
datadog.env => no value => no value
datadog.autofinish_spans => On => On
datadog.trace.url_as_resource_names_enabled => On => On
datadog.integrations_disabled => default => default
datadog.priority_sampling => On => On
datadog.service => broker.offer-service => broker.offer-service
datadog.service_name => broker.offer-service => broker.offer-service
datadog.service_mapping => pdo:broker.offer-service.db,phpredis:broker.offer-service.redis => pdo:broker.offer-service.db,phpredis:broker.offer-service.redis
datadog.tags => no value => no value
datadog.trace.global_tags => no value => no value
datadog.trace.agent_port => 8126 => 8126
datadog.trace.analytics_enabled => Off => Off
datadog.trace.auto_flush_enabled => On => On
datadog.trace.cli_enabled => On => On
datadog.trace.measure_compile_time => On => On
datadog.trace.debug => Off => Off
datadog.trace.enabled => On => On
datadog.trace.health_metrics_enabled => Off => Off
datadog.trace.health_metrics_heartbeat_sample_rate => 0.001 => 0.001
datadog.trace.db_client_split_by_instance => Off => Off
datadog.trace.http_client_split_by_domain => Off => Off
datadog.trace.redis_client_split_by_host => On => On
datadog.trace.memory_limit => no value => no value
datadog.trace.report_hostname => Off => Off
datadog.trace.resource_uri_fragment_regex => no value => no value
datadog.trace.resource_uri_mapping_incoming => no value => no value
datadog.trace.resource_uri_mapping_outgoing => no value => no value
datadog.trace.resource_uri_query_param_allowed => no value => no value
datadog.trace.http_url_query_param_allowed => * => *
datadog.trace.rate_limit => 0 => 0
datadog.trace.sample_rate => 1 => 1
datadog.sampling_rate => 1 => 1
datadog.trace.sampling_rules => [] => []
datadog.span_sampling_rules => [] => []
datadog.span_sampling_rules_file => no value => no value
datadog.trace.header_tags => no value => no value
datadog.trace.x_datadog_tags_max_length => 512 => 512
datadog.trace.propagate_service => Off => Off
datadog.propagation_style_extract => Datadog,B3,B3 single header => Datadog,B3,B3 single header
datadog.propagation_style_inject => Datadog => Datadog
datadog.trace.traced_internal_functions => no value => no value
datadog.trace.agent_timeout => 500 => 500
datadog.trace.agent_connect_timeout => 100 => 100
datadog.trace.debug_prng_seed => -1 => -1
datadog.log_backtrace => Off => Off
datadog.trace.generate_root_span => Off => Off
datadog.trace.spans_limit => 5000 => 5000
datadog.trace.agent_max_consecutive_failures => 3 => 3
datadog.trace.agent_attempt_retry_time_msec => 5000 => 5000
datadog.trace.bgs_connect_timeout => 2000 => 2000
datadog.trace.bgs_timeout => 5000 => 5000
datadog.trace.agent_flush_interval => 5000 => 5000
datadog.trace.agent_flush_after_n_requests => 10 => 10
datadog.trace.shutdown_timeout => 5000 => 5000
datadog.trace.startup_logs => On => On
datadog.trace.agent_debug_verbose_curl => Off => Off
datadog.trace.debug_curl_output => Off => Off
datadog.trace.beta_high_memory_pressure_percent => 80 => 80
datadog.trace.warn_legacy_dd_trace => On => On
datadog.trace.retain_thread_capabilities => Off => Off
datadog.version => no value => no value
datadog.trace.obfuscation_query_string_regexp => (?i)(?:p(?:ass)?w(?:or)?d|pass(?:_?phrase)?|secret|(?:api_?|private_?|public_?|access_?|secret_?)key(?:_?id)?|token|consumer_?(?:id|key|secret)|sign(?:ed|ature)?|auth(?:entication|orization)?)(?:(?:\s|%20)*(?:=|%3D)[^&]+|(?:"|%22)(?:\s|%20)*(?::|%3A)(?:\s|%20)*(?:"|%22)(?:%2[^2]|%[^2]|[^"%])+(?:"|%22))|bearer(?:\s|%20)+[a-z0-9\._\-]|token(?::|%3A)[a-z0-9]{13}|gh[opsu]_[0-9a-zA-Z]{36}|ey[I-L](?:[\w=-]|%3D)+\.ey[I-L](?:[\w=-]|%3D)+(?:\.(?:[\w.+\/=-]|%3D|%2F|%2B)+)?|[\-]{5}BEGIN(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY[\-]{5}[^\-]+[\-]{5}END(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY|ssh-rsa(?:\s|%20)*(?:[a-z0-9\/\.+]|%2F|%5C|%2B){100,} => (?i)(?:p(?:ass)?w(?:or)?d|pass(?:_?phrase)?|secret|(?:api_?|private_?|public_?|access_?|secret_?)key(?:_?id)?|token|consumer_?(?:id|key|secret)|sign(?:ed|ature)?|auth(?:entication|orization)?)(?:(?:\s|%20)*(?:=|%3D)[^&]+|(?:"|%22)(?:\s|%20)*(?::|%3A)(?:\s|%20)*(?:"|%22)(?:%2[^2]|%[^2]|[^"%])+(?:"|%22))|bearer(?:\s|%20)+[a-z0-9\._\-]|token(?::|%3A)[a-z0-9]{13}|gh[opsu]_[0-9a-zA-Z]{36}|ey[I-L](?:[\w=-]|%3D)+\.ey[I-L](?:[\w=-]|%3D)+(?:\.(?:[\w.+\/=-]|%3D|%2F|%2B)+)?|[\-]{5}BEGIN(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY[\-]{5}[^\-]+[\-]{5}END(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY|ssh-rsa(?:\s|%20)*(?:[a-z0-9\/\.+]|%2F|%5C|%2B){100,}
datadog.trace.forked_process => On => On
datadog.trace.agent_max_payload_size => 52428800 => 52428800
datadog.trace.agent_stack_initial_size => 131072 => 131072
datadog.trace.agent_stack_backlog => 12 => 12
datadog.trace.cakephp_enabled => On => On
datadog.trace.cakephp_analytics_enabled => Off => Off
datadog.cakephp_analytics_enabled => Off => Off
datadog.trace.cakephp_analytics_sample_rate => 1 => 1
datadog.cakephp_analytics_sample_rate => 1 => 1
datadog.trace.codeigniter_enabled => On => On
datadog.trace.codeigniter_analytics_enabled => Off => Off
datadog.codeigniter_analytics_enabled => Off => Off
datadog.trace.codeigniter_analytics_sample_rate => 1 => 1
datadog.codeigniter_analytics_sample_rate => 1 => 1
datadog.trace.curl_enabled => On => On
datadog.trace.curl_analytics_enabled => Off => Off
datadog.curl_analytics_enabled => Off => Off
datadog.trace.curl_analytics_sample_rate => 1 => 1
datadog.curl_analytics_sample_rate => 1 => 1
datadog.trace.elasticsearch_enabled => On => On
datadog.trace.elasticsearch_analytics_enabled => Off => Off
datadog.elasticsearch_analytics_enabled => Off => Off
datadog.trace.elasticsearch_analytics_sample_rate => 1 => 1
datadog.elasticsearch_analytics_sample_rate => 1 => 1
datadog.trace.eloquent_enabled => On => On
datadog.trace.eloquent_analytics_enabled => Off => Off
datadog.eloquent_analytics_enabled => Off => Off
datadog.trace.eloquent_analytics_sample_rate => 1 => 1
datadog.eloquent_analytics_sample_rate => 1 => 1
datadog.trace.guzzle_enabled => On => On
datadog.trace.guzzle_analytics_enabled => Off => Off
datadog.guzzle_analytics_enabled => Off => Off
datadog.trace.guzzle_analytics_sample_rate => 1 => 1
datadog.guzzle_analytics_sample_rate => 1 => 1
datadog.trace.laravel_enabled => On => On
datadog.trace.laravel_analytics_enabled => Off => Off
datadog.laravel_analytics_enabled => Off => Off
datadog.trace.laravel_analytics_sample_rate => 1 => 1
datadog.laravel_analytics_sample_rate => 1 => 1
datadog.trace.lumen_enabled => On => On
datadog.trace.lumen_analytics_enabled => Off => Off
datadog.lumen_analytics_enabled => Off => Off
datadog.trace.lumen_analytics_sample_rate => 1 => 1
datadog.lumen_analytics_sample_rate => 1 => 1
datadog.trace.memcached_enabled => On => On
datadog.trace.memcached_analytics_enabled => Off => Off
datadog.memcached_analytics_enabled => Off => Off
datadog.trace.memcached_analytics_sample_rate => 1 => 1
datadog.memcached_analytics_sample_rate => 1 => 1
datadog.trace.mongo_enabled => On => On
datadog.trace.mongo_analytics_enabled => Off => Off
datadog.mongo_analytics_enabled => Off => Off
datadog.trace.mongo_analytics_sample_rate => 1 => 1
datadog.mongo_analytics_sample_rate => 1 => 1
datadog.trace.mongodb_enabled => On => On
datadog.trace.mongodb_analytics_enabled => Off => Off
datadog.mongodb_analytics_enabled => Off => Off
datadog.trace.mongodb_analytics_sample_rate => 1 => 1
datadog.mongodb_analytics_sample_rate => 1 => 1
datadog.trace.mysqli_enabled => On => On
datadog.trace.mysqli_analytics_enabled => Off => Off
datadog.mysqli_analytics_enabled => Off => Off
datadog.trace.mysqli_analytics_sample_rate => 1 => 1
datadog.mysqli_analytics_sample_rate => 1 => 1
datadog.trace.nette_enabled => On => On
datadog.trace.nette_analytics_enabled => Off => Off
datadog.nette_analytics_enabled => Off => Off
datadog.trace.nette_analytics_sample_rate => 1 => 1
datadog.nette_analytics_sample_rate => 1 => 1
datadog.trace.pcntl_enabled => On => On
datadog.trace.pcntl_analytics_enabled => Off => Off
datadog.pcntl_analytics_enabled => Off => Off
datadog.trace.pcntl_analytics_sample_rate => 1 => 1
datadog.pcntl_analytics_sample_rate => 1 => 1
datadog.trace.pdo_enabled => On => On
datadog.trace.pdo_analytics_enabled => Off => Off
datadog.pdo_analytics_enabled => Off => Off
datadog.trace.pdo_analytics_sample_rate => 1 => 1
datadog.pdo_analytics_sample_rate => 1 => 1
datadog.trace.phpredis_enabled => On => On
datadog.trace.phpredis_analytics_enabled => Off => Off
datadog.phpredis_analytics_enabled => Off => Off
datadog.trace.phpredis_analytics_sample_rate => 1 => 1
datadog.phpredis_analytics_sample_rate => 1 => 1
datadog.trace.predis_enabled => On => On
datadog.trace.predis_analytics_enabled => Off => Off
datadog.predis_analytics_enabled => Off => Off
datadog.trace.predis_analytics_sample_rate => 1 => 1
datadog.predis_analytics_sample_rate => 1 => 1
datadog.trace.roadrunner_enabled => On => On
datadog.trace.roadrunner_analytics_enabled => Off => Off
datadog.roadrunner_analytics_enabled => Off => Off
datadog.trace.roadrunner_analytics_sample_rate => 1 => 1
datadog.roadrunner_analytics_sample_rate => 1 => 1
datadog.trace.slim_enabled => On => On
datadog.trace.slim_analytics_enabled => Off => Off
datadog.slim_analytics_enabled => Off => Off
datadog.trace.slim_analytics_sample_rate => 1 => 1
datadog.slim_analytics_sample_rate => 1 => 1
datadog.trace.symfony_enabled => On => On
datadog.trace.symfony_analytics_enabled => Off => Off
datadog.symfony_analytics_enabled => Off => Off
datadog.trace.symfony_analytics_sample_rate => 1 => 1
datadog.symfony_analytics_sample_rate => 1 => 1
datadog.trace.web_enabled => On => On
datadog.trace.web_analytics_enabled => Off => Off
datadog.web_analytics_enabled => Off => Off
datadog.trace.web_analytics_sample_rate => 1 => 1
datadog.web_analytics_sample_rate => 1 => 1
datadog.trace.wordpress_enabled => On => On
datadog.trace.wordpress_analytics_enabled => Off => Off
datadog.wordpress_analytics_enabled => Off => Off
datadog.trace.wordpress_analytics_sample_rate => 1 => 1
datadog.wordpress_analytics_sample_rate => 1 => 1
datadog.trace.yii_enabled => On => On
datadog.trace.yii_analytics_enabled => Off => Off
datadog.yii_analytics_enabled => Off => Off
datadog.trace.yii_analytics_sample_rate => 1 => 1
datadog.yii_analytics_sample_rate => 1 => 1
datadog.trace.zendframework_enabled => On => On
datadog.trace.zendframework_analytics_enabled => Off => Off
datadog.zendframework_analytics_enabled => Off => Off
datadog.trace.zendframework_analytics_sample_rate => 1 => 1
datadog.zendframework_analytics_sample_rate => 1 => 1

Output from Valgrind

/var/www/app # USER_ZEND_ALLOC=0 valgrind -- php artisan offers:offer-create
==87== Memcheck, a memory error detector
==87== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==87== Using Valgrind-3.20.0 and LibVEX; rerun with -h for copyright info
==87== Command: php artisan offers:offer-create
==87== 
{"message":"GuzzleFeatureRequester is not using an HTTP cache because Kevinrob\\GuzzleCache\\CacheMiddleware was not installed","context":{},"level_name":"INFO","channel":"local","datetime":"2023-01-30T10:56:17.317965+00:00","extra":{},"dd":{"trace_id":"0","span_id":"0"}}
==87== Invalid read of size 8
==87==    at 0x5A9665D: ddtrace_close_userland_spans_until (span.c:352)
==87==    by 0x5A84B96: zif_flush (ddtrace.c:1788)
==87==    by 0x5A17D24: xdebug_execute_internal (base.c:952)
==87==    by 0x5A6C0CD: zai_interceptor_execute_internal_impl (interceptor.c:694)
==87==    by 0x341AC2: ??? (in /usr/local/bin/php)
==87==    by 0x34290B: ??? (in /usr/local/bin/php)
==87==    by 0x5A17305: xdebug_execute_ex (base.c:830)
==87==    by 0x341A32: ??? (in /usr/local/bin/php)
==87==    by 0x34290B: ??? (in /usr/local/bin/php)
==87==    by 0x5A17305: xdebug_execute_ex (base.c:830)
==87==    by 0x341A32: ??? (in /usr/local/bin/php)
==87==    by 0x34290B: ??? (in /usr/local/bin/php)
==87==  Address 0xb8 is not stack'd, malloc'd or (recently) free'd
==87== 
==87== 
==87== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==87==  Access not within mapped region at address 0xB8
==87==    at 0x5A9665D: ddtrace_close_userland_spans_until (span.c:352)
==87==    by 0x5A84B96: zif_flush (ddtrace.c:1788)
==87==    by 0x5A17D24: xdebug_execute_internal (base.c:952)
==87==    by 0x5A6C0CD: zai_interceptor_execute_internal_impl (interceptor.c:694)
==87==    by 0x341AC2: ??? (in /usr/local/bin/php)
==87==    by 0x34290B: ??? (in /usr/local/bin/php)
==87==    by 0x5A17305: xdebug_execute_ex (base.c:830)
==87==    by 0x341A32: ??? (in /usr/local/bin/php)
==87==    by 0x34290B: ??? (in /usr/local/bin/php)
==87==    by 0x5A17305: xdebug_execute_ex (base.c:830)
==87==    by 0x341A32: ??? (in /usr/local/bin/php)
==87==    by 0x34290B: ??? (in /usr/local/bin/php)
==87==  If you believe this happened as a result of a stack
==87==  overflow in your program's main thread (unlikely but
==87==  possible), you can try to increase the size of the
==87==  main thread stack using the --main-stacksize= flag.
==87==  The main thread stack size used in this run was 8388608.
==87== 
==87== HEAP SUMMARY:
==87==     in use at exit: 6,004,106 bytes in 71,706 blocks
==87==   total heap usage: 160,275 allocs, 88,569 frees, 14,040,800 bytes allocated
==87== 
==87== LEAK SUMMARY:
==87==    definitely lost: 30,240 bytes in 946 blocks
==87==    indirectly lost: 138 bytes in 7 blocks
==87==      possibly lost: 2,950,792 bytes in 22,833 blocks
==87==    still reachable: 3,022,936 bytes in 47,920 blocks
==87==         suppressed: 0 bytes in 0 blocks
==87== Rerun with --leak-check=full to see details of leaked memory
==87== 
==87== For lists of detected and suppressed errors, rerun with: -s
==87== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Segmentation fault (core dumped)

Upgrading info

We encountered this issue after upgrading from 0.73.0 to 0.83.1.

bwoebi commented 1 year ago

Hey @iNviNho,

thanks for the bug report, that function was called with NULL in flushing after a refactor in 0.81.0, causing that issue. We'll add a regression test and fix it.

iNviNho commented 1 year ago

Thank you @bwoebi

So how should we proceed from our side?

We are tracing in long running CLI command so we start global tracer and close/flush manually after each message we process.

Like this we start the tracing

       protected function startDatadogTracing(
        string $topicName,
        string $traceId = null,
        string $parentId = null
    ): void {
        if (!empty($traceId) && !empty($parentId)) {
            set_distributed_tracing_context($traceId, $parentId);
        }

        $tracer = new Tracer();
        GlobalTracer::set($tracer);

        $resourceName = $topicName . '::process';

        $options = ['start_time' => Time::now()];
        $rootSpan = $tracer->startRootSpan(
            'Jobs',
            $options
        )
            ->getSpan();

        $rootSpan->setTag(Tag::SPAN_TYPE, Type::CLI);
        $rootSpan->setTag(Tag::SERVICE_NAME, 'broker.offer-service');
        $rootSpan->setTag(Tag::RESOURCE_NAME, $resourceName, true);
        $rootSpan->setTag('command_origin', $this->signature);

        // set up tracing ...
    }

Like this we send traces

                GlobalTracer::get()->getRootScope()?->close();
                GlobalTracer::get()->flush();

Thank you for your fast reaction.

bwoebi commented 1 year ago

Currently DDTrace\flush() with DD_AUTOFINISH_SPANS=1 is completely broken.

An equivalent would be:

ini_set("datadog.autofinish_spans", 0);
DDTrace\close_spans_until(null);
GlobalTracer::get()->flush();
ini_set("datadog.autofinish_spans", 1);

If you want to generally use autofinish functionality.

iNviNho commented 1 year ago

Does DD_AUTOFINISH_SPANS=1 mean SDK automatically closes global/root span?

Can we set DD_AUTOFINISH_SPANS=0 in Dockerfile for all the commands? Indeed we do close manually all global spans before we flush them.

We do

bwoebi commented 1 year ago

Yes, it means that it will automagically close spans, which happen to not be closed manually - otherwise you may lose unclosed spans.

If you do not need that feature, just turn it off altogether :-)

iNviNho commented 1 year ago

Just for my understanding...

Why would spans not be closed automatically?

If I define tracing a method of a class like this

        trace_method(Producer::class, 'produce', function (SpanData $span) {
            $span->service = 'broker.offer-service.kafka';
        });

Doesn't SDK start span right before this method is called and close span right after the execution?

bwoebi commented 1 year ago

It's just about manually started spans, automatic root spans and tracing spans are always automatically properly started and closed.

iNviNho commented 1 year ago

Thank you for your help.

Setting DD_AUTOFINISH_SPANS=false did the trick :)

bwoebi commented 1 year ago

@iNviNho And now 0.84.0 is out, and you can revert to DD_AUTOFINISH_SPANS=true again :-)