ci: add tracer only option to framework tests

gnufede commented 3 months ago

CI: add tracer only option to help debug root cause of framework test failures.

Also includes some fixes for some failing framework tests.

Checklist

[x] Change(s) are motivated and described in the PR description
[x] Testing strategy is described if automated tests are not included in the PR
[x] Risks are described (performance impact, potential for breakage, maintainability)
[x] Change is maintainable (easy to change, telemetry, documentation)
[x] Library release note guidelines are followed or label changelog/no-changelog is set
[x] Documentation is included (in-code, generated user docs, public corp docs)
[x] Backport labels are set (if applicable)
[x] If this PR changes the public interface, I've notified @DataDog/apm-tees.

Reviewer Checklist

[x] Title is accurate
[x] All changes are related to the pull request's stated goal
[x] Description motivates each change
[x] Avoids breaking API changes
[x] Testing strategy adequately addresses listed risks
[x] Change is maintainable (easy to change, telemetry, documentation)
[x] Release note makes sense to a user of the library
[x] Author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
[x] Backport labels are set in a manner that is consistent with the release branch maintenance policy

datadog-dd-trace-py-rkomorn[bot] commented 3 months ago

Datadog Report

Branch report: gnufede/fw-tests-tracer-only-option Commit report: 1005b03 Test service: dd-trace-py

:x: 14 Failed (0 Known Flaky), 138494 Passed, 38442 Skipped, 7h 33m 12.9s Total duration (2h 43m 23.79s time saved) :snowflake: 2 New Flaky

:x: Failed Tests (14)

This report shows up to 5 failed tests.

test_on_executor_custom_kwarg - test_executor_decorator.py - Details

<details>
<summary>Expand for error</summary>

```
assert 'tornado-web' == None
 +  where None = <Span(id=11335254185948437149,trace_id=136190548978847266991836019365796295587,parent_id=13119912086889495610,name=tornado.executor.with)>.service
```
</details>

test_on_executor_custom_kwarg - test_executor_decorator.py - Details

Expand for error
``` assert 'tornado-web' == None + where None = .service ```
test_on_executor_exception_handler - test_executor_decorator.py - Details

Expand for error
``` assert 'tornado-web' == None + where None = .service ```
test_on_executor_exception_handler - test_executor_decorator.py - Details

Expand for error
``` assert 'tornado-web' == None + where None = .service ```
test_on_executor_handler - test_executor_decorator.py - Details

Expand for error
``` assert 'tornado-web' == None + where None = .service ```

New Flaky Tests (2)

test_sqli - test_pygoat.py - Last Failure

Expand for error
``` HTTPConnectionPool(host='0.0.0.0', port=9126): Max retries exceeded with url: /test/session/traces?test_session_token=pygoat_test (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) ```
test_ssrf2 - test_pygoat.py - Last Failure

Expand for error
``` HTTPConnectionPool(host='0.0.0.0', port=9126): Max retries exceeded with url: /test/session/start?test_session_token=pygoat_test (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) ```

pr-commenter[bot] commented 3 months ago

Benchmarks

Benchmark execution time: 2024-06-24 18:20:52

Comparing candidate commit b03330312a2694aeed9860a27a2531d01b380482 in PR branch gnufede/fw-tests-tracer-only-option with baseline commit 0babcb1198f754f6cd37fda7d00287bef8110e1c in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 221 metrics, 9 unstable metrics.

DataDog / dd-trace-py