Truncate irrelevant parts of stacktrace

[socksy]

Resolves #58

This is directly based on @pesterhazy's PR https://github.com/lambdaisland/kaocha/pull/321

What I've added:

• made the binding dynamic, making it possible to rebind in your tests.edn
• added a line to the docs
• added tests for both existing stacktrace filtering, and stacktrace truncating
• removed orchestra from default stacktrace filtering, since people might use it in their own code (ala @tonsky's comment)

[pesterhazy]

Very nice – I'll close my PR in favor of this one!

Currently failing the tests though:

FAIL in config.bindings/stacktrace-shortening (test/features/config/bindings.feature:110)
the output should not contain
expected: (not (str/includes? (:out m) output))
  actual: (not (not true))

[alysbrooks]

Thanks for picking this up, @socksy! I think we can likely merge this after the failing test is fixed on CI. (Strangely, it passes when I run it locally, but I don't have time to dig in further, unfortunately.)

[socksy]

Yeah, it works for me locally when developing, so I figured this must somehow be a CI runner setup? Unfortunately I'm away from the keyboard for at least the next 2 weeks and unable to debug for a while.

Is the CI running a different command than when you test locally? Or is this maybe a Linux/Mac thing?

[socksy]

I also noticed that the CI error has

(Rest of stacktrace elided)

which is explicitly disabled in the tests.edn, for obvious reasons 😅. Can it be that this file is ignored when running the tests in CI?!

[alysbrooks]

@socksy I think that's from within the test of eliding itself, so the stack eliding (elision?) is configured by the test rather than the overall project settings.

My suspicion is that this test might be inheriting a setting from the overall test suite that's only enabled on CI and is causing kaocha.ns to appear apart from the stacktrace itself because the error is coming from line 110 of the feature test.

[alysbrooks]

Trying to mimic the CI invocation locally didn't reproduce the CI failure. A couple things I tried:

• bin/kaocha --reporter documentation integration --plugin cloverage --codecov
• bin/kaocha --plugin :kaocha.plugin/profiling --plugin :kaocha.plugin/capture-output --plugin :kaocha.plugin.alpha/info --plugin :kaocha.plugin/cloverage --print-env --reporter kaocha.report/documentation --codecov --focus 'config.bindings/stacktrace-shortening'

This is pretty strange, since the differences should be pretty marginal between my environment and CI if I pass all those flags, especially since I'm running a Linux system. The main difference remaining is that CI is a headless environment, but I'm pretty sure the only part of Kaocha that is affected by that is the notification plugin.

[alysbrooks]

Turns out the missing ingredient was setting the CI environment variable. (You can also use --profile ci). I knew Kaocha pays attention to that flag, but I didn't realize any stacktrace stuff was affected by it. I'm not actually sure how to fix it, though. Getting closer, though!

[codecov[bot]]

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.04% :tada:

Comparison is base (7dcb230) 75.10% compared to head (8ed006d) 75.15%.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #420 +/- ## ========================================== + Coverage 75.10% 75.15% +0.04% ========================================== Files 52 52 Lines 2804 2809 +5 Branches 289 290 +1 ========================================== + Hits 2106 2111 +5 Misses 511 511 Partials 187 187 ``` | [Flag](https://app.codecov.io/gh/lambdaisland/kaocha/pull/420/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) | Coverage Δ | | |---|---|---| | [integration](https://app.codecov.io/gh/lambdaisland/kaocha/pull/420/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) | `56.31% <100.00%> (+0.07%)` | :arrow_up: | | [unit](https://app.codecov.io/gh/lambdaisland/kaocha/pull/420/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) | `69.41% <60.00%> (-0.02%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files Changed](https://app.codecov.io/gh/lambdaisland/kaocha/pull/420?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) | Coverage Δ | | |---|---|---| | [src/kaocha/stacktrace.clj](https://app.codecov.io/gh/lambdaisland/kaocha/pull/420?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None#diff-c3JjL2thb2NoYS9zdGFja3RyYWNlLmNsag==) | `87.80% <100.00%> (+1.69%)` | :arrow_up: |

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[socksy]

@alysbrooks wdyt? seems to finally pass CI :)

[plexus]

Thanks, @socksy

[plexus]

Released in v1.87.1366

[lambdaisland/kaocha "1.87.1366"]                 ;; deps.edn
{lambdaisland/kaocha {:mvn/version "1.87.1366"}}  ;; project.clj