Provide copy-and-paste unparsed test "string"

[Adal3n3]

Problem to solve

Initially, we received feedback that our test names were too long and unreadable, so we parsed them into "Class Name" and "Test Name." However, we have since heard from developers who want to search for the failed test in their terminal, and the parsed names don't provide an easy copy-and-paste experience. Since we prioritize a developer-first experience, we should provide the full string that is easy to copy and paste.

For example In our current PR:

graphql_api/tests/test_owner.py::TestOwnerType::test_fetch_current_user_is_not_okta_authenticated

Steps to Reproduce / Current UX

1. update the PR comment design
2. tests dashboard

Solution

Developers need the entire failed test string to allow them to copy and paste it into their terminal to find the test.

Action Checklist for Issue Creation

• [ ] Add the design project (to the right under "projects" > settings icon > select "design team")
• [ ] Share in relevant Slack channel (#discuss-codecov-product-design) for visiblity
• [ ] Add targeted launch date/delivery date as/if needed

Additional Information

[Adal3n3]

You can find the design here on Figma

[Adal3n3]

This work might require 1-2 sprints.

[rohan-at-sentry]

This will require implementation on a language by language basis. For example

Pytest allows

1. Running all tests within a module pytest test_mod.py where test_mod.py is the whole module (contains many tests)
2. Running all tests in a directory pytest testing/
3. Running tests by keyword expressions pytest -k "MyClass and not method
4. Running a specific test within a module pytest test_mod.py::TestClass::test_method

for Pytest we'd likely want to support case 4

Vitest allows

1. Running all tests within a "file" npm test -- SomeTestFileToRun.

There is no support for running individual test.

Jest allows

1. A single test to be run node <path-to-jest> -i <your-test-file> -c <jest-config> -t "<test-block-name>" {this is super hacky}
2. A single test jest my-test
3. Running tests that match a pattern jest -t 'fix-order-test'
4. Running all tests in a file jest path/to/my-test.js

Of the 4 options for jest, we'd want to support 3

Ruby allows

1. A single test to be run rails <path to test file.rb> -n "testname"
2. All tests in a given file `rails

TLDR

1. For python, we'll need test file path, class name, method name. (This is also true for Maven AFAICT)
2. For Ruby, Vitest and Jest, we'll need test file path and test name

[Adal3n3]

For PHPunit test:

This is the example from Michi: https://github.com/getsentry/sentry-symfony/actions/runs/10280796079/job/28454975345?pr=861

Michi will type his own script: vendor/bin/phpunit --filter End2EndTest::testMessengerCaptureHardFailure

Format: To rerun a specific failed test: path/to/file/phpunit --filter ClassName::testMethodName

[Adal3n3]

For Jest and Vitest:

• Jest and Vitest debugging workflows are identical.
• Workflow 1: rerun the test suite ---> (eg. test/sdk/api.test.ts ) expensive to run but most commonly used
• Workflow 2: test/**/*.test.ts ---> will run everything
• Workflow 3: test --watch ---> run in watch mode (press F to only run failed tests)

Meeting recording: https://sentry.rewatch.com/video/wnmln0deuhqkq5si-vitest-and-jest-c-p-test-name-september-26-2024

All of these different packages have the same test name and the same path or the same file name because of the structure of our conventions. And so we actually use the full path to identify this properly for us.

We learned the test name is not unique so developers usually use the full file path to locate their test.

---

Format: To rerun a specific failed test: path/to/file > test name (Confirmed w @joseph-sentry )

---

Note from @joseph-sentry

By default Jest does not include the filepath whereas vitest does

Joey has modified the doc so users will ask to include the filepath.

We need to update the UI onboarding to include this change. @ajay-sentry here is the issue: https://github.com/codecov/engineering-team/issues/2605

[Adal3n3]

Completed w all languages.