Allow duplicate tests on JiraUseCase

[MinZe25]

Is your feature request related to a problem? Please describe. It's not possible to have two tests related to a single jira usecase.

Describe the solution you'd like Having an environment variable to control if it's active or not.

Additional context The problem it's on the method: matchTestIssuesAgainstTestResults on JiraUseCase.

[serverhorror]

I think that would also have a positive effect on sub tests.

Quite often used in Go and Python.

[metmajer]

@MinZe25 the ability to report back the test result to Jira requires a 1:1 relationship. I suggest you break down your test case into several ones if you require multiple implementations.

[serverhorror]

@metmajer

I don't think that is a good choice. Subtests are recommended as best practices now.

Especially since test parametrization is quite useful to implement tests for a single ticket and have these tests cover all the cases that are relevant for a ticket.

Here's a simple Go example:

package main_test

import (
    "fmt"
    "testing"
)

func divisableByTwo(v int) error {
    if v%2 != 0 {
        return fmt.Errorf("not divisible by two")
    }
    return nil
}

func TestTICKET123(t *testing.T) {
    tc := []struct {
        v       int
        wantErr bool
    }{
        {v: 1, wantErr: true},
        {v: 2, wantErr: false},
    }
    for idx, tt := range tc {
        tName := fmt.Sprintf("Test-%d", idx)
        t.Run(tName, func(t *testing.T) {
            if err := divisableByTwo(tt.v); (err != nil) && !tt.wantErr {
                t.Error(err)
            }
        })
    }
}

Output

Running tool: C:\Program Files\Go\bin\go.exe test -timeout 10m -run ^TestTICKET123$ example.invalid/go-subtests

=== RUN   TestTICKET123
=== RUN   TestTICKET123/Test-0
=== RUN   TestTICKET123/Test-1
--- PASS: TestTICKET123 (0.00s)
    --- PASS: TestTICKET123/Test-0 (0.00s)
    --- PASS: TestTICKET123/Test-1 (0.00s)
PASS
ok      example.invalid/go-subtests     0.212s

> Test run finished at 12/21/2022, 7:37:43 PM <

XML Output

<?xml version="1.0" encoding="UTF-8"?>
<testsuites tests="3">
        <testsuite name="" tests="3" failures="0" errors="0" id="0" hostname="..." time="0.000" timestamp="2022-12-21T20:07:23+01:00">
                <testcase name="TestTICKET123" classname="" time="0.000"></testcase>
                <testcase name="TestTICKET123/Test-0" classname="" time="0.000"></testcase>       
                <testcase name="TestTICKET123/Test-1" classname="" time="0.000"></testcase>       
                <system-out><![CDATA[coverage: [no statements]
ok      example.invalid/go-subtests     0.203s  coverage: [no statements]]]></system-out>
        </testsuite>
</testsuites>

Here's a simple Python example (for unittest and pytest):

from typing import Optional
import unittest
from dataclasses import dataclass

import pytest
import pytest_subtests as subtests

def divisableByTwo(v: int) -> Optional[Exception]:
    if v % 2 != 0:
        raise Exception("not divisable by two")
    return None

@dataclass
class Testcase:
    v: int
    wantErr: bool
    raises: Exception

class TestExampleSubtest(unittest.TestCase):
    def test_TICKET123(self):
        tc = [
            Testcase(v=1, wantErr=True, raises=Exception),
            Testcase(v=2, wantErr=False, raises=None),
        ]
        for tt in tc:
            with self.subTest("TICKET-123", i=tt.v):
                if tt.wantErr:
                    self.assertRaises(tt.raises, divisableByTwo, tt.v)
                elif not tt.wantErr:
                    self.assertEqual(divisableByTwo(tt.v), None)

def test_permutations(subtests):
    tc = [
        Testcase(v=1, wantErr=True, raises=Exception),
        Testcase(v=2, wantErr=False, raises=None),
    ]
    for tt in tc:
        with subtests.test():
            if tt.wantErr:
                with pytest.raises(tt.raises):
                    divisableByTwo(tt.v)
            elif not tt.wantErr:
                assert divisableByTwo(tt.v) == None

Output

(.venv) > python -m pytest --junitxml=tests.xml -o junit_family=xunit2 --cov-report term-missing --cov-report xml --cov=src -o testpaths=tests -v
==================================================================== test session starts ====================================================================
platform win32 -- Python 3.10.6, pytest-7.2.0, pluggy-1.0.0 -- .venv\Scripts\python.exe
cachedir: .pytest_cache
rootdir: python_subtests
plugins: cov-4.0.0, subtests-0.9.0
collected 2 items

subtests_test.py::TestExampleSubtest::test_TICKET123 PASSED                                                                                            [ 50%]
subtests_test.py::test_permutations SUBPASS                                                                                                            [100%]
subtests_test.py::test_permutations SUBPASS                                                                                                            [100%]
subtests_test.py::test_permutations PASSED                                                                                                             [100%]
.venv\lib\site-packages\coverage\inorout.py:520: CoverageWarning: Module src was never imported. (module-not-imported)
  self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")
.venv\lib\site-packages\coverage\control.py:825: CoverageWarning: No data was collected. (no-data-collected)
  self._warn("No data was collected.", slug="no-data-collected")
WARNING: Failed to generate report: No data to report.

.venv\lib\site-packages\pytest_cov\plugin.py:311: CovReportWarning: Failed to generate report: No data to report.

  warnings.warn(CovReportWarning(message))

===================================================================== warnings summary ======================================================================
subtests_test.py:15
  python_subtests\subtests_test.py:15: PytestCollectionWarning: cannot collect test class 'Testcase' because it has a __init__ constructor (from: subtests_test.py)
    @dataclass

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
----------- generated xml file: python_subtests\tests.xml -----------

---------- coverage: platform win32, python 3.10.6-final-0 -----------

====================================================== 2 passed, 1 warning, 2 subtests passed in 0.10s ======================================================
(.venv) >

xml Output

<?xml version="1.0" encoding="utf-8"?>
<testsuites>
  <testsuite name="pytest" errors="0" failures="0" skipped="0" tests="4" time="0.117" timestamp="2022-12-21T20:08:19.911246" hostname="...">
    <testcase classname="subtests_test.TestExampleSubtest" name="test_TICKET123" time="0.003" />
    <testcase classname="subtests_test" name="test_permutations" time="0.013" />
  </testsuite>
</testsuites>