Open jazerix opened 9 months ago
I have added a test case for this scenario :)
@bitcoder will this be merged?
@Testinator-X I don't know at this point. Needs to be analyzed as other pending issues. I don't have any ETA for it, sorry
@jazerix , would the originating JUnit XML leading to this issue be something like this?
<testsuites name="test suites root">
<testsuite failures="0" errors="0" skipped="1" tests="1" time="3049" name="tests.LoginTests">
<properties>
<property name="setting1" value="True"/>
<property name="setting2" value="value2"/>
</properties>
<testcase classname="tests.LoginTests" name="dummy" time="159">
<skipped type="pytest.skip" message="Please skip">skipped by user</skipped>
</testcase>
<testcase classname="tests.LoginTests" name="dummy" time="121">
<failure type="pytest.failure" message="Fail due to...">failed due to...</failure>
</testcase>
<testcase classname="tests.LoginTests" name="dummy" time="650">
</testcase>
</testsuite>
</testsuites>
@bitcoder, yes exactly, whenever the classname would overlap :)
is there an ETA on this merge?
is there an ETA on this merge?
Hi everyone, and happy new year. No, currently there's still no ETA.
In JUnit XML files where multiple test_case ids overlap, TestRail CLI currently reports both of them. This behavior results in only the last being "persisted" on the platform, which is especially problematic, when the first test_id fails and subsequent ones succeeds, as the platform now reports the test as being successful.
In instances where testcase ids collide, TestRail should opt for a pessimistic approach where the negative outcome takes the highest priority.
Issue being resolved: https://github.com/gurock/trcli/issues/161
Solution description
During upload,
data_provider/api_data_provider.py
now checks if another tests case is to be submitted with the samecase_id
, if not the test case is added to the list of `bodies.If the code encounters the same case id, one of two things happen:
This is a very simplistic approach that only takes the
failure
status code into account.Changes
A
takes_priority(self, statusId1, statusId2)
method has been added that returns true or false if the first statusId takes priority (is a failure):This method definitely leaves a lot to be desired, and it could be worth looking into creating a more sophisticated way of ordering the different status codes.
Appending a result body has been refactored and extracted to its own method:
append_result_body(self, bodies, case)
It is probably also worth noting that this implementation also does not take into account overlapping test_ids from different files.
Potential impacts
This change specifically impacts instances where JUnit xml files have repetitions of the same test_id. However, it should not have any impact on other instances.
All tests pass.
Steps to test
PR Tasks