search

reportportal / agent-Python-RobotFramework

Robot Framework integration for Report Portal
Apache License 2.0
59 stars 34 forks source link

Rerun feature merges test cases with different attributes and puts them under different Test Suite #190

Open petr-kus opened 1 month ago

petr-kus commented 1 month ago

Describe the bug When I rerun my launch with more robot framework test suites, they are almost identical except for the station/configuration (e.g., they have the same name). During the merging process, Report Portal ignores the differing attribute "hostname: [name_of_the_station]" in both suites and tests. As a result, Report Portal merges all tests with the same name under a different test suite (likely the last one processed).

This leads to a state in the Report Portal results where I have three test suites, but only "Steps Before" and "Steps After" are correct. The rest of the test cases are incorrectly clustered in one of the three test suites.

Interestingly, during the initial run, the first import placed tests in the correct suite. However, after some time, they are regrouped into the wrong test suite.

The bug image: the_bug

Steps to Reproduce Difficult to describe. First, you need to understand our test infrastructure capabilities.

We can restart/relog the station as part of the test. (It leads to the situation when the test case is rerun and firstly fails with fail "I need to restart" and after restart it starts again and the result is pass). We run the same test suite on different configurations and stations in parallel.

To resolve correct grouping test cases for restart I need to run the launch from the beginning as a rerun (you are not supporting this possibility - btw. I will probably report this as a wanted new feature).

I wrote a workaround 1 for our infrastructure: 1) Firstly I create a launch through your API as an envelope. I get UUID 1 2) Almost immediately I stop this launch with state INFO. 3) Rerun this empty launch (I use rerunof with specific UUID 1 ) and take new UUID 2 (end of workaround 1) 4) after that started a bunch of the same test suites on different stations in parallel (tested 3). The stations have the same UUID 2 of the launch in the Report portal RF Agent and have rerun enabled. Each Test and Suite has as part of attributes hostname which is different according to station.

Actual behavior 5) when I watch results in RP firstly are correctly sorted SUITE 1 has TC's from suite 1. But after restart the station / or after finished run from different stations the TC disappears from SUITE 1 and shows as Rerun of TC in SUITE 2 (for example). Btw. in reruned TC is only one hostname (but there was 3 diffrent).

Expected behavior

  1. I would expect that I have safe things under the right test suite! (don't move them between test suites. Especially when I don't want! - create a new switch for this?).
  2. Configurations (described in attribute hostname) are taken into consideration during the merge (not merge if there is a difference in attribute). According to the description of deep merge and description in the rerun implementation task, it seems to me like a BUG.

https://github.com/reportportal/reportportal/issues/363

Package versions Server Reporportal - Build: 5.9.0 should be reportportal-client==5.5.6 robotframework-reportportal==5.5.3 robotframework==5.0.1

Additional context There is probably a workaround 2 to put different names to test cases. When I tried it behaved correctly. (but I could not test it fully - our restart infrastructure doesn't support yet variables in test case names for RF).

More detailed description of workaround 2: In Roboto Framework is possible to put variables as part of the test case name. When you put there an ID of Test Suite run the Test Cases are getherd and sorted correctly. But, it is a big disadvantage during the analysis, because the same test cases only with different run stations have different names!

Additional Images test_suites_overview first_testsuite workaround_2

Big Thank You! To fix this https://github.com/reportportal/agent-Python-RobotFramework/issues/178 - at least I can somehow put attributes to Test Suites through the meta-data feature in RF! And it seems to be a working feature as expected - I tested it for you :-) ! (maybe except for the merge? - can relate to this bug...)

HardNorth commented 3 weeks ago

Duplicates: https://github.com/reportportal/reportportal/issues/2294

petr-kus commented 3 weeks ago

@HardNorth, I strongly disagree with closing this as a duplicate of this issue. These are absolutely two different things.

Why not duplicate: Why? For example, when I define test_case_id:[some_id] for my test cases, I still have the issue that my test cases in different executions with different parameters/variables/attributes are merged together. So, it is not connected with the auto-generation of test_case_id at all.

Core of the issue: Moreover, I don't know how to put things inside parameters that are considered during the merging.

I think the core of the problem is the parameters that are sent to ReportPortal from the Robot Framework agent. How could I define them (parameters) from Robot Framework for a specific test (test case)? If somebody answers this question, it will probably solve my issue. I will just put some kind of ID of a specific station/configuration there.

My workaround: I already had to write a workaround in our infrastructure using other Robot Framework listeners that run sooner than reportportal agent listener to rename test cases and test suites during the run to be unique according to attributes/executions that I want to play a role in merging. This was done to artificially include "parameters" in the merge through names and tree structure. This obviously affects the dashboards and AI, so it is not the best workaround.