search

reportportal / agent-python-pytest

Framework integration with PyTest
Apache License 2.0
94 stars 102 forks source link

Reporting against pre-existing suite (parent) #294

Closed davebrochu-es closed 2 years ago

davebrochu-es commented 2 years ago

name: Bug report about: Create a report to help us improve title: Reporting against pre-existing suite (parent) for multiple tests fails labels: bug assignees: ''

Describe the bug I would like to use a "launch" in report portal to cover multiple phases/runs of pytests for a specific system configuration so that I can include pre-upgrade/mid-upgrade/post-upgrade results in a single place. I'd like the organization in report portal to look like the following

Ex:

launch - "System Functional Testing of Configuration ABC" (id =12345)
  | ----> suite - "Before Upgrade - version 1.5.0" (id = 54321)
    | -----> Test Case - "Verify foo"
    | -----> Test Case - "Verify bar"
    | -----> Test Case - ....
 | ----> suite - "During Upgrade - version 1.5.0"
    | -----> Test Case - "Verify foo"
    | -----> Test Case - "Verify bar"
    | -----> Test Case - ....
 | ----> suite - "After Upgrade - version 1.5.0"
    | -----> Test Case - "Verify foo"
    | -----> Test Case - "Verify bar"
    | -----> Test Case - ....

To accomplish this, I have a call to create a launch outside of the pytest reportportal plugin. I save the launch ID and then use that value to create a suite (again outside the pytest reportportal plugin). I then have both a launch ID and suite ID.

I feed these values to the reportportal pytest plugin --reportportal -o rp_launch_id=12345 -o rp_parent_id=54321 (using sample IDs from example above) My expectation when I provide these values is that when I run pytest, any test I run will appear as a test case under the suite "Before Upgrade - version 1.5.0" (id = 54321)" (example above). What actually happens is that the first test that executes properly nests as a test case under the suite, but all the remaining tests that run show as siblings of that suite in the launch (rather than children).

When I look at debug output, I see that the next test to run actually starts a suite, with no parent_item_id set, and with item_type = SUITE, which explains why the remaining tests that run appear as siblings to the manually created suite.

Is my expectation that providing a parent_id to the pytest plugin should result in that parent being used for ALL tests within the test run? If not, is there a way I can accomplish the nesting outlined above with this plugin?

Steps to Reproduce Steps to reproduce the behavior:

  1. Create a launch using the python reportportal client. Save the returned ID
  2. Create a suite using the python reportportal client while setting the launchId to the ID returned in #1. Save the returned suite ID
  3. Pass the suite ID as the parent ID and the launch ID as the launch ID to the reportportal pytest plugin.

Expected behavior All test runs in the pytest execution should report as a child of the parent indicated.

Actual behavior Only the first executed test appears as a child to the parent suite. Other tests appear as siblings to the suite.

Package versions 5.6.0

iivanou commented 2 years ago

@davebrochu-es Hi. The rp_parent_id option is not currently implemented in the agent. I think it can be accomplished.

davebrochu-es commented 2 years ago

Thanks @iivanou . I'd be happy to take a stab at adding support. Looking through the code though, I do see that rp_parent_item_id is used. https://github.com/reportportal/agent-python-pytest/search?q=rp_parent_item_id

davebrochu-es commented 2 years ago

I'm not as familiar with the codebase, but I think the issue lies in this line https://github.com/reportportal/agent-python-pytest/blob/325f56938009ae89ae7739410adcccc0d94f2cde/pytest_reportportal/service.py#L324

When a user passes rp_parent_item_id from the command line, they are indicating all tests are children of the identified parent (suite in this case). But after each test executes, the parent_id gets overwritten to None. This explains why I am seeing the first test properly added to the parent suite, but all others that follow after the first completes end up as their own suite.

Is there an additional variable that could be used such that if a user specifies from the command line we don't reset the parent to None?

davebrochu-es commented 2 years ago

I see that this was merged to master, but how would I go about using it? It doesn't look like there is a new version in PyPI.

iivanou commented 2 years ago

I see that this was merged to master, but how would I go about using it? It doesn't look like there is a new version in PyPI.

We do not publish a new release on PyPi for each change usually. You can download a new code directly from the master branch or apply the fix locally until a new package version is released.

davebrochu-es commented 2 years ago

We do not publish a new release on PyPi for each change usually. You can download a new code directly from the master branch or apply the fix locally until a new package version is released.

Thanks. Is there a rough timeline when the next release would come out?