reportportal / agent-Python-RobotFramework

Robot Framework integration for Report Portal
Apache License 2.0
59 stars 34 forks source link

Skipped steps showing as PASSED in Report Portal UI #166

Closed juhheikk closed 11 months ago

juhheikk commented 1 year ago

Describe the bug Steps that are skipped in Robot Framework test execution (IF/ELSE trees) sometimes show as PASSED in Report Portal UI.

Robot test file:

*** Test Cases ***
First
  [Documentation]  first testcase
  Log  Use builtin logger

#Second
#  [Documentation]  second testcase
#  Fail  utterly

Third
  [Documentation]  third testcase
  ${var_x}=  Set Variable  ${FALSE}
  IF  ${TRUE}
    Log  In true IF
    IF  ${var_x}
      Log  IN IF
    ELSE
      Testing
    END
  END

*** Keywords ***
Testing
  [Arguments]  ${no_test}=${FALSE}  ${test}=${TRUE}
  IF  ${no_test}
    ${status}=  Run Keyword And Return Status  Fail  utterfly
    Log  error
  END
  IF  ${no_test}
    ${status}=  Run Keyword And Return Status  Fail  utterfly
    Log  error
  END
  IF  ${no_test}
    ${status}=  Run Keyword And Return Status  Fail  utterfly
    Log  error
  END
  IF  ${no_test}
    ${status}=  Run Keyword And Return Status  Fail  utterfly
    Log  error
  END
  IF  ${no_test}
    ${status}=  Run Keyword And Return Status  Fail  utterfly
    Log  error
  END
  IF  ${no_test}
    ${status}=  Run Keyword And Return Status  Fail  utterfly
    Log  error
  END
  IF  ${test}
    ${status}=  Run Keyword And Return Status  Log  test
    Log  testing
  END

Steps to Reproduce Steps to reproduce the behavior:

  1. run with pabot ($ pabot test.robot)
  2. use post_report to send results ($ python3 post_report.py -v RP_ENDPOINT:http://localhost:8080 -v RP_UUID:xx-xx-xx-xx-xx -v RP_PROJECT:local_demo -v RP_LAUNCH:superadmin_TEST_EXAMPLE -v RP_ATTACH_LOG:True -v RP_SKIPPED_ISSUE:False ./output.xml)

Expected behavior Skipped IF/ELSE blocks are shown as SKIPPED, not PASSED.

Actual behavior Skipped IF/ELSE block are shown as PASSED, not SKIPPED. image

Package versions

$ python3 -m pip list |grep reportportal
reportportal-client         5.3.0
robotframework-reportportal 5.3.2

Additional context Might be related to issue https://github.com/reportportal/agent-Python-RobotFramework/issues/145 (or not).

HardNorth commented 11 months ago

Looks like it's fixed in 5.4.0.