search

reportportal / agent-Python-RobotFramework

Robot Framework integration for Report Portal
Apache License 2.0
59 stars 33 forks source link

In a test execution with an IF-ELSE IF condition sometimes the ELSE IF block is being marked as Failed even when the IF condition succeeds when logging to reportportal #129

Closed prajeeth07 closed 2 years ago

prajeeth07 commented 3 years ago

name: Bug report title: 'Issue observed when logging the results from robotframework back to reportportal' labels: bug assignees: ''

Describe the bug When tests are executed on Docker, sometimes for tests having an IF-ELSE IF logic, the ELSE IF block is marked as Failed (however, the steps inside the ELSE block are shown as Skipped.) on reportportal. The HTML report generated by Robotframework shows the test as passed. A similar issue is observed sometimes on the high-level keyword which is marked as Failed when none of the lower level keywords has failed - only on reportportal results.

Steps to Reproduce

Expected behavior The step should be marked as Skipped similar to robotframework reportportal results

Actual behavior Sometimes the ELSE IF or ELSE block is marked as failed even when the IF block is successful and marked as Passed

Screenshot: ReportPortal screenshot for the test executed on docker: image

Robotframework HTML report attached along with the above execution. Only report.html is attached: image

Robotframework HTML report when executed locally with detailed steps: image

Package versions -Lib Versions used - robotframework - 4.0.3 robotframework-reportportal - 5.2.0

Note: The issue is observed intermittently on random tests. The test in the above-attached screenshots does not fail every time.

HardNorth commented 3 years ago

@prajeeth07 It looks like a very complex case. Could you please provide a project to reproduce the issue?

prajeeth07 commented 3 years ago

@HardNorth I did try to increase the log level to TRACE and debug this but there is no information logged on why it happens. Seems like an issue with the listener when logging back to reportportal - not entirely sure though. So when a keyword has IF / ELSE it fails (rather than marking it as Skip) as I mentioned above OR at times just marks the high-level keyword which has an IF/ELSE block as Failed even though none of the steps inside the keyword fails

Also, as I mentioned above this happens intermittently, so not sure how to share an example for reproducibility. I will try to get back to you with a sample test using a public website if possible.

iivanou commented 3 years ago

The listener sends out to RP exactly what it receives from the Robot Framework. There is not additional logic hidden in the agent that may impact keyword/test/suite status.

prajeeth07 commented 3 years ago

@iivanou understood, But if you look at the screenshots above, 1st is the screenshot from reportportal where the test is marked as Failed. 2nd screenshot is the attached robotframework HTML report for the same run which shows the test as passed. I was not able to add the detailed step screenshot from the same run as currently the functionality only allows to attach report.html file.

HardNorth commented 3 years ago

@prajeeth07 We won't able to debug without an example project :-/

HardNorth commented 2 years ago

Abandoned