RP makes each keyword as a separate test in launch

[petr0ff]

Running single test-case from suite results in 100+ tests, RP counts each keyword as a separate test.

I have a suite (.txt file) with about 10 tests, and run single test via -t arg, like this: sudo robot -x junit.xml --listener robotframework_reportportal.listener --variablefile=resources/stages/conf_main.py --variablefile=resources/stages/conf_rp.py --variable RP_LAUNCH:QA_German_TestJob #29 -t f321_watch_empty_asset mpx/create_assets/incomplete_packages.txt

where conf_rp.py is the file with RP parameters, mpx/create_assets/incomplete_packages.txt is the suite name.

So, I run it from Jenkins and assume that it will result in a launch on my project and set Total tests to 1 and it will be passed. But here is what I see in RP (launch_log.png), RP assumes each test instruction, each KEYWORD as a test :)

There are some conditional keywords that RP assumed as failed and it resulted in an error:

16:22:39 Traceback (most recent call last):
16:22:39   File "/usr/local/lib/python2.7/dist-packages/reportportal_client/service_async.py", line 204, in process_item
16:22:39     getattr(self.rp_client, method)(**kwargs)
16:22:39   File "/usr/local/lib/python2.7/dist-packages/reportportal_client/service.py", line 140, in finish_launch
16:22:39     return _get_msg(r)
16:22:39   File "/usr/local/lib/python2.7/dist-packages/reportportal_client/service.py", line 22, in _get_msg
16:22:39     return _get_data(response)["msg"]
16:22:39   File "/usr/local/lib/python2.7/dist-packages/reportportal_client/service.py", line 34, in _get_data
16:22:39     raise ResponseError(error_messages[0])
16:22:39 ResponseError: 4065: Cannot finish launch '595d12183cdea200067133e9' with calculated automatically status 'FAILED' as 'PASSED'. Did you provide correct status in request?

Here how it looks in RP:

I've attached Jenkins' output here job_log.txt

[Tset-Noitamotua]

Also each single step of a keyword which is imported from a resource file is counted as a single test.

[DzmitryHumianiuk]

@petr0ff @Tset-Noitamotua this actually can be covered with BDD feature this will collapse all keywords as a steps of scenario.

currently it's under development https://github.com/reportportal/reportportal/issues/83

[petr0ff]

@DzmitryHumianiuk and there is no way to have robot's run results to be displayed on the usual/regular launches? As I understood, that BDD feature is a separate, specific dashboard. But I need results on a usual launch like it is for other agents (tesntg, junit, nunit, etc.), where launch shows total tests count. For now robot-agent looks completely useless :) no offense please!

[DzmitryHumianiuk]

@petr0ff no. BDD will be project specific config.

Basically, it would calculate statistics at one step higher (count scenarios, not steps/keywords). and have different representation of those level, under launch view.

[petr0ff]

@DzmitryHumianiuk thanks, will wait for implementation.

[livelace]

Hello everybody. I join to @petr0ff and I'm interested in this feature implementation too. @DzmitryHumianiuk This behaviour (show results for every keywords) doesn't have any practical uses. When do you plan to finish and expand community with robotframework users :) ?

[DzmitryHumianiuk]

@livelace we don't have python devs in team. And do expect contribution from community for this update.

BDD (nested step) is on the roadmap. And it can cover needs of listed above. but still, require some updates in python client and agent.

this is a small team of RP, and a huge list of possible integrations and features. Do not hesitate to contribute!

[livelace]

@DzmitryHumianiuk I don't see any dependencies on the robot client, it does things properly - sends test results to RP, RP should show and count those tests in a "right" way (keyword != test).

PS. There is a keyword "Run Keyword And Ignore Error" that RP recognized as failed (but it wasn't) and didn't allow to finish a test suite.

ResponseError: 4065: Cannot finish launch '5a2acd7524aa9a0001fbeb25' with calculated automatically status 'FAILED' as 'PASSED'. Did you provide correct status in request?

PSPS. Anyway, lets wait BDD feature implementation (March 2018 ?) and see it again.

[ailjushkin]

Yes, this is known issue. To be able to adequate log test items, you should change type of keyword for a test. There should be additional level where type is defined.

1. If test item is setup, then type = BEFORE_SUITE
2. If test item is teardown, then type = AFTER_SUITE
3. If test item is keyword, then type = STEP
4. If test item has no keywords, then type = SUITE
5. If test item has keywords, then type = TEST

After labeling test item as a TEST, your results should be placed into low level log.

[livelace]

@ailjushkin Hello. Did I understand properly that you develop a new version of robotframework-reportportal-ng ?

[ailjushkin]

@livelace yeah... that's kinda refreshed listener and there is no mentioned problem. but it still needs to be updated to maintain new BDD style in report portal 4.0

[livelace]

@DzmitryHumianiuk It seems that the community already exists :) @ailjushkin It works :) Instead of a bunch of results I see now what really happens. Is there tags support ?

[DzmitryHumianiuk]

@livelace

[moijes12]

@DzmitryHumianiuk It seems that the community already exists :) @ailjushkin It works :) Instead of a bunch of results I see now what really happens. Is there tags support ?

@livelace Does it now report the result at test case level ? If yes, how did you get it to work. Please help as I am facing the same problem.

@DzmitryHumianiuk How long would it be before the BDD feature is available ? If there is help required to get this, I can contribute

[DzmitryHumianiuk]

@moijes12 we do it as a part of migration to PostgreSQL. a lot of stuff have to be changed under the hood.

BDD feature, as a part of statistics calculation. and the way how agents send data to RP.

as for statistics, new object so called nested steps should be introduced. Practically, it will be equal part to testItem, but will have no statistics. And will be a container for logs.

It's a long story with this feature. It have been re-designed like 6 times already. refactored api available here.

btw, there is a good workaround https://github.com/reportportal/agent-Python-RobotFramework/issues/13#issuecomment-350972256

just update you levels in agent for now, and it will represent it more closely to BDD style.

[testautomation]

Hi! what's the status of this issue. Do you have any update?

btw. I am back in the game to continue where my best friend tset-noitamotua left off 🤣

Just set up RP v4.3 with one dummy test which consists of three keywords. It's displayed in lauches as three tests

Not sure what to do with info in https://github.com/reportportal/agent-Python-RobotFramework/issues/13#issuecomment-350972256

To be able to adequate log test items, you should change type of keyword for a test.

Where do I change type of keyword?

Another question: What about above mentioned BDD feature? Is it available in RP v4.3? Is it a custom dashboard/widget I have to configure in the GUI?

[iivanou]

@testautomation Hi, BDD feature is available in RP version 5. This issue will be resolved in agent that will support RP version 5.

[testautomation]

Is where a workaround I can use meanwhile on v4.3?

[DzmitryHumianiuk]

@testautomation the only option for 4.3 to have keywords as a part of Test is to modify robot agent. trick is here: https://github.com/reportportal/agent-Python-RobotFramework/blob/master/robotframework_reportportal/service.py#L144

keywork sent as test_item type, which can be suite or test itself. but if you will change it to log record - you will get it as log, and your upper level will become a test.

Version 5 is officially released. We have implemented Nested Step feature in it https://github.com/reportportal/reportportal/issues/275 dev guide https://github.com/reportportal/documentation/blob/master/src/md/src/DevGuides/reporting.md#start-childnested-step-item

basically you in same place, you just can send RQ as

        start_rq = {
            "name": keyword.get_name(),
            "description": keyword.doc,
            "tags": keyword.tags,
            "hasStats": false,
            "start_time": timestamp(),
            "item_type": keyword.get_type()
        }

where "hasStats": false, is a key

approach how to use it in Java https://github.com/reportportal/client-java/wiki/Nested-steps

BTW. Demo instance is running on latest v5 https://web.demo.reportportal.io/ui/

[iivanou]

Fixed in client version 5.0.2. Will not be fixed for Report Portal versions below 5.