search

reportportal / agent-Python-RobotFramework

Robot Framework integration for Report Portal
Apache License 2.0
59 stars 34 forks source link

post_report.py to send results asynchronously #176

Closed juhheikk closed 8 months ago

juhheikk commented 11 months ago

Is your feature request related to a problem? Please describe. We have over 800 testcases in a single report that needs to be sent to ReportPortal. The speed with SaaS instance currently seems to be aprx. 3-4 testcases per minute, therefore it would be nice to able to just send the results quickly and let the reportportal do the processing. The .xml file is a bit over 400MB, and .xml file upload size is currently limited to 32Mb so we can't even just do the import which would be even nicer. We don't wish to spend 4hours in sending test results to ReportPortal after each test run, 5-10minutes should do it.

Describe the solution you'd like Would be nice to be able to send the test results in under 10minutes to ReportPortal, perhaps by making the post_report.py to process and send the results to ReportPortal asynchronously.

Describe alternatives you've considered Alternatively we have been discussing about the sending the whole .XML file to ReportPortal, but due to current limitations that is not possible either.

HardNorth commented 8 months ago

@juhheikk Please use agent version 5.5.0 which was released just now. It's already faster than previous versions, but for better performance you can play around RP_CLIENT_TYPE configuration variable. For me the best performance is for ASYNC_THREAD value.

HardNorth commented 8 months ago

@juhheikk UPD after a lot of testing. It looks like the order in which the agent reports steps is really matter for our particular case (Robot Framework agent), unfortunately. That means if a paren reported as PASSED step, but its child reported as FAILED after the parent was finished the whole case will be marked as FAILED. For async clients this happens regularly, since parents don't really have to wait for all children. And especially often this happens for Robot Framework because it has plenty of constructions such as Expected to Fail, where a child should fail and it's in the process of reporting, but parent already reported as PASSED, so the last status overrides everything. That's true even for post_report.py, but happens relatively more rarely.

So my recommendation to use default values for RP_CLIENT_TYPE (default is SYNC). It's faster, than previous versions, but not light speed fast.

HardNorth commented 8 months ago

@juhheikk UPD2 I do believe this is not a client/agent issue, agent reports everything accurate and Async client does exactly what it's supposed to do: do not wait for other requests/responses. So it looks like we need a server fix.