mdjastrzebski
commented
4 months ago The first picture you attached seems to show reasonably stable results. Although I am not sure what exactly is running there, as this seems to be some custom report, not the one produced by Reassure by default.
The second picture seems to have some significant variation (~3k secs vs ~5k secs) in the first line. I can only guess where does it come from. My recommendation would be to make sure that your CI agent is not running any other jobs during the performance test runs as they will significantly impact your measurements. This is something that Reassure as a tool has no control over, as it's your CI setup.
yakupdurmus
Describe the bug I have integrated my Reassure tests into a Bitbucket pipeline. My tests run on the pipeline, and I display a summary of my test report in the reports section on Bitbucket. However, the issue is that the tests do not consistently yield the same results when executed.
Reassurance base branch also works. Then it works in the branch where development is carried out and produces the results. This process is done every time a pipeline is triggered.
When I produce the result several times, there is sometimes a big difference between them.
To Reproduce Steps to reproduce the behavior:
Go to the Bitbucket pipeline with Reassure tests. Run the pipeline. Again Run the pipeline Compare the difference between two test results. See the inconsistency in the test results. Expected behavior I expected the tests to produce consistent results every time they are run on the pipeline.
Screenshots
As shown in the images, the inconsistency in test results undermines my confidence in Reassure.