search

jenkinsci / peass-ci-plugin

Jenkins plugin for peass to support performance measurement in CI
https://plugins.jenkins.io/peass-ci/
GNU Affero General Public License v3.0
4 stars 13 forks source link

Check setting to unstable for builds with erroneous tests #137

Closed mawHBT closed 2 years ago

mawHBT commented 2 years ago

What feature do you want to see added?

Marking builds as unstable seems not to work as expected. We have builds (with failOnRtsError: true) containig erroneous tests, which are not marked as unstable, they are green and "successful". So one won't recognize the sporadic errors as long as the build is checked manually.

On the other hand, we have builds (containig erroneous tests) marked as unstable, even with failOnRtsError: false.

Shouldn't the default behaviour not be, to always mark builds as unstable if they contain erroneous tests? By setting failOnRtsError, you could chose to set the build as failed, if errors occured in RTS.

Upstream changes

No response

DaGeRe commented 2 years ago

Thanks for the issue, I created a test for this and a potential fix: https://github.com/jenkinsci/peass-ci-plugin/commit/f6630cf2cbad507150a95d0d96f0eeff9e598efc could you check whether this works for you?

mawHBT commented 2 years ago

Having a failure in a single test causes marking of the build as "unstable" without further settings. By setting failOnRtsError:true, you can explicitly mark the build with "failure" in this case. Seems fine to me.