Open vmassol opened 6 years ago
bbaudry
commented
6 years ago thanks @vmassol for running this experiment. It will be super interesting to see how the Descartes metrics (score, nb live mutants, nb killing tests, etc.) evolve over time Performance is a challenge, which we need to discuss next week.
monperrus
commented
6 years ago Hi Vincent,
FYI I've implemented on XWiki (in xwiki-commons to be precise) the strategy of breaking the build when the mutation score computed by Pitest/Descartes is below a defined threshold (see https://markmail.org/message/bonb5f7f37omnnog ).
Excellent!
What threshold do you use? It would be great to write a tutorial documenting how to do this!
See you next week,
--Martin
vmassol
commented
6 years ago You're going a bit fast @monperrus ;)
I still need to validate a few things before I can write about this:
Re thresholds, I use the mutation score computed by pitest/descartes and set is as the threshold. The idea is to progress, and not to reach a given value. Same strategy that we're applying with test coverage and jacoco.
Thx. Stay tuned.
monperrus
commented
6 years ago Hi,
Martin, are you thinking of a tutorial internal to STAMP? Or a tutorial/tool demo at a conference, which would also be useful internally, but also gives some visibility externally
Tutorial deadline for ASE is April 30 Tool demo deadline for ASE is May 31
Best regards,
Brice.
-----Original Message----- From: stamp-request@ow2.org [mailto:stamp-request@ow2.org] On Behalf Of Martin Monperrus Sent: onsdag 28. mars 2018 09.48 To: STAMP-project/docs-forum reply@reply.github.com; stamp@ow2.org Cc: Subscribed subscribed@noreply.github.com Subject: Re: [stamp] [STAMP-project/docs-forum] Use of Pitest/Descartes (#11)
Hi Vincent,
FYI I've implemented on XWiki (in xwiki-commons to be precise) the strategy of breaking the build when the mutation score computed by Pitest/Descartes is below a defined threshold (see https://markmail.org/message/bonb5f7f37omnnog ).
Excellent!
What threshold do you use? It would be great to write a tutorial documenting how to do this!
See you next week,
--Martin
vmassol
commented
6 years ago @monperrus and @oscarlvp (cc @danglotb): I did some tests on score evolution in one xwiki-commons module and reported my findings at https://github.com/STAMP-project/pitest-descartes/issues/54
In short: it seems the strategy is not working that well. It's not completely useless but less interesting than I thought it would be.
Feel free to comment on https://github.com/STAMP-project/pitest-descartes/issues/54
Thx
monperrus
commented
6 years ago Hi Brice,
I'm thinking of a written document for the general audience. It would be self-contained and it could be understood by people without the Stamp context. This document would increase the visibility and impact of our work.
That said, it's excellent idea to make a "live tutorial" at a conference!
Cheers,
--Martin
--
Martin Monperrus, Professor, KTH Royal Institute of Technology
On Wed, 2018-03-28 at 08:18 +0000, Brice Morin wrote:
Hi,
Martin, are you thinking of a tutorial internal to STAMP? Or a tutorial/tool demo at a conference, which would also be useful internally, but also gives some visibility externally
Tutorial deadline for ASE is April 30 Tool demo deadline for ASE is May 31
Best regards,
Brice.
-----Original Message----- From: stamp-request@ow2.orgmailto:stamp-request@ow2.org [mailto:stamp-request@ow2.org] On Behalf Of Martin Monperrus Sent: onsdag 28. mars 2018 09.48 To: STAMP-project/docs-forum reply@reply.github.com<mailto:reply@reply.github.com>; stamp@ow2.orgmailto:stamp@ow2.org Cc: Subscribed subscribed@noreply.github.com<mailto:subscribed@noreply.github.com> Subject: Re: [stamp] [STAMP-project/docs-forum] Use of Pitest/Descartes (#11)
Hi Vincent,
FYI I've implemented on XWiki (in xwiki-commons to be precise) the strategy of breaking the build when the mutation score computed by Pitest/Descartes is below a defined threshold (see https://markmail.org/message/bonb5f7f37omnnog ).
Excellent!
What threshold do you use? It would be great to write a tutorial documenting how to do this!
See you next week,
--Martin
FYI I've implemented on XWiki (in xwiki-commons to be precise) the strategy of breaking the build when the mutation score computed by Pitest/Descartes is below a defined threshold (see https://markmail.org/message/bonb5f7f37omnnog ).
However it has an important performance cost. For example on xwiki-commons: