search

typelevel / scalacheck

Property-based testing for Scala
http://www.scalacheck.org
BSD 3-Clause "New" or "Revised" License
1.93k stars 404 forks source link

Verify only props matching the given TestSelectors #1031

Open Duhemm opened 5 months ago

Duhemm commented 5 months ago

Previously, ScalaCheck ignored the selectors that it receives as input in the "root task". This prevented users from running only a subset of properties in a specification by passing their TestSelector to ScalaCheck in a TaskDef.

This patch fixes this by having the root task program the execution of only the requested properties if the TaskDef contains only TestSelectors. ScalaCheck's behavior remains unchanged if the TaskDef contains any other kind of Selector.

Duhemm commented 4 months ago

Hi @satorg! Would you mind taking a look at this PR, please?

Duhemm commented 2 months ago

@satorg Hi! Any concerns with this PR?

satorg commented 2 months ago

Hi @Duhemm ! Thank you for the ping. I'm really sorry that it takes so much time to review your PR.

Honestly, I made a couple of attempts to look into it, but every time was realizing that I'm not really sure what the problem the PR is aimed to solve.

It does not mean that there's anything wrong with your PR, but rather it seems that I've never encountered that kind of issue in my personal experience.

So I would really appreciate if you could elaborate a bit more on the issue the PR is addressing, maybe with some really small examples. I believe it could help to move it forward.

Anyway, thank you for contributing to the project!

Duhemm commented 4 days ago

Hi @satorg, thanks for taking a look and sorry for taking so long to get back to you!

sbt's test-interface API has a concept of Selector which can be passed to a test framework, so that only the selected subset of the entire test suite is run. They make it possible to encode, in a programmatic and framework agnostic way, the command line filter that most test frameworks also support (eg. in ScalaCheck: -propFilter).

For instance, Bloop uses TestSelector to ask test frameworks to run only a selected subset of test cases within a test class: https://github.com/scalacenter/bloop/blob/c505385edf0bbd420e19401ae1beabae8895df8f/frontend/src/main/scala/bloop/engine/tasks/TestTask.scala#L343-L348

In my case, I am working on a plugin that adds test retry in sbt. When a test case ("property", in ScalaCheck) fails, it is automatically retried a few times until it passes so that the build is not blocked by flaky tests. The changes in this PR will make it possible for the plugin to retry only the failing test case instead of the entire test object.

Please let me know if you have other questions or if the benefits of this PR are still unclear.