search

typelevel / scalacheck

Property-based testing for Scala
http://www.scalacheck.org
BSD 3-Clause "New" or "Revised" License
1.94k stars 406 forks source link

Plan for 1.14.1 and beyond #500

Closed non closed 5 years ago

non commented 5 years ago

Hi folks,

First of all, thanks to @ashawley for working to keep the lights on. I've been using https://github.com/typelevel/scalacheck/issues/453 to help guide things, but I figured making a new issue here to talk about what I'm imagining should happen.

I'm hoping to go through all the current PRs and make a go/no-go decision on whether they can be merged before a release. Given the need to preserve binary compatibility I'm not considering any PRs which would break it until after the release (at least).

At this point there are only two things that might be tricky:

Once master is in a good state, I'll be publishing 1.14.1 to org.scalacheck. I'd really prefer to set up sbt-release and sbt-sonatype (they have made my other projects fairly easy to do releases for) but since we're trying to support two versions of ScalaJS this may not be practical.

Once 1.14.1 is out, I'd like to think about how to evolve the ScalaCheck API. Shrinking has been broken for a long time and something like the strategy Hedgehog uses is probably the right way forward. One idea I had was to use a new package (for example org.scalacheck.v2 or org.typelevel.scalacheck) to create a relatively source-compatible version of the API with major internal changes, while leaving the existing API alone to preserve binary compatibility. Alternately, we could start work on Scalacheck 2.x.

What do folks think? Any thoughts, opinions, or concerns?

non commented 5 years ago

One other question: do people feel strongly that we need to be publishing snapshots?

Travis has been really flaky recently and I'd like to cut down on the number of builds we do. Currently we do 32 distinct JVM builds, whereas I'd be happy just doing 4 (one per Scala version). The "other" builds do things like testing parallel-vs-non-parallel builds, publishing snapshots, 1 worker vs 4 workers, etc.

I'd like to move to a single parallel build with 4 JVM workers for each Scala version. That should cut down on the Travis build times (since this is the fastest variant) and also cut down on the likelihood that a flaky Travis worker fails the build. (EDIT: Maybe it makes sense to do a few extra builds to test both JDK8 and JDK11.)

non commented 5 years ago

cc @SethTisue @ashawley @rickynils et al.

ashawley commented 5 years ago

Thanks for facilitating the 1.14.1 release. I've been surprised by how small changes to ScalaCheck have had secondary effects downstream, so I think it's good we bundle up a release with just conservative fixes for now. It's been a while since a release, so there are a lot of fixes available and it will be a big win to get this out.

I'm hesitant to support forking the ScalaCheck namespace, but I'd need to know what changes we'd be entertaining. I have a list of changes for 2.0 that I've thought of and I'll try to share to contribute to that discussion.

The reliability issues with Travis are unfortunate. I'm not sure if there is a way to reorganize the build to improve the running times or not. There are tradeoffs to the different setups.

In the meantime, should we consider dropping the test size of 1000 to 100 for JVM builds (maybe drop to 50 for JS builds) in Travis?

https://github.com/typelevel/scalacheck/blob/ce0e23ea9c/tools/travis-script.sh#L9

That might give a 25% improvement in Travis. The motivation to identify brittle tests was a good one, but I don't think it's necessary and I don't think it's panned out. I feel like test generators are good enough now that it's not really necessary, either.

non commented 5 years ago

PR which may help address the Travis flakiness: https://github.com/typelevel/scalacheck/pull/503

non commented 5 years ago

I agree that forking the namespace is not ideal (and probably isn't what people are hoping for) but given how rocky the transition to 1.14.0 has been for people (since the last breaking change) I am nervous about trying to do this incrementally with a number of breaking releases. There are so many parts of the API that could really do with a serious clean up, but almost any change we consider will induce some amount of breakage.

We should compare notes for what kind of breaking changes we're thinking of.

rickynils commented 5 years ago

@non I don't see the need for forking the namespace for ScalaCheck 2. Is it to accommodate projects that want to use both ScalaCheck 1.x and 2.x at the same time?

I don't think anyone would expect that moving from 1.x to 2.x would be trivial. We can still maintain 1.x bugfix releases in parallel with 2.x.

non commented 5 years ago

@rickynils That's a good point -- we can use branches for this I guess. I was imagining trying to support some interoperation between 1 and 2 but maybe that would be too complex.

smarter commented 5 years ago

Is it to accommodate projects that want to use both ScalaCheck 1.x and 2.x at the same time?

I think it's more to allow projects to start using scalacheck 2.x even if they depend on some other projects still on 1.x.

ashawley commented 5 years ago

With any luck, @non and I will work on getting 1.14.1 across the finish line this week. The hope is the build can be improved during this period, and the deployment system will be better scripted, as well. This should help with any more 1.14.x releases, should they be necessary, but also better position for the future work on the 1.15.x series. The build is very complex since it needs to publish for multiple targets and versions of targets, so it would behoove us to get this ironed out and make releases reproducible.

With consideration for #498 and Dotty compilation, the plan will be to deprecate Prop.BooleanProperties in 1.14.1 and remove it as implicit in a subsequent 1.15.0 release. This will make 1.15.0 binary compatible, but it will not be source compatible. Other improvements to the library that qualify under this policy will also be considered for 1.15. For instance, there are a couple of opportunities for improving the experience with shrinking for users. Regardless, following up with a 1.15.0 release will provide an opportunity to make more involved improvements and prepare for any significant changes for a 2.0 release.

non commented 5 years ago

As usual things are moving slower than I'd like. If we can get #522 merged I think we're in position for 1.14.1 in the next couple of days.

smarter commented 5 years ago

I think https://github.com/typelevel/scalacheck/pull/423 is ready to be merged too :)

non commented 5 years ago

@ashawley So at this point I've merged #423. If we merge #522 I think we're good for a binary-compatible (and source-compatible) 1.14.1 release. Is there anything else we're missing?

It occurred to me that we could do a 1.14.1-RC1 release to help ensure that we are as compatible as we think we are. But on the other hand MiMA seems OK, and we can rush out a 1.14.2 release if needed. What do you think?

EDIT: Often I don't do a great job of this, but we should probably make sure we have fairly good and informative release notes for this next release. And it occurs to me I need to revisit #502 before the release as well.

ashawley commented 5 years ago

I agree. If we run in to issues, we could release 1.14.2. The Community Build compiles ScalaCheck and runs it through a few high profile downstream projects. Unfortunately, the Community Build has been frozen to a ScalaCheck commit from a few weeks ago. I sense the Scala team is trying to stabilize the compiler as a priority in this moment.

https://github.com/scala/community-builds/blob/de800d4dbf/configs/project-refs.conf#L143

SethTisue commented 5 years ago

I'd be happy to do a run with a newer SHA. (And I agree it would be wise to do so before release.) Ping me when you're ready.

non commented 5 years ago

This sounds good. Once we have a SHA we want to release I'll ping Seth.

non commented 5 years ago

@SethTisue I think the master branch (eb7a85c) is ready to be released -- would you be willing to check it out for us?

SethTisue commented 5 years ago

will do (hopefully today, but it's a travel day so I'm not certain)

SethTisue commented 5 years ago

run queued, but we might not get a result until Saturday, because there's a test of sbt 1.3.0-RC5 ahead of it in the queue

non commented 5 years ago

@SethTisue Thanks for setting it up!

SethTisue commented 5 years ago

refined failed, log is at https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-integrate-community-build/4994/artifact/logs/refined-build.log

(and once that's addressed, six more downstream projects will run)

ashawley commented 5 years ago

Thanks, Seth!

I am unable to interpret that test failure in refined, so I'm not sure what change it could be in ScalaCheck either. I'll see if I can run the community build for refined locally. If I can reproduce it, I can bisect ScalaCheck to find the cause.

non commented 5 years ago

@ashawley I was able to reproduce it locally after doing publishLocal from master (for 2.12) and updating refined's dependencies.

non commented 5 years ago

@ashawley I made some progress, see #530.

non commented 5 years ago

@SethTisue Looks like this was a genuine bug. Thanks for catching it for us!

I have a fix: https://github.com/typelevel/scalacheck/pull/531/files

Once that is merged, I think we'd be OK to try again. That said, we do have some other minor PRs in flight (around help messages) so we could wait until things shake out a bit.

non commented 5 years ago

@SethTisue It's the weekend, but master should be ready to go once you're able to queue it up again.

SethTisue commented 5 years ago

queued: https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-integrate-community-build/4999/

SethTisue commented 5 years ago

run is 🐢🥒🥦💹🇨🇨🦠💚

JD557 commented 5 years ago

Is there anything still holding up the 1.14.1 release?

Right now, it's really hard to track the current progress... From this thread and from the open PRs, it looks that everything is ready (and some PRs are waiting to be merged only after 1.14.1 is released). Do you need help with some issues that are blocking the new version?

non commented 5 years ago

I just released 1.14.1-RC1.

The release process was held up by me not being able to figure out how to release successfully, which I think I just did. Once it's hit Maven Central I'd like for a few folks to help confirm that the JVM, JS, and Native flavors released successfully (this is especially important since we're cross building for two different versions of both JS and Native). Assuming things are fine, we can officially put out the 1.14.1 release.

I'll be sending a PR soon with documentation on how to do the release.

non commented 5 years ago

Here's the PR I made as part of the release: https://github.com/typelevel/scalacheck/pull/535

non commented 5 years ago

If you try out the RC and it works (or doesn't) please let us know here. I'm specifically hoping that folks who are interested in the bleeding edge versions of ScalaJS and Scala Native will check out their respective versions and make sure I published everything correctly.

JD557 commented 5 years ago

Right now, I can only confirm that it's working fine (for me) in the JVM and #447 is finally fixed! 🎉

Regarding scala.js, I had some small problems with the "1.14.1-RC1" version that are not present in the "1.14.0" version, namely:

org.scalajs.core.ir.IRVersionNotSupportedException: Failed to deserialize info of file compiled with Scala.js 1.0.0-M8 (supported: 0.6.0, 0.6.13, 0.6.3, 0.6.17, 0.6.4, 0.6.6, 0.6.14, 0.6.8, 0.6.5, 0.6.15):

The scala.js tutorial still uses version 0.6.28. I'm not a scala.js user, so I don't know if this is an actual problem or if "everyone" is using the unstable 1.0.0-M8 version.

I couldn't test scala-native since (AFAIK) it doesn't support scala 2.12, and I don't have any scala 2.11 project 😞

non commented 5 years ago

This issue was also reported by @xuwei-k; it seems like the publish for Scala.js 0.6.28 didn't work correctly: https://github.com/typelevel/scalacheck/issues/536

non commented 5 years ago

Just published 1.14.1-RC2 which should fix the IR error. 🤞

non commented 5 years ago

I did some more testing today and things look OK.

Unless problems with RC2 are discovered my current plan is to cut the release this Wednesday. I know @ashawley has been working on release notes, which we'll be including as well.

non commented 5 years ago

Now that 1.14.1 is out (https://github.com/typelevel/scalacheck/releases/tag/1.14.1) I propose we close this issue and track other work in finer-grained tickets.

JD557 commented 5 years ago

It seems like something is still wrong with the 1.14.1 release process, the documentation was not published successfully: https://www.scalacheck.org/documentation.html points to https://www.scalacheck.org/files/scalacheck_2.11-1.14.1-api/index.html

I can open another issue, but this still seems related to this.