Migrate sbt-launcher URL to download one for sbt 1.x.
Update plugins versions where required by sbt update.
Change sbt version to be used to latest released at the moment, 1.3.13
Adjust build settings according to plugins and sbt changes.
Migration to sbt 1.x:
enhances dev experience in development
updates build plugins to bring there new features/to fix bugs in them
enhances build performance on sbt side
eases movement to Scala 3 / dotty
No.
All existing tests passed, both on Jenkins and via Github Actions, also manually for Scala 2.13 profile.
There are a few non-trivial changes related to versions.
First of all the original PR was introduced on top of spark 3.1 so we had to adapt a few things for it to work on top of spark 3.0
[Important] The sbt bump introduces a mima bump and that introduces some binary breaks. If comparing the binary breaks with version 2.4 of spark, there are 200+ listed breaks. However comparing with 3.0, we get away with the breaks that are also added as excludes in the original PR (MimaExcludes.scala contains them) and only 3 additional breaks. According to what discussed in this comment from upstream it is correct for us to compare with version 3.0 and the upstream commit included this change here only because they forgot to do it after releasing 3.0. You can see they encountered the same problem upstream with this bump, from this comment.
The breaks are listed under v30excludes instead of the upstream's v31excludes because current version is 3.0.0
[Workaround] The bump broke sbt unidoc. It attempts to generate docs for generated classes and fails with NoClassDef errors. Since we don't care about generated unidocs from sbt build, we can exclude the unidoc task for now.
[Small added detail] We have to avoid loading sbt/build from cache in circle builds because circle cached the existing file that points at old sbt location (bintray).
[Important note] I reduced circle parallelism from 12 to 8 for scala tests because there is an issue where too many parallel tests might interact one with the other and that causes the task to flake immensely. It seems like a lower parallelism reduces the chances of flaking at the expense of bringing the test time up from around 33 minutes to 42 minutes)
Why are the changes needed?
The SBT bump is needed because:
sbt 0.x (current) is only available in bintray, that is gonna shutdown in a matter of days
sbt 0.x introduces some test dependency on scala module 2.10 which is an issue when bumping avro version as 2.12 is needed.
Original PR description
Migrate sbt-launcher URL to download one for sbt 1.x. Update plugins versions where required by sbt update. Change sbt version to be used to latest released at the moment, 1.3.13 Adjust build settings according to plugins and sbt changes.
Migration to sbt 1.x:
No.
All existing tests passed, both on Jenkins and via Github Actions, also manually for Scala 2.13 profile.
Closes #29286 from gemelen/feature/sbt-1.x.
Authored-by: Denis Pyshev git@gemelen.net Signed-off-by: Dongjoon Hyun dhyun@apple.com
Upstream SPARK-XXXXX ticket and PR link (if not applicable, explain)
[SPARK-21708][BUILD] Migrate build to sbt 1.x Commit: https://github.com/apache/spark/commit/6daa2aeb0164277088396102897b2ea4426b9f1c PR: https://github.com/apache/spark/pull/29286
What changes were proposed in this pull request?
Bump sbt version to 1.x.
There are a few non-trivial changes related to versions. First of all the original PR was introduced on top of spark 3.1 so we had to adapt a few things for it to work on top of spark 3.0
v30excludesinstead of the upstream'sv31excludesbecause current version is 3.0.0Why are the changes needed?
The SBT bump is needed because:
Does this PR introduce any user-facing change?
No