search

scala / scala-jenkins-infra

A Chef cookbook that manages Scala's CI infrastructure.
https://scala-ci.typesafe.com
Apache License 2.0
14 stars 17 forks source link

start publishing Scala snapshots again #133

Closed SethTisue closed 3 years ago

SethTisue commented 8 years ago

so that people can specify their Scala version as e.g. 2.11.8-SNAPSHOT and 2.12.0-SNAPSHOT and it just works.

this was formerly handled on the old CI at e.g. https://scala-webapps.epfl.ch/jenkins/view/2.11.x/job/scala-nightly-main-2.11.x/ but has not been moved to the new CI

we want to publish them as snapshots, available from e.g.https://oss.sonatype.org/content/repositories/snapshots/org/scala-lang/scala-library/2.11.8-SNAPSHOT/ and https://oss.sonatype.org/content/repositories/snapshots/org/scala-lang/scala-library/2.12.0-SNAPSHOT/

we are already publishing distributions nightly to e.g. http://www.scala-lang.org/files/archive/nightly/2.11.x/ but they are distribution packages only, not usable via sbt's set scalaVersion := ...

reported/requested by @dragos. he writes:

there’s three use cases that come to mind:

  • reproducing a bug on “master" before reporting it
  • integration builds with “fragile” projects. Besides IDE (which is quite tightly coupled to the compiler), I think scala-async or scala-meta might do this.
  • benefit from a bug fix in the presentation compiler (like the one Jason just fixed). There are no more milestones on 2.11.8 (edited)

but the issue is I have no idea what git hash I want, I just want the latest one. It’s an integration build. I want to see if anything broke since yesterday

Anyway, the reason nobody else said anything might also be because this wasn’t discussed or announced anywhere, internally or externally. People might simply not know (2.11.8-SNAPSHOT still resolves to the Sept. 15 version)

SethTisue commented 8 years ago

@lrytz comments:

just need to find the right spot in https://github.com/scala/scala/blob/2.11.x/scripts/jobs/validate/publish-core

SethTisue commented 8 years ago

the question arose whether we actually prefer "nightly" builds or "mergely" builds ("mergely" meaning "whenever a PR is merged")

dragos commented 8 years ago

There's no point in mergely builds. Maven for sure, and Sbt probably, won't check more often than once a day for SNAPSHOT updates.

You could either designate one mergely build as the nightly that gets published as a snapshot, or build and publish once a day, off master (whatever branch that is).

SethTisue commented 8 years ago

We discussed this at the Scala team meeting today and agreed that we (meaning me) will get the snapshots going again just like before. I'm likely to tackle it sooner rather than later just to get it over with.

SethTisue commented 8 years ago

@smarter wants to do this for Dotty as well

SethTisue commented 8 years ago

re mergely vs. nightly, we average 1 or 2 merges/day, so there's no major difference in cost. @retronym says "end users don’t care” so "just do whatever’s easier"

SethTisue commented 8 years ago

fwiw (it's perhaps useful to know that the causes had nothing to do with e.g. how we talk to Sonatype)

(Adriaan later disabled both jobs)

SethTisue commented 8 years ago

our existing nightly job is integrate/bootstrap (script contents, Jenkins job). the job description claims "This job is run on every merge" but it's clear from the run log that it's actually just running nightly. I don't see in anything our current setup that runs mergely. (I guess the lack of that is https://github.com/scala/scala-jenkins-infra/issues/9.)

tweaking integrate/bootstrap to do this looks to me like it will be easier than messing with validate/publish-core. note that integrate/bootstrap is also how we publish real Scala releases, so we know that it has the right Sonatype smarts. whereas publish-core looks to me like it's oriented around publishing to Artifactory (https://scala-ci.typesafe.com/artifactory), not Sonatype.

so, it looks like nightly snapshots it will be.

SethTisue commented 8 years ago

some investigation of the existing integrate/bootstrap script:

SethTisue commented 8 years ago

the version numbers of the old snapshots published from the old Jenkins looked like e.g. 2.11.8-20150915-122802-bb52b24bab (this is what I got when I did ;resolvers += Resolver.sonatypeRepo("snapshots") ;scalaVersion := "2.11.8-SNAPSHOT" just now).

is the SNAPSHOT stuff picky about the exact format of our version numbers? do I need to match the old format, or is it OK to stick with the 2.11.8-72b855f-nightly style format? (it's kind of nice to have the date right there in the number, alongside the SHA)

SethTisue commented 8 years ago

integrate/bootstrap is pretty much just calling ant publish-signed — there is a great deal of logic in build.xml that does the rest. for example there are lines like:

<if><contains string="${maven.version.number}" substring="-SNAPSHOT"/><then>
      <property name="remote.repository" value="${remote.snapshot.repository}"/>

so it looks like if we set the variables up right, Ant will take care of publishing to https://oss.sonatype.org/content/repositories/snapshots and not https://oss.sonatype.org/content/repositories/releases

SethTisue commented 8 years ago

it appears to me that just setting publishToSonatype to yes (while leaving SCALA_VER_BASE unset) might actually do the right thing. YOLO so here goes: https://scala-ci.typesafe.com/job/scala-2.11.x-integrate-bootstrap/367/consoleFull

EDIT: hmm though, I can see in the run log already we're getting SCALA_VER=2.11.8-9ed51f2-nightly and later publishSonatype is going to do ant -Dmaven.version.number=$SCALA_VER $publishSonatypeTaskCore so that looks wrong already. we don't want to specify -Dmaven.version.number to ant here — if we don't, I think it will attach SNAPSHOT itself?

EDIT 2: regardless, I'm going to let that one run to the end, so I can see what ends up getting published and what the delta is between that and what we want.

EDIT 3: ++ publishToSonatype=no haha I failed to notice these lines in the script:

      # TODO: publish nightly snapshot using this script - currently it's a separate jenkins job still running at EPFL.
      publishToSonatype="no"
kiritsuku commented 8 years ago

The Scala IDE build has been broken for a few days. Probably because you managed to publish new nightlies - at least I can see new ones. For some reason our test suit fails because scala-actors.jar could not be found. I'm investigating but still don't understand why the test suite can break because scala-actors is also published. Did you change anything related to it?

dragos commented 8 years ago

The problem is that the scala-actors jar has a weird (different) version number. For sanity reasons we expect all scala-* bundles to have the same version number. Here's the issue (from MANIFEST.MF in the respective jars):

scala-actors: 2.11.8.v20151124-185958-8363c20-nightly-8363c20de8
scala-compiler: 2.11.8.v20151124-185958-8363c20de8

Previously, scala actors had this version number:

Bundle-Version: 2.11.8.v20150915-122802-bb52b24bab
SethTisue commented 8 years ago

thanks for the details. I'll attend to this as soon as I can. (customer work is cutting in front.)

dragos commented 8 years ago

Just wondering: isn't this breaking all PR validation runs?

SethTisue commented 8 years ago

PR validation goes through https://scala-ci.typesafe.com/artifactory/scala-pr-validation-snapshots/ and shouldn't, as far as I can see, be affected by what's going on at over at oss.sonatype.org

dragos commented 8 years ago

thanks @SethTisue! I'm pretty sure this affects releases as well, not just snapshots.

(If I recall correctly, this will be on the critical path to delivering that work to the said customer).

SethTisue commented 8 years ago

pretty sure this affects releases as well, not just snapshots

not sure what you mean? as far as I know, I haven't done anything that would affect anyone who isn't using a -SNAPSHOT version number.

this will be on the critical path to delivering that work to the said customer

indeed. I'm not talking about a delay of weeks, I just mean I can't be sure I'll get to it today or tomorrow.

dragos commented 8 years ago

I was thinking that the way the OSGi version number is computed is the same for releases and snapshots, and this might be related to moving the build to Sbt. So the mismatch between the version of scala-library/compiler/reflect and that of actors might exist for releases, too.

SethTisue commented 8 years ago

ah. no, I'm working in ant-land for the time being, though I'll probably switch over to sbt-land soon; I'll coordinate with @szeiger

kiritsuku commented 8 years ago

On 11/30/2015 05:24 PM, Iulian Dragos wrote:

Just wondering: isn't this breaking all PR validation runs?

I wondered about that too and found out that they disabled it. ;) They are on 2.12 nowadays but our codebase is not and therefore PR validation can't succeed.

kiritsuku commented 8 years ago

We fixed our failing test in https://github.com/scala-ide/scala-ide/pull/1042, therefore from our side there is no pressure to get a change here.

SethTisue commented 8 years ago

cool, thanks for letting me know. sorry about the trouble.

dragos commented 8 years ago

I still think the versions should be the same (and not sure if this will continue to work for a real release). But for the moment our nightlies are back up. But please come back to this eventually.

SethTisue commented 8 years ago

absolutely.

SethTisue commented 8 years ago

on my snapshots-return branch I did some partial work on this (enough to get the main Scala jars publishing, not the modules yet though), but it's on hold at the moment while we complete the ant->sbt transition, at which point I'll pick it up again

SethTisue commented 7 years ago

new arguments for picking this up again:

new arguments against:

I've unassigned myself. I'll see if someone else on the Scala team is interested. I'll also ask scala/contributors

kiritsuku commented 7 years ago

I don't get why publishing nightlies is difficult to achieve. In my book this is just a sbt publishLocal on the CI infrastructure and that's it. Don't know if publishLocal is supposed to work for scala/scala though.

Beside from that we don't really need to consume nightlies of the Scala anymore in Scala IDE. Since no one is actively working on the PC, it is unlikely that a lot of changes are introduced in the nightlies that would be useful for the IDE. And we could consume the artifacts that are published after successfully building a PR if we really wanted.

SethTisue commented 7 years ago

I don't get why publishing nightlies is difficult to achieve

we already publish nightlies. this is about publishing -SNAPSHOT versions, which is a little different.

I don't claim it's difficult to achieve. it might be fairly easy, now that the sbt build is in place.

dragos commented 7 years ago

Scala IDE was supposed to consume the nightlies not because of new features but for dog-fooding. As you already remarked, it's very hard to move to 2.12 now because of all the little regressions and other internal changes that slowly creeped in.

Besides that, I thought 2.11.x will be sort of a long-term support version, given that 2.12 requires Java 8. Maybe I misunderstood you, but it seems too quick to dismiss 2.11, compared to how long 2.10 lived by 2.11 in the past, 2.11 having a close-to-zero cost to upgrade.

szeiger commented 7 years ago

Modules would need to be handled differently. They have their own version number and are cross-versioned for a particular Scala version (like any regular Scala library). This doesn't work for snapshots. You would need to use the same snapshot version for all modules and stop cross-versioning.

SethTisue commented 7 years ago

I thought 2.11.x will be sort of a long-term support version

we have already announced multiple times, over a pretty long timespan now, that 2.11.9 will be the last "planned" release in the 2.11.x series.

so after 2.11.9 comes out the development velocity on 2.11.x will become zero unless some critical problem is found, so there will be no real need to have -SNAPSHOTs for it (especially given that we have nightlies)

dragos commented 7 years ago

I am not sure about the long timespan, but you are right. There's an email on scala internals (Oct. 25) and the 2.12 release notes. :)

I agree that at this point it makes no sense to have SNAPSHOTS for the 2.11 series. It would make sense for 2.12, if downstream projects like ensime and scala IDE want to have some form of CI, but that's another story.

SethTisue commented 7 years ago

I am not sure about the long timespan

http://www.scala-lang.org/news/2016-schedule.html, published December 2015, says "We expect to conclude the 2.11.x series with 2.11.8 in Q1 2016 and 2.11.9 in Q3 2016"

dragos commented 7 years ago

All right, I stand corrected!

SethTisue commented 7 years ago

instructions for using a -nightly instead of a -SNAPSHOT: http://stackoverflow.com/q/40622878/86485

dwijnand commented 3 years ago

Using scala-runners:

What's the version for the latest mergely?

$ scala -2.13.head --print-scala-version
2.13.4-bin-57ae2a6

What's the version for the next version, having been validated by the community build?

$ scala -2.13.next --print-scala-version
2.13.4-bin-55feb1a
SethTisue commented 3 years ago

meh, mergelies are fine, we don't need snapshots