both org.scala-lang and org.typelevel versions of scala-library are sometimes pulled in

[tpolecat]

Via @nigredo-tori here is a reproduction that shows both libs being pulled in.

tmp$ mkdir tls-test
tmp$ cd !$
cd tls-test
tls-test$ cat - > build.sbt
scalaVersion in ThisBuild := "2.12.3"
libraryDependencies += "org.http4s" %% "http4s-dsl" % "0.17.0-M3"
tls-test$ mkdir project
tls-test$ cat - > project/build.properties
sbt.version=0.13.16
tls-test$ sbt 'show managedClasspath' | grep 'scala-library'
[info] Resolving org.scala-lang#scala-library;2.10.6 ...
[info] Resolving org.scala-lang#scala-library;2.10.6 ...
[info] Resolving org.typelevel#scala-library;2.12.1 ...
[info] * Attributed(/Users/rnorris/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.3.jar)
[info] * Attributed(/Users/rnorris/.ivy2/cache/org.typelevel/scala-library/jars/scala-library-2.12.1.jar)
tls-test$ 

Adding an exclusion "fixes" the problem, however one could imagine cases where this wouldn't work if the libs diverge.

tls-test$ cat - >> build.sbt 
excludeDependencies += "org.typelevel" % "scala-library"
tls-test$ sbt 'show managedClasspath' | grep 'scala-library'
[info] Resolving org.scala-lang#scala-library;2.12.3 ...
[info] * Attributed(/Users/rnorris/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.3.jar)
tls-test$ 

Can we review what the desired/expected behavior is here? @eed3si9n has offered to help us track this down. As a lead-in perhaps this observation would be of use.

[rossabaker]

if the libs diverge

My understanding of the compatibility guarantee is that this should not happen: all code compiled with TLS should link with the LBS runtime, and vice versa. Am I reading too much into that?

I've received a few complaints from https + sbt-assembly users. Perhaps the right thing to do for library authors using TLS is to post process the POM to reference LBS?

[nigredo-tori]

all code compiled with TLS should link with the LBS runtime

I assume you mean "of the same version" here. E.g. code compiled against TLS 2.12.3 should work with LBS 2.12.3. However, once SBT sees the two as separate libraries, the versions can diverge. In my case I had TLS 2.12.1 with LBS 2.12.3. Here excluding TLS works, since it's replaced with a backwards compatible version. If, however, I had TLS 2.12.3 and LBS 2.12.1, this would have had the opposite effect (making the crash deterministic).

[rossabaker]

I also thought that scala-library was bidirectionally compatible across patch releases since 2.10. I would rewrite the POM from TLS-2.12.3 to LBS-2.12.3, so we'd be fine, but I'm also surprised that patch versions matter.

[tpolecat]

Re: divergence I think we're seeing it. LBS added Properties.coloredOutputEnabled and TLS added ValueOf, so if you need both you're out of luck. If I'm reading the smoke correctly.

[milessabin]

There is a TLS version with both.

[tpolecat]

So which library should people be getting? Should sbt always prefer TLS?

[milessabin]

As far as I can tell the problem here is the mix of 2.12.1 and 2.12.3. You should always use the most recent. If you're using ValueOf then TLS-4 for 2.12.3 is what you need. But if you're not using ValueOf then LBS 2.12.3 should be fine.

[tpolecat]

How should sbt resolve this? Find the most recent version and select TLS for it? So given LBS 2.12.3 and TLS 2.12.1 it should go for TLS 2.12.3? What if it doesn't exist yet?

[eed3si9n]

Would it make sense to put the diverged bits to another library that gets added in addition to the org.scala-lang/Lightbend standard library?

[milessabin]

I think that if http4s is publishing binaries built with TLS they really need to republish for the most recent TLS version. I can commit to getting a TLS version out within a single digit number of days of the corresponding LBS release.

[tpolecat]

I think this diminishes the appeal of TLS quite a bit. The promise has been that you can use it without any extra requirements for you or for your downstream users. What do we think of Eugene's idea? If TLS didn't publish scala-library at all then we might have an easier path.

[milessabin]

Is http4s using ValueOf?

[milessabin]

Looking at http4s master it seems not. In fact it looks as though the only TLS feature it's using is -Ypartial-unification which is included in 2.12.x anyway. So I'm not quite sure I understand why it's being published for 2.12.x with a TLS dependency at all.

[rossabaker]

http4s is publishing with the latest TLS version, and does not use ValueOf. We adopted TLS early to help test it, with the understanding that it was transparent to downstream users. We cross build for 2.11, so we still need partial unification by some means.

I like Eugene's idea for resolving the scala-library conflict. However, I expect a TLS-compiled library would still have a declared org.typelevel runtime dependency on that extras library, even if that dependency is not used. So it would resolve the conflict but still not achieve transparency.

[milessabin]

Then I'm confused ... the http4s artefact being pulled in in the example at the top of this ticket is relative to 2.12.1 ... is there a later published artefact that is built against 2.12.3 which should be used instead?

[milessabin]

@nigredo-tori's comment above,

I assume you mean "of the same version" here. E.g. code compiled against TLS 2.12.3 should work with LBS 2.12.3.

is spot on ... as I said above the issue is the mixing of 2.12.1 and 2.12.3 irrespective of TLS vs. LBS.

[rossabaker]

@nigredo-tori mentioned http4s-0.17.0-M3 on Gitter, which depends on TLS 2.12.1. http4s-0.17.0-RC1 depends on TLS' 2.12.3-bin-typelevel-4. To achieve transparency, I think we need to be producing POMs that declare LBS 2.12.3, either by reverting to LBS compiler or by POM postprocessing. I want to show our support for TLS, so a revert makes me sad and postprocessing makes me nervous.

I remain confused why mixing 2.12.x and 2.12.y should ever cause a problem, as those are supposed to be bidirectionally compatible.

[milessabin]

I remain confused why mixing 2.12.x and 2.12.y should ever cause a problem, as those are supposed to be bidirectionally compatible.

They are within an organization. The problem here is mixing 2.12.x from LBS with 2.12.y from TLS.

Also TLS x.y.z is backwards compatible with LBS x.y.z.

So, you can fix the version and vary the organization, or fix the organization and vary the version, but not both at the same time.

[rossabaker]

And this problem mixing organizations applies through transitive dependencies? I would think this mixing would be quite common:

• foolib builds against LBS 2.12.x
• barlib builds against TLS 2.12.y
• my app depends on foolib and barlib -- what versions of Scala are safe for my app?

[SethTisue]

I remain confused why mixing 2.12.x and 2.12.y should ever cause a problem, as those are supposed to be bidirectionally compatible

note that scala-reflect.jar and scala-compiler.jar require an exactly matching scala-library.jar. you won't always have trouble in practice if there's a mismatch, but you might. for example, here's a ticket where scala-library 2.12.2 plus scala-reflect 2.12.3 equals boom: https://github.com/tpolecat/doobie/issues/568 . @wsargent was bitten recently by this, too, in a different codebase.

[fommil]

@SethTisue there is a counter example in an older version of scala, which is why sbt doesn't enforce that by default. I'd much prefer enforcing scalaVersion across all the scala artefacts as you suggest... so I do that in my standard sbt plugins.