Resource should be covariant

[djspiewak]

It's a concrete datatype. To that end, it should be covariant. It's only abstractions which should be invariant.

[kevinmeredith]

It's a concrete datatype. To that end, it should be covariant. It's only abstractions which should be invariant.

Could you please explain why, i.e. why concrete datatypes should be covariant, but abstractions should be invariant?

I ask to learn, not dispute.

[djspiewak]

@kevinmeredith Great question!

I have a longer explanation of this that I'm writing up, but there are a couple short form versions that I'll try to give here. The absolutely easiest short-form version is simply an unsatisfying appeal to authority: scalaz was variant in all the things back in 7.0, and it was ripped out in 7.1 because it just… didn't work. Compiler bugs are often blamed for this, but actually it's some more fundamental things about the language and subtyping in general.

The problem is that things start to go very, very viral throughout an ecosystem as soon as you make abstractions specifically variant. Remember that F[_] doesn't mean "invariant type constructor F[_]" in the same way that class F[A] means "invariant". Instead, the abstraction F[_] actually means "the polyvariant type constructor F[_]". That's a very significant difference. F[+_] may only be instantiated with covariant type constructors, while F[_] may be instantiated with co-, contra-, and invariant type constructors, making it considerably more general.

As soon as you start constraining your abstract type constructors to have a particular variance, you find that you need to reflect these changes elsewhere throughout your ecosystem, otherwise the polyvariance vs specific-variance distinction trips you up. As a simple example, if we made Sync take a F[+_] instead of just F[_], then it would become impossible to define a Sync instance for WriterT, without making its F[_] also covariant, but then that propagates further as it has its own constraints, forcing variance into Applicative, Monad, and finally (devastatingly) Equal.

You don't want Equal to be contravariant in its type parameter, because that can yield some very strange type inference artifacts, ambiguities, and can allow some things to typecheck which you do not want to typecheck. But if it's not contravariant, then you end up with existential type bound hell everywhere that it intersects with your now-covariant monad transformers, typeclasses, and such. It's a huge mess.

In the end, it's just… not tractable. It's unfortunate because variance has a lot of nice surface properties, and it would be quite nice if we could use it more aggressively. For example, the following catches me every damn time:

if (...)
  fa.map(a => Some(a))
else
  fa.as(None)

The above will work just fine if fa is IO[A], but it won't work if it's an abstract F[A]… unless that F[A] is F[+_]. So it would be really, really nice if we could do this, but honestly we cannot.

One area that I think might be interesting to explore is an implicit liskov which allows us to give a more controlled form of subtype variance, encoded via implicit resolution. Basically it would result in implicit calls to widen as necessary. I think that would be a really interesting space to explore, but I haven't had time to work on it.

[neko-kai]

@djspiewak

and finally (devastatingly) Equal

Sorry, but I'm not following why? Defining these instances seems to work out fine with an invariant Eq, unless I'm confusing something:

import cats.kernel.Eq

final case class WriterT[F[+_], +W, +A](run: F[(W, A)])

object WriterT {
  implicit def catsDataEqForWriterT[F[+_], L, V](implicit F: Eq[F[(L, V)]]): Eq[WriterT[F, L, V]] =
    Eq.by[WriterT[F, L, V], F[(L, V)]](_.run)
}

final case class EitherT[F[+_], +E, +A](run: F[Either[E, A]])

object EitherT {
  implicit def catsDataEqForEitherT[F[+_], L, R](implicit F: Eq[F[Either[L, R]]]): Eq[EitherT[F, L, R]] =
    Eq.by[EitherT[F, L, R], F[Either[L, R]]](_.run)
}

[djspiewak]

@neko-kai I'll have to dig up my notes. Like I said, I have a much longer explanation for this written up somewhere. It most definitely doesn't work when you allow it to go viral to a full framework like cats or scalaz, which is why scalaz backed away from it in 7.1.

[kubukoz]

So here we're looking at Resource[+F[_], +A]?

[alexandru]

Out of curiosity, what would +F help with? I think I've seen it in fs2.

[SystemFw]

yeah, in fs2 we actually moved away from

explore is an implicit liskov which allows us to give a more controlled form of subtype variance, encoded via implicit resolution

which had turned out to be a mess, to normal variance plus some tricks.

@alexandru it helps when having things like Stream[Pure, A] (Stream.range, take and so on), combining them with effectful stream (e.g. concurrent ones), and have the resulting types be inferred.

Michael has written a great post about it https://mpilquist.github.io/blog/2018/07/04/fs2/

[kubukoz]

It probably doesn't make much sense here, as Resource is not pure by definition. Also, effects don't subclass each other. So maybe just +A for now...

Although I wonder how it'd work in situations where a covariant monad transformer was F.

Edit: words

[alexandru]

Resource isn't impure by definition, its usage might not make sense without an effect type, but I believe we only depend on Bracket.

Being too clever about covariance might not be wise, so just +A is fine.

[neko-kai]

@kubukoz

Also, effects don't subclass each other.

With bifunctor IO they do, i.e. F[Nothing, ?] <:< F[Throwable, ?]. Having a covariant F means one can avoid boilerplate such as .mapK(Lambda[F[Nothing, ?] ~> F[Throwable, ?]](f => f)) when dealing with a Resource[F[Nothing, ?], A]

There are some implicit resolution bugs associated with a covariant effect parameter, but they might not affect Resource, since its usage is similar to fs2.Stream – which behaved very well in my experience.

[djspiewak]

I'd be okay with Resource[+F[_], +A]. As @neko-kai points out, it helps partially-applied covariant constructors quite a bit, and I doubt it introduces any problems in this scenario, but I'd like to see it attempted. I'm always caught off guard by variance issues.

I was originally thinking of Resource[F[_], +A].

[smarter]

You don't want Equal to be contravariant in its type parameter, because that can yield some very strange type inference artifacts, ambiguities, and can allow some things to typecheck which you do not want to typecheck.

I'd also be interested in hearing more about this, in Dotty Eql is contravariant (see multiversal equality), this works better than in Scala 2 for implicit resolution because of https://github.com/lampepfl/dotty/blob/d7896b54df4affc8b7cef47a6796664f5449e48e/compiler/src/dotty/tools/dotc/typer/Applications.scala#L1327-L1356, and @milessabin implemented something similar in scalac under a flag: https://github.com/scala/scala/pull/6037 which can hopefully be made default in 2.14 and allow us to make Ordering contravariant. Would that be good enough to make Equal and other types contravariant or do you have other concerns ?