Generic instances for user-defined Mirror instances

[tschuchortdev]

Shapeless 3 currently does not work with user-defined instances of scala.deriving.Mirror. Usually, these instances are generated automatically by the compiler for ADTs, but it is actually possible to define your own instances. The need for such instances may arise for example in the context of structural refinement types and transparent def macros (To make a long story short: It is impossible with Scala 3 macros to define new types that escape the macro scope. A trick to circumvent this is to define a Selectable or Dynamic "stub" type and then add refinement type information through a transparent def macro and if you want that type to work with type-class derivation, you need to define your own Mirror instances). The reason that these custom instances don't work with Shapeless 3 is that Shapeless 3 does not use Mirror directly but instead K0.Generic and company, which are just refinements on Mirror to fix the kind:

type Generic[O] = Mirror {
  type Kind = K0.type
  type MirroredType = O
  type MirroredMonoType = O
  type MirroredElemTypes <: Tuple
}

As far as I can tell, these Generic/Mirror instances are not derived in code anywhere. It seems to me that they are generated by the compiler on-demand where they are used. The crux of the issue here is the additional type Kind refinement that is not present on Mirror: From my experiments, it appears that the compiler is happy to generate any arbitrary Mirror { type FooBar = Noxarpe } for you, no matter what FooBar is, but the type refinement will prevent regular Mirror instances coming from another place from being considered in the implicit search. If you summon[Mirror.ProductOf[T]] manually and try to plug it into a parameter of K0.Generic[T] it will not work. Letting the compiler generate the Mirror { type Kind = K0.type } in-place where a K0.Generic[T] is asked, will.

case class Beta(val a: Int, b: String)
K0.mkProductInstances[Show, Beta]   // works
K0.mkProductInstances[Show, Beta](using summon[K0.Generic[Beta]]) // works
K0.mkProductInstances[Show, Beta](using summon[Mirror.ProductOf[Beta] { type Kind = K0.type }]) // works
K0.mkProductInstances[Show, Beta](using summon[Mirror.ProductOf[Beta] {
  type Kind = K0.type
  type FooBar = Int
}]) // works
K0.mkProductInstances[Show, Beta](using summon[Mirror.ProductOf[Beta]]) // error

I propose to add a bunch of "implicit conversions" to the library to convert Mirror instances to Generic where necessary. Example:

inline given [T, MET <: Tuple, MEL <: Tuple](using inline m: Mirror.ProductOf[T] {
  type MirroredElemTypes = MET
  type MirroredElemLabels = MEL
}): (K0.ProductGeneric[T] {
  type Kind = K0.type
  type MirroredElemLabels = MEL
  type MirroredElemTypes = MET
  type MirroredType = T
  type MirroredMonoType = T
}) = m.asInstanceOf

Adding this given to the Beta example will make it compile correctly. My experiments suggest that the function and m parameter have to be inline and this Aux-pattern have to be used to make the compiler remember what MirroredElemTypes is concretely, or else uses of Tuple.Size and LiftP in Shapeless 3 will not work. Unfortunately, this means we can not write & m.type thus the types have to be spelled out for every Generic, CoproductGeneric, ProductGeneric for every K to make it work. Liberal use of inline givens in Shapeless 3 also prevents use of transparent inline here.

Perhaps there is an easier way, but this is what I came up with. It is, of course, also possible to define your own Generic instances from the get-go whereever you are defining Mirror, but I'm not happy with that: It couples every library that wants to define their own types with custom Mirror instances to Shapeless, even when they don't depend on Shapeless themselves. It is also sometimes difficult to make sense of the error messages, especially when you don't know that you have to do it in the first place!

[joroKr21]

How about we just change the signatures?

  inline given mkInstances[F[_], T](using gen: Mirror.Of[T]): Instances[F, T] =
    inline gen match
      case p: ProductGeneric[T] => mkProductInstances[F, T](using p)
      case c: CoproductGeneric[T] => mkCoproductInstances[F, T](using c)

  inline given mkProductInstances[F[_], T](using gen: Mirror.ProductOf[T]): ProductInstances[F, T] =
    ErasedProductInstances[K0.type, F[T], LiftP[F, gen.MirroredElemTypes]](gen)

  inline given mkCoproductInstances[F[_], T](using gen: Mirror.SumOf[T]): CoproductInstances[F, T] =
    ErasedCoproductInstances[K0.type, F[T], LiftP[F, gen.MirroredElemTypes]](gen): CoproductInstances[F, T]

There is no Mirror.Of for higher kinds though, so I guess we have to add our own type aliases.

[tschuchortdev]

If that is possible then it would probably be the simplest solution. However, I think there's some value in having Kn.Generic for exactly the reason that there are no higher-kinded versions of Mirror. I suppose removing the type Kind refinement from Generic would also work and keep everything consistent.

[joroKr21]

I suppose removing the type Kind refinement from Generic would also work and keep everything consistent.

The refinement is necessary to add an implicit scope. So we can add extension methods to Generic and co.

[tschuchortdev]

If the { type Kind = ... } refinement can be eliminated completely from all mkInstances declaration, that should be sufficient to make the implicit search work correctly. If not, then I think it would be good to have the aforementioned "implicit conversions". That way, anyone who is creating Mirror instances (perhaps in a completely different third-party library) need not know about Shapeless 3 and everything will work together automatically. Without implicit conversions, custom Mirror instances will always have to extend Generic (whether by name or by structure), thereby sort of depending on implementation details of Shapeless.