mojodna
commented
5 years ago For dynamic generation of Encoders within generic methods, I have:
private def buildEncoder[T](implicit tag: TypeTag[T]): Encoder[T] = {
EncoderCache
.getOrElseUpdate(
tag, {
val pkg = "osmesa.common.impl"
val traits = traitsIn[T]
val name = s"$pkg.${traits.map(_.name.toString).mkString("With")}"
// https://stackoverflow.com/a/23792152/507685
val c = try {
Class.forName(name) // obtain java.lang.Class object from a string
} catch {
case e: ClassNotFoundException =>
throw new RuntimeException(
s"${e.getMessage} must be an implementation of the following traits: ${traits
.map(_.name.toString)
.mkString(", ")}")
case e => throw e
}
val mirror = runtimeMirror(c.getClassLoader) // obtain runtime mirror
val sym = mirror.staticClass(name) // obtain class symbol for `c`
val tpe = sym.selfType // obtain type object for `c`
// create a type tag which contains the above type object
val targetType = TypeTag(
mirror,
new api.TypeCreator {
def apply[U <: api.Universe with Singleton](m: api.Mirror[U]): U#Type =
if (m == mirror) tpe.asInstanceOf[U#Type]
else
throw new IllegalArgumentException(
s"Type tag defined in $mirror cannot be migrated to other mirrors.")
}
).asInstanceOf[TypeTag[Product]]
Encoders.product(targetType)
}
)
.asInstanceOf[Encoder[T]]
}
// determine closest traits
def traitsIn[T](implicit tag: TypeTag[T]): Set[TypeSymbol] = {
val tpe = tag.tpe
val t = tpe.baseClasses.filter(s => s.isAbstract && s != typeOf[Any].typeSymbol).map(_.asType)
t.foldLeft(Seq.empty[TypeSymbol]) {
case (acc, x) => {
// if x is a super type of anything in acc, skip it
if (acc.exists(y => y.toType <:< x.toType)) {
acc
} else {
// filter out anything in acc that's a super type of x
acc.filterNot(y => x.toType <:< y.toType) :+ x
}
}
}
.sortBy(_.toString)
.toSet
}This assumes that osmesa.common.impl.<case class> exists (since these can't be created at runtime) and is named alphabetically.
This then facilitates:
implicit class HistoricalNodeWithTimestampDatasetExtension[T <: Node with Timestamp](
history: Dataset[T] with History) {
import history.sparkSession.implicits._
def withValidity[U >: Node with Validity](tag: TypeTag[U]): Dataset[U] with History = {
implicit val encoder: Encoder[U] = buildEncoder[U]
history.withValidityInternal
.as[U]
.asInstanceOf[Dataset[U] with History]
}
}The case class NodeWithValidity will be used under the hood by Spark.
The goal is to define implicit classes with the core required fields as type parameters and allow them to be used by additionally refined types, e.g., Node with GeometryChanged with Timestamp, with similarly-enhanced return types.
I.e.
Node with Timestamp → Node with Validity
Node with GeometryChanged with Timestamp → Node with GeometryChanged with Validity
I think changing the class signature to this is part of the equation:
implicit class HistoricalNodeWithTimestampDatasetExtension[T <: Timestamp](
history: Dataset[Node with T] with History)I'm running into a type-related problem: I want the return type of a method to be a subset of the class's type parameters. I.e. Node with __ with Timestamp → Node with __ with Validity or Node with __ with Validity → Point with __ with Validity.
Is this even possible without needing to write boilerplate for each combination?
This begins the process of providing a more user-friendly API for processing OSM data. A variety of discrete traits are provided to provide clear indicators of what data is available within a given
Datasetand allows types to provide indications on what pre-defined actions are available.The driving motivations are:
Datasetincludes historical data (with History) or is a snapshot (Snapshot[Dataset[T]]), possibly with theTimestamp(or replication sequence) it applies to (Snapshot[Dataset[T]] with Timestamp).VersionControl(changeset),Authorship(uid/user),Identity (id/version) that can be composed and used to provide compile-time constraints on which methods can be applied to certain types ofDatasets (i.e..asPointsshouldn't apply toDataset[Way]).Dataset's type parameter(s).E.g.:
I could use some feedback on how to break this up, particularly the traits, the implementations (for serialization), and implicits (extensions). There are some name collisions that require imports to be more explicit than they should be otherwise, and moving traits, implementations, etc. to their own packages / classes / objects will likely help.
Naming is hard. Ideas and suggestions for conventions are appreciated.