RFC

[nevillelyh]

Motivation

Right now we use a couple of shapeless based libraries in data pipelines.

• https://github.com/nevillelyh/shapeless-datatype for:
  • conversion between case classes <-> Avro GenericRecord, BigQuery TableRow, Datastore Entity, TensorFlow Example
  • RecordMapper, RecordMatcher, LenseMatcher for some generic data type mapping/matching ¹
• https://github.com/alexarchambault/scalacheck-shapeless for deriving ScalaCheck Gen[T] random generators for tuples and case classes
• CaseClassDiffy in ratatool to derive Diffy[T] for comparing "diff" between 2 primitive types, case class, or Avro/Protobuf/BigQuery records
• Algebird macros, not shapeless based but also for deriving Semigroup[T], Monoid[T] for tuples & case classes ²

The biggest problem of shapeless is slow compilation. Most of the type class derivation cases can be replaced with magnolia. We already have tfexample-derive for TF Example derivation. For 2, magnolia should be more flexible and maintainable than handcrafted macros.

Not sure if 1 can be covered with magnolia but they're also not widely used.

Scopes

• Minimal dependencies, Magnolia and a few necessary ones to reduce migration impact
• Support Scala 2.11-2.13, with our forked Magnolia for 2.11 support
• Everything serializable, so they can be used in pipelines
• Shared interface for similar components, e.g. Converter[T] for type conversion ³

3 may require a core sub-project. Another benefit of converters sharing a root trait is reusable tests, etc. but that's debatable.

Tasks

A breakdown of tasks ranked by priority and size.

• scalacheck Gen[T] derivation
• avro/bigquery/datastore/tensorflow/protobuf Converter[T] derivation, among them tensorflow already exists, protobuf might be doable but tricky
• cats Semigroup[T] etc. derivation, since Algebird is not available for Scala 2.13 yet
• diffy Diffy[T] derivation, with the goal of replacing that in ratatool

[nevillelyh]

Notes

• implicit fallback and Gen#{sized,resize} to deal with stack overflow in recursive ADTs.
  • scalacheck-shapeless code, commit
  • https://github.com/typelevel/scalacheck/issues/305
• implicit Cogen[A] Arbitrary[B] needed for deriving Arbitrary[A => B] Erik's talk
• Arbitrary[C[T]] for container C requires CanBuildFrom which is not serializable. ScalaCheck has it's own but seems not comprehensive enough.

Checklist for first release:

• [x] Rethink Converter[T, R] design
• [x] Support nested records in TF ExampleType
• [x] Schema for TableRowType
• [x] AvroType
• [x] Schema for AvroType

Nice to have for the future:

• [ ] Diffy
• [ ] ProtobufType

[nevillelyh]

Basic requirements are done, closing.