ryan-deak-zefr
commented
7 years ago Here's a question for the consumer. Is it easier to consume:
- a score whose value has a polymorphic type? OR
- a score with many key-value pairs of monomorphic type, where exactly zero or one key-value pair is populated? (zero for a "no prediction", one for a prediction). Note that a type hint could still allow for O(1) value extraction. It should be based on the
@switchannotation. The byte code should be checked for atableswitchinstruction. See here for more details.
For instance:
Polymorphic Type
{
"scores": [
{ "value": 1 },
{ "value": "car" },
{ "value": true }
]
}Many Monomorphic Types
{
"scores": [
{ "intValue": 1, "valueType": "int" },
{ "stringValue": "car", "valueType": "string" },
{ "booleanValue": true, "valueType": "boolean" }
]
}
deaktator
Current Implementation
The current aloha_avro_score.avdl uses Avro's support for co-products based on the built-in
uniontype. As such, the description of a score looks like:Deficiencies
There are some deficiencies with this approach. Using the
avro-toolstool chain to compile this Java code results in aScore.valueof typeObjectsince Java has poor support for co-products.Most non-default tool chains don't support
uniontypes with 3 or more constituent types and only support 2-typeunions when one of the types isnull. The support for the latter is done to encodeOptiontypes in Scala.Additionally,
uniontypes in Avro can't support a union of differentarrays. Therefore,arraytypes appearing in aunionmust be mixed, even if that's not desired (as it's a more lax type).Proposal
This proposal is to make the Avro
Scoremessage more like the protobuf version that encodes co-products via multiple nullable variables, one per desired output type and a type indicator variable.While the type indicator variable isn't strictly necessary, it can speed lookups from O(T) (where T is the number of types) to O(1). Obviously, for this to be type-safe, we need type classes (one instance per possible type) to extract the
valuefrom theScore.This approach will work better with current tool chains, which will allow us to not have to special case Aloha scores.
Avro IDL for Aloha Score
So, the proposed Avro description of
Scorewould be something like:Aloha Score extensions
Scala Extraction Syntax
Like in the protobuf case for protobuf Scores, we should have type classes to extract the value. For simplicity, it would be nice to have
getandapplysyntax analogous to Scala's scala.collection.GenMapLike trait. This should be very easy to accomplish and would look like thisThis should involve importing the syntax, but the conversions should all be in the appropriate implicit scope so that converters don't need to be explicitly pulled into scope.
Additional Coproduct Support
Perhaps it's also useful to consider coproducts at the type level using an encoding with something like cats or iota.
Java Extraction Syntax
T.B.D.
Perhaps something a rich score wrapper with
getTmethods for each output typeT. These should probably throw when the incorrect type is provided. A determination of up-casting should be considered. For instance, what happens when the consumers asks for aDoublebut the type was aLongor aFloat. The current protobuf Score code does this but this should be readdressed at this time and the Avro and protobuf should be made consistent. Perhaps common interfaces could be extracted and we could have rich implementations that provide more seamless interoperability.Implementation Details
Map types are coming to Avro soon for multilabel learning and this presents a problem. Quadratically many types in the key and value spaces will be required to encode this type of score coproduct. Therefore, it might be a good time to think about generating the Avro schema programmatically. This might be possible with something like the sbt-boilerplate plugin. If not, we might want to think about writing such a plugin.