Mimir has a much cruder internal type system than Spark. In addition to lacking collection types, there's a lot of capabilities (e.g., integer precision) that Mimir just does not support at the moment. Spark already has a well-developed type system, and it would be fantastic if we could replace this part of Mimir with the corresponding parts from Spark.
[ ] Replace Mimir's Type with Spark's DataType
[ ] Update or replace Typechecker with corresponding parts of Spark (not sure where this lives... perhaps the Catalyst analyzer
[ ] Tack on support for unit-based types (Maybe use User-Defined Types?)
[ ] Move to serialization using foo.json and foo.as[DataType](make sure to keep track of everywhere this happens)
[ ] Add an upgrade path to migrate old, encoded types to the new form
Possibly required for #361 Would subsume #333
Mimir has a much cruder internal type system than Spark. In addition to lacking collection types, there's a lot of capabilities (e.g., integer precision) that Mimir just does not support at the moment. Spark already has a well-developed type system, and it would be fantastic if we could replace this part of Mimir with the corresponding parts from Spark.
Typewith Spark'sDataTypeTypecheckerwith corresponding parts of Spark (not sure where this lives... perhaps the Catalyst analyzerfoo.jsonandfoo.as[DataType](make sure to keep track of everywhere this happens)