Open tixxit opened 8 years ago
@avibryant So, this has some of the stuff I've been doing (still very WIP), but the important bits are:
FValue
ADT, which is basically just Number | Text | Boolean
right nowFeatureParser[-A]
that is basically just A => Map[String, FValue]
(eg A
could be CsvRow
)
TrainingDataFeatureParser[-A]
which includes ID/timestamp/target parsersFeatureEncoder[K, +V]
that can convert some Map[String, FValue] => Map[K, V]
FeatureEncoder
s will be (de)serialized and stored alongside the modelFeatureEncoding[K, V, T]
that holds a FeatureEncoder[K, V]
and Splitter[V, T]
(we can get rid of the T
after your PR lands)
TrainingPlatform#Trainer[-A, +B]
which describes some series of passes over data of type A
to end up with a B
, while passing some state between each of the passes
type Trainer[A, B] = Free[({ type f[x] = Aggregator[A, _, x] })#f, B]
currently, though we tag along some platform-specific context with the aggregatorThere is an implementation, DispatchedFeatureEncoding
, of a FeatureEncoding
for the dispatched type, which has a "trainer" that does a pass over the data and attempts to infer the sub-type of Dispatched to use for it.
The main goal of the FeatureParser
vs FeatureEncoder
split is so that we can separate the input type from the input-type agnostic feature encoding bits from the tree K/V type. So, we can train off CSV data or thrift and still write a web service that accepts JSON.
This is super early work, but we have a CsvTrainerJob, which can run on ~arbitrary CSVs, with the labels provided by the user. The actual types of the values will be inferred before training.