vruusmann
commented
7 years ago I do have a custom estimator that merges rare categorical values into one value as 'RARE' so that I can group all the rare labels as together.
Basically, you want to perform mapping between discrete values - map "popular" values back to themselves, and all "unpopular" values to some default value.
The PMML specification provides the MapValues element exactly for this purpose:
<MapValues name="simplified_color" defaultValue="rare" outputColumn="outputValue">
<FieldColumnPair field="color" column="inputValue"/>
<InlineTable>
<row>
<inputValue>red</inputValue>
<outputValue>red</outputValue>
</row>
<row>
<inputValue>yellow</inputValue>
<outputValue>yellow</outputValue>
</row>
<row>
<inputValue>green</inputValue>
<outputValue>green</outputValue>
</row>
</InlineTable>
</MapValues>The above transformation would keep color values "red", "yellow" and "green" as-is, and change all other color values to "rare" (note the MapValues@defaultValue attribute).
I would like to know if it is possible and how can I add my custom modelconverter as you did for spark standard ml-features.
Here's a code example about generating a MapValues element-based transformation:
https://github.com/jpmml/jpmml-sparkml/blob/master/src/main/java/org/jpmml/sparkml/feature/VectorIndexerModelConverter.java
The org.dmg.pmml.InlineTable element is rather tricky to generate in Java, because you need to be working with low-level W3C DOM APIs in some point.
Even if I manage it, I am not sure if jpmml-evaluater will be able to make it run.
JPMML-Evaluator is able to run all PMML documents that conform to PMML 3.X and 4.X specifications.
fatihtekin
Hi
I do have a custom estimator that merges rare categorical values into one value as 'RARE' so that I can group all the rare labels as together. I would like to know if it is possible and how can I add my custom modelconverter as you did for spark standard ml-features.
Ti give an example my custom estimator handles rare columns for categorical columns. So, if there are 1000 categories and only 30 of them are used in most of the time the rest 970 columns will be marked as RARE. So in my model I only save the rare labels. If you need I can paste the code itself as well.
Even if I manage it, I am not sure if jpmml-evaluater will be able to make it run.