Closed dilipsundar closed 5 years ago
Functional duplicate of https://github.com/jpmml/sklearn2pmml/issues/91
The conversion of user-defined text preprocessors is not supported, and it's unlikely that it will be ever supported for the following reasons:
main.TextPreprocessor
) and translate it to PMML so that it's functionality is exactly preserved.What you should try/do:
Thanks for the suggestion @vruusmann So if I get rid of functions word_tag and lemmatize_sentence, I'll be able to include clean_sentence as part of the PipeLine?
So if I get rid of functions word_tag and lemmatize_sentence, I'll be able to include clean_sentence as part of the PipeLine?
Yes, it should be possible to translate the business logic of the clean_text
method in PMML. The only problematic bit is the statement text = contractions.fix(text)
, which is probably calling some 3rd party library again.
Here's a Python to PMML mapping:
re.sub
-> sklearn2pmml.preprocessing.ReplaceTransformer
: https://github.com/jpmml/sklearn2pmml/blob/master/sklearn2pmml/preprocessing/__init__.py#L241-L255string.join
-> sklearn2pmml.preprocessing.ConcatTransformer
: https://github.com/jpmml/sklearn2pmml/blob/master/sklearn2pmml/preprocessing/__init__.py#L211-L224str.lower
, str.upper
and str.trim
-> sklearn2pmml.preprocessing.StringNormalizer
: https://github.com/jpmml/sklearn2pmml/blob/master/sklearn2pmml/preprocessing/__init__.py#L276-L300I've modified the Preprocessor class as per your suggestion and it looks like this.
class TextPreprocessor(BaseEstimator, TransformerMixin):
def clean_text(self, text):
text = text.lower()
text = text.strip()
text = re.sub(' +', ' ',text)
text = re.sub('[^A-Za-z]+', ' ', text)
return text
def fit(self, X, y=None):
return self
def transform(self, X, y=None):
if type(X)==str:
X = [X]
return [self.clean_text(text) for text in X]
logit_pipeline = PMMLPipeline([('preproc', TextPreprocessor()),
('vect', CountVectorizer(ngram_range=(1,2))),
('tfidf', TfidfTransformer(use_idf=True)),
('clf', LogisticRegression(C=11.3))])
logit_pipeline.active_fields = np.asarray(["Input Text"])
logit_pipeline.target_fields = np.asarray(["Output"])
logit_pipeline.fit(X, y)
sklearn2pmml(logit_pipeline, 'logit.pmml', user_classpath = ["/path/sklearn2pmml-plugin/target/sklearn2pmml-plugin-1.0-SNAPSHOT.jar"])
The error I'm getting:
Standard output is empty
Standard error:
May 08, 2019 5:31:50 AM org.jpmml.sklearn.Main run
INFO: Parsing PKL..
May 08, 2019 5:31:54 AM org.jpmml.sklearn.Main run
INFO: Parsed PKL in 3791 ms.
May 08, 2019 5:31:54 AM org.jpmml.sklearn.Main run
INFO: Converting..
May 08, 2019 5:31:56 AM org.jpmml.sklearn.Main run
SEVERE: Failed to convert
java.lang.IllegalArgumentException: Tuple contains an unsupported value (Python class __main__.TextPreprocessor)
at org.jpmml.sklearn.CastFunction.apply(CastFunction.java:43)
at com.google.common.collect.Lists$TransformingRandomAccessList.get(Lists.java:599)
at sklearn.TransformerUtil.getHead(TransformerUtil.java:35)
at sklearn2pmml.pipeline.PMMLPipeline.encodePMML(PMMLPipeline.java:187)
at org.jpmml.sklearn.Main.run(Main.java:145)
at org.jpmml.sklearn.Main.main(Main.java:94)
Caused by: java.lang.ClassCastException: Cannot cast net.razorvine.pickle.objects.ClassDict to sklearn.Transformer
at java.lang.Class.cast(Class.java:3369)
at org.jpmml.sklearn.CastFunction.apply(CastFunction.java:41)
... 5 more
Exception in thread "main" java.lang.IllegalArgumentException: Tuple contains an unsupported value (Python class __main__.TextPreprocessor)
at org.jpmml.sklearn.CastFunction.apply(CastFunction.java:43)
at com.google.common.collect.Lists$TransformingRandomAccessList.get(Lists.java:599)
at sklearn.TransformerUtil.getHead(TransformerUtil.java:35)
at sklearn2pmml.pipeline.PMMLPipeline.encodePMML(PMMLPipeline.java:187)
at org.jpmml.sklearn.Main.run(Main.java:145)
at org.jpmml.sklearn.Main.main(Main.java:94)
Caused by: java.lang.ClassCastException: Cannot cast net.razorvine.pickle.objects.ClassDict to sklearn.Transformer
at java.lang.Class.cast(Class.java:3369)
at org.jpmml.sklearn.CastFunction.apply(CastFunction.java:41)
... 5 more
Traceback (most recent call last):
File "test-pmml.py", line 126, in <module>
sklearn2pmml(logit_pipeline, 'logit.pmml', user_classpath = ["/path/sklearn2pmml-plugin/target/sklearn2pmml-plugin-1.0-SNAPSHOT.jar"])
File "/home/devuser/.local/lib/python2.7/site-packages/sklearn2pmml/__init__.py", line 252, in sklearn2pmml
raise RuntimeError("The JPMML-SkLearn conversion application has failed. The Java executable should have printed more information about the failure into its standard output and/or standard error streams")
RuntimeError: The JPMML-SkLearn conversion application has failed. The Java executable should have printed more information about the failure into its standard output and/or standard error streams
Is there something else I've not removed as part of the clean up?
The idea is to expand the business logic of your main.TextPreprocessor.clean_text
method into a list of SkLearn2PMML package's built-in transformer classes.
Something like this:
from sklearn2pmml.preprocessing import ReplaceTransformer, StringNormalizer
mapper = DataFrameMapper([
(["Input Text"], [StringNormalizer("lowercase", trim_blanks = True), ReplaceTransformer(' +', ' '), ReplaceTransformer('[^A-Za-z]+', ' ')])
])
pipeline = PMMLPipeline([
("mapper", mapper),
('vect', CountVectorizer(ngram_range=(1,2))),
('tfidf', TfidfTransformer(use_idf=True)),
('clf', LogisticRegression(C=11.3)
])
pipeline.fit(X, y)
DataFrameMapper doesn't work on objects that aren't dataframes right?
I'm expecting a string input, which I'm cleaning and converting into an array (the output of the TextPreprocessor's transform function) before passing it to the CountVectorizer and others.
DataFrameMapper doesn't work on objects that aren't dataframes right?
It is designed to work with pandas.DataFrame
, because it selects columns based on name (not possible with Numpy arrays).
I'm expecting a string input, which I'm cleaning and converting into an array
You have a one-dimensional input space, right? Just "convert" it temporarily from Numpy array to pandas.DataFrame
. You can name this singleton column whatever you like, it doesn't have to be called "Input Text".
Alternatively, you can replace DataFrameMapper
with a plain sklearn.pipeline.Pipeline
:
pipeline = PMMLPipeline([
("text_proc", Pipeline([
("lower_trim", StringNormalizer(..)),
("rep1", ReplaceTransformer(..)),
("rep2", ReplaceTransformer(..))
])),
('vect', CountVectorizer(ngram_range=(1,2))),
('tfidf', TfidfTransformer(use_idf=True)),
('clf', LogisticRegression(C=11.3)
])
You have a one-dimensional input space, right? Just "convert" it temporarily from Numpy array to
pandas.DataFrame
.
I'm expecting a text input i.e. str object. Is that possible to do this conversion within the pipeline itself?
I am sorry if I meddling in this old discussion, but when I try your example on https://github.com/jpmml/sklearn2pmml/issues/150#issuecomment-490780505 it seems that ReplaceTransformer
returns numpy.ndarray
which doesn't have attribute lower (that called in CountVectorizer
). Am I missing some steps?
@nwxxb
Did you try running the referenced example? What was the Python error? Or were you simply hypothesizing?
it seems that ReplaceTransformer returns
numpy.ndarray
which doesn't have attribute lower
All SkLearn transformers return array-like data containers. What matters is the element type of a data container.
In the current case, the ReplaceTransformer
should return an "array of strings" (the data container type is numpy.ndarray
, its element type is str
).
The next pipeline step will be calling the lower()
function on array elements (which are strings), not the array itself.
Thank you for your response and yes I did, here is my code when I running it:
# df_try.shape == (8, 2), with two columns:
# `text_content (str)` and `label_id (float)`
text_try = pd.Series(data=d, name="text_content")
label_try = pd.Series(data=labels, name="label_id")
df_try = pd.concat([text_try, label_try], axis=1)
pipeline = PMMLPipeline([
("preprocessing", Pipeline([
("remove_number", ReplaceTransformer('\d+', '')),
])),
('vect', CountVectorizer(ngram_range=(1,2))),
])
pipeline.fit_transform(df_try['text_content'])
yield error:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
[<ipython-input-20-54aadab62a28>](https://localhost:8080/#) in <cell line: 13>()
11 ])
12
---> 13 pipeline.fit_transform(df_try['text_content'])
4 frames
[/usr/local/lib/python3.10/dist-packages/sklearn/pipeline.py](https://localhost:8080/#) in fit_transform(self, X, y, **fit_params)
443 fit_params_last_step = fit_params_steps[self.steps[-1][0]]
444 if hasattr(last_step, "fit_transform"):
--> 445 return last_step.fit_transform(Xt, y, **fit_params_last_step)
446 else:
447 return last_step.fit(Xt, y, **fit_params_last_step).transform(Xt)
[/usr/local/lib/python3.10/dist-packages/sklearn/feature_extraction/text.py](https://localhost:8080/#) in fit_transform(self, raw_documents, y)
1386 break
1387
-> 1388 vocabulary, X = self._count_vocab(raw_documents, self.fixed_vocabulary_)
1389
1390 if self.binary:
[/usr/local/lib/python3.10/dist-packages/sklearn/feature_extraction/text.py](https://localhost:8080/#) in _count_vocab(self, raw_documents, fixed_vocab)
1273 for doc in raw_documents:
1274 feature_counter = {}
-> 1275 for feature in analyze(doc):
1276 try:
1277 feature_idx = vocabulary[feature]
[/usr/local/lib/python3.10/dist-packages/sklearn/feature_extraction/text.py](https://localhost:8080/#) in _analyze(doc, analyzer, tokenizer, ngrams, preprocessor, decoder, stop_words)
109 else:
110 if preprocessor is not None:
--> 111 doc = preprocessor(doc)
112 if tokenizer is not None:
113 doc = tokenizer(doc)
[/usr/local/lib/python3.10/dist-packages/sklearn/feature_extraction/text.py](https://localhost:8080/#) in _preprocess(doc, accent_function, lower)
67 """
68 if lower:
---> 69 doc = doc.lower()
70 if accent_function is not None:
71 doc = accent_function(doc)
AttributeError: 'numpy.ndarray' object has no attribute 'lower'
it seems that ReplaceTransformer actually wrap the input string on numpy.ndarray
which CustomVectorizer
calls lower
method on it.
@nwxxb Thanks for bringing this issue to my attention. Not exactly a bug, but needs fixing nonetheless.
I started playing with your example code, and discovered it's an "array shape mismatch". Specifically, the ReplaceTransformer
is returning a 2-D array (n_rows, 1)
. However, the CountVectorizer
only supports 1-D arrays (aka column vectors), which have a shape (n_rows, )
(note the missing second dimension!).
The issue can be fixed by reshaping the array between ReplaceTransformer
and CountVectorizer
steps from 2-D to 1-D (aka column vector) using the sklearn2pmml.util.Reshaper
meta-transformer:
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.pipeline import Pipeline
from sklearn2pmml.preprocessing import ReplaceTransformer
from sklearn2pmml.util import Reshaper
pipeline = Pipeline([
("replace", ReplaceTransformer("\d+", "")),
# THIS! Reshape from 2-D array of strings to 1-D array of strings
("reshaper", Reshaper((-1, ))),
("count_vectorize", CountVectorizer(ngram_range = (1, 2)))
])
Looks like Numpy arrays support string functions (lower
, upper
, trim
etc.) only on 1-D (row or column vectors).
Thank you for your response it solved my problem :smile:
@nwxxb If you're still doing [ReplaceTransformer(), CountVectorizer()]
workflows, then starting from SkLearn2PMML version 0.103.2 it's possible to omit the Reshaper()
helper step.
See https://github.com/jpmml/sklearn2pmml/commit/e6bd044b7c3ddaf9cdac3281971cfe70c1710407
Hi @vruusmann
I'm trying to clean and vectorize my text before feeding it to my model and this is what I've written to do that.
When I run the last line, I'm getting this as an exception.
Can you please help me understand what I'm doing wrong? Or does sklearn2pmml-plugin not support some of the things I'm trying to do here?