Sklearn compatible transformer and parallelization

[dafajon]

When working on documents on a dataframe feature extraction requires a sklearn transformer so that the process can be a part of pipeline and serialized along with it. The issues so far:

• Serialization of feature transformer states (e.g. tf_type, idf_type, tokenizer)
• Parallelization of feature extraction.
• Placement of this feature (core or sadedegel extensions)
• Consume str and List[str] for inference time.

[husnusensoy]

I see two different questions in here please split into two separate issues for us to proceed.

[husnusensoy]

Check develop branch for drop-in replacement

from sadedegel.dataset import load_raw_corpus
from sadedegel.extension.sklearn import TfidfVectorizer

tra = TfidfVectorizer()

X = tra.transform(load_raw_corpus())

[dafajon]

I checked on List and pd.DataFrame. Works fine. I will also use it with a pipeline. Will provide feedback. I had a FunctionTransformer implementation in sentiment work. Will update that accordingly when this is in the new release.

[husnusensoy]

Can you create a seperate issue for parallel processing. This issue is already closed with a commit currently available on develop branch.