Currently, the function py_entitymatching.feature.extractfeatures.extract_feature_vecs() relies on a single implementation of feature extraction that splits the candidate set into chunks and extracts features from each chunk in parallel. This works well in many situations, though feature extraction can take hours with 4-8 cores on a candidate set with tens of millions of rows and tens of features.
To give users some flexibility and control over how features are extracted, this PR does the following:
Defines an interface for feature extraction
Factors out the existing implementation into a class that implements the interface
Provides an alternate implementation that computes features over distinct input values
Currently, the function
py_entitymatching.feature.extractfeatures.extract_feature_vecs()
relies on a single implementation of feature extraction that splits the candidate set into chunks and extracts features from each chunk in parallel. This works well in many situations, though feature extraction can take hours with 4-8 cores on a candidate set with tens of millions of rows and tens of features.To give users some flexibility and control over how features are extracted, this PR does the following: