Add Language primitive - Githubissues

alteryx / nlp_primitives

Natural Language Processing primitives for Featuretools

https://blog.featurelabs.com/natural-language-processing-featuretools/

BSD 3-Clause "New" or "Revised" License

37 stars 11 forks source link

Add Language primitive #142

Open gsheni opened 2 years ago

gsheni commented 2 years ago

We can use this library to detect the language of a text
- https://github.com/facebookresearch/fastText/
- https://fasttext.cc/docs/en/language-identification.html
```
import fasttext
filepath = os.path.join(FILE_PATH, "../data/lid.176.ftz")
model = fasttext.load_model(filepath)
predictions = self.model.predict(text="hej", k=2) # returns top 2 matching languages
predictions
```
  The object returned by the model is of the form ((‘labelpl’, ‘labelsv’), array([0.40688798, 0.23321952])) of <class 'tuple'> where pl and sv are the ISO 639 code for Polish and Swedish. The prediction for both languages is correct as Hej means Hello in both languages. The second part indicates the respective confidence of the sentence belonging to those languages.

sbadithe commented 2 years ago

Can I pick this up?

gsheni commented 2 years ago

Yes, we can tackle this after the other, currently assigned NLP primitives