search

invenia / FeatureTransforms.jl

Transformations for performing feature engineering in machine learning applications
MIT License
37 stars 1 forks source link

Support for text representation transforms? #102

Open pietrolesci opened 2 years ago

pietrolesci commented 2 years ago

Hi there,

First of all, thanks for creating this amazing library - investing time and money to support it, and making it open source. I discovered it at JuliaCon 2021 and have been immediately fascinated by it.

Question: Is there interest in supporting text representation (e.g., using ScikitLearn terminology: CountVectorizer, TfidfVectorizer, NgramVectorizer, etc) features in FeatureTransforms.jl?

Some of them are covered in TextAnalysis but are quite different, from the user perspective, from what anyone would expect looking at ScikitLearn. In particular, they are not "pipe-able" nor do they immediately return what is interesting for the user (i.e., do not implement the common "fit_transform" paradigm). In other words, it's quite non-trivial getting from text to input representations that can be fed to a machine learning model (e.g., MLJ, Flux, etc).

FeatureTransforms.jl seems a good place to support these transforms and, if interesting, I'd be happy to work on adding support for them.

glennmoy commented 2 years ago

Hi, thanks for getting in touch, glad you like the package :) Sorry it hasn't been my focus since JuliaCon so I haven't checked up on it since.

Question: Is there interest in supporting text representation (e.g., using ScikitLearn terminology: CountVectorizer, TfidfVectorizer, NgramVectorizer, etc) features in FeatureTransforms.jl?

TBH not particularly since we don't have any use cases for "textual features" so it wasn't in the original roadmap. But that being said I'd happily support adding this functionality provided it didn't introduce any heavy / non-standard dependencies. (I'd rather not have users who don't work with NLP to have to load in dependencies they don't need).

However if this was unavoidable, another option is to create a package that extends this API but adds in the functionality you need. For instance, we have a private package for precisely this purpose.

pietrolesci commented 2 years ago

Hi @glennmoy,

Thanks for your answer. I went through the linked issue and I really see the point of having a lightweight alternative to MLJ.

Given the use-cases you are interested in at Invenia, it seems there is no interest in supporting NLP-related transformers. On the other hand, having an independent package maintained by one person (me, in this case) seems to add noise to the already crowded feature engineering packages sub-community within the larger Julia ecosystem.

For the time being, I will refrain to contribute NLP transformers. Happy to connect in the future if there will be interest from Invenia's side.

Thank you very much for your attention. I wish you a very good day.

Best, Pietro

glennmoy commented 2 years ago

No problem, thanks for getting in touch!