Implement word_stem in prestosql

facebookincubator / velox

A C++ vectorized database acceleration library aimed to optimizing query engines and data processing systems.

https://velox-lib.io/

Apache License 2.0

3.45k stars 1.13k forks source link

Open arundpanicker opened 1 year ago

arundpanicker commented 1 year ago

Add missing function to handle word_stem prestosql function

arundpanicker commented 1 year ago

The Folly library does not have an implementation for word stemming.
The word_stem java implementation uses snowball (libstemmer) library (https://github.com/prestodb/presto/blob/5534a43f9fe1b843a5b011c446a659f559565d32/p[…]/java/com/facebook/presto/operator/scalar/WordStemFunction.java).
Shall we use the libstemmer c library (http://snowball.tartarus.org/download.html) for velox? This will need to add script in the setup files (all platforms) to download the library.

aditi-pandit commented 1 year ago

@mbasmanova, @Yuhta : Any thoughts ?

mbasmanova commented 1 year ago

CC: @kgpai @majetideepak

mbasmanova commented 1 year ago

FYI, we have some code at Meta that uses snowball (libstemmer) library.

// This will stem words according to the Snowball Stemmer algorithm
// http://snowballstem.org/
class Stemmer {