wikit-ai / olaf

6 stars 1 forks source link

Define au way to generalize the token filtering process #2

Closed ba-talibe closed 5 months ago

ba-talibe commented 5 months ago

One idea is to create filter components that tag a custom Boolean attribute of the token class (let's say Token._.filter). We could then use the functools.reduce function to pipe the "filters". We just need to define a signature for the filter functions. Say: