Closed StanislasDOZIAS closed 2 years ago
This would be a nice addition to the EDSNLP stack! @percevalw is currently working on a re-write of the PhraseMatcher, I think we'll wait until this change is ready before tackling fuzzy matching. But this is definitely on our radar, and the comparison helps a lot :)
Since https://github.com/aphp/edsnlp/pull/43 and https://github.com/aphp/edsnlp/pull/94 have been merged, I think we can close this one.
Future work on this topic will likely focus on speeding up simstring.
Feature type
Some performance comparisons between different matching algorithms (
SimString
,FlashText
andEDS.Matcher
)Description
SimString
Search with a given measure (for example Cosine Similarity) if a string match another string. It is also the base of QuickUMLS. Quite annoying to use (you have to split the text in order to have a list of query word/expressions).
FlashText
Build a Tree with all the words in the dictionary, and go through it with the query tokens. Built to find exact match. Possibilitie to allow some misspeling through the Levensthein's Distance, already implemented in the GitHub but not in the
pip
version, so the code is inedsnlp\u tils\flashtext.py
.Global Comparison with SimString, FlashText (with and without a spaCy pipeline) and the EDS.matcher pipeline :
FTspaCy
is theFlashText
algorithm included in a spaCy pipeline to include all the computation added by it.Making the number of keywords evolves (we search them in 10k words)
Making the number of total words evolves (we search 2587 keywords)
FUZZY COMPETITION