Open jbesomi opened 4 years ago
Dask vs. spaCy
It's faster to use pipe
from spaCy or to directly use Dask
(Dask Dataframe)?
Dask
might be the solution we were looking for ...
As described in #162 , dask is not feasible from a UX perspective. Here are our results from experimenting with tokenize. See THE ATTACHED PDF for a notebook of the results.
We now compare:
Results below.
We can see that
Thus, our options:
use_spacy
that works like our tokenize_with_spacy_own_parallelization
above, and explain to them that this might give them better results but takes about 3x as long.We don't really have a preference.
The actual tokenizer is very fast as it uses a simple regex pattern but at the same time very imprecise.
A better alternative might be to make use of
spaCy
.Naively tokenize a Pandas Series with spaCy is very simple:
This should be somehow fast as in theory
spaCy
makes use of multi-threading.The reason we haven't implemented this yet is that we want to make sure this solution is enough faster. We want to provide a simple tool to analyze a quite large amount of text data; say 100k Pandas Row should take no longer than 15-30 seconds to
tokenize
... ?For now, the task consists in: