How to speed up the progress of adding patterns

erre-quadro / spikex

SpikeX - SpaCy Pipes for Knowledge Extraction

Apache License 2.0

398 stars 28 forks source link

How to speed up the progress of adding patterns #10

Open Hunter-Leo opened 3 years ago

Hunter-Leo commented 3 years ago

spikex version: 0.5.0
Python version:
Operating System: linux

Description

Hey, guys. I found your tool is very powerful, thx for sharing. I met a problem that the time cost is huge, when I was trying to add 30 thousands patterns to initialize LabelX. And this progress is much slower than the spacy, so that I wonder if any solution you guys can propose?

paoloq commented 2 years ago

Hi @Hunter-Leo!

Time cost indexing patterns depends on many factors. I'm thinking that a couple of things could help in identifying where's the issue:

You can investigate "spying" an indexing, maybe using py-spy. In this way, we know where is most of the time consumption.
You can share some of patterns you're using, just to have an idea of what kind of complexity they have.

If you have any other suggestion, of course it's welcome!