louismullie / treat

Natural language processing framework for Ruby.
Other
1.36k stars 128 forks source link

Treat being loaded causes slowness in stanford-core-nlp #56

Open neil-ca-moore opened 11 years ago

neil-ca-moore commented 11 years ago

When treat is loaded in my Gemfile, stanford-core-nlp runs substantially slower.

Here's the code that runs quite slow. I don't think it's anything very complex.

text.get(:sentences).each do |sentence|
  sentence.get(:tokens).each do |token|
    value =  token.get(:original_text).to_s
    pos = token.get(:part_of_speech).to_s
    et = token.get(:named_entity_tag).to_s
    ner_list << [et,value]
    insert_unique(pos_hash, pos, value)
  end
end

I've enclosed a couple of profile trace images that shows the same code before and after loading treat.

Hope this helps and apologies if it's down to my ignorance.

Regards,

Neil

With treat: before

Without treat: after