Kimtaro / ve

A linguistic framework that's easy to use.
MIT License
215 stars 25 forks source link

example doesn't work on Debian with installed freeling v3.1 #25

Open filterfish opened 9 years ago

filterfish commented 9 years ago

I tried the example from the README but it returns an empty array. FreeLing is installed from the dpkg downloaded from the FreeLing site. MeCab isn't installed.

dpkg -l | grep freeling
ii  freeling     3.1    amd64        Library of natural language analysis facilities.

I piped the following through pry:

require 've'
provider = Ve::Manager.provider_for(:en, :words)
parse = provider.parse('I like melons.')
parse.send(:words)

I get:

[1] pry(main)> require 've'
/home/rgh/.gem/ruby/2.2.0/gems/ve-0.0.3/lib/providers/japanese_transliterators.rb:100: warning: duplicated key at line 101 ignored: "gwi"
/home/rgh/.gem/ruby/2.2.0/gems/ve-0.0.3/lib/providers/japanese_transliterators.rb:100: warning: duplicated key at line 101 ignored: "gwu"
/home/rgh/.gem/ruby/2.2.0/gems/ve-0.0.3/lib/providers/japanese_transliterators.rb:100: warning: duplicated key at line 101 ignored: "gwe"
/home/rgh/.gem/ruby/2.2.0/gems/ve-0.0.3/lib/providers/japanese_transliterators.rb:100: warning: duplicated key at line 101 ignored: "gwo"
=> true
[2] pry(main)> provider = Ve::Manager.provider_for(:en, :words)
=> #<Ve::Provider::FreelingEn:0x007fdb841021d0
 @config={:app=>"/usr/bin/analyzer", :path=>"", :flags=>"-f /usr/share/FreeLing/config/en.cfg --flush --nonumb --nodate"},
 @stderr=#<IO:fd 12>,
 @stdin=#<IO:fd 9>,
 @stdout=#<IO:fd 10>>
[3] pry(main)> parse = provider.parse('I like melons.')
=> #<Ve::Parse::FreelingEn:0x007fdb84239080 @text="I like melons.", @tokens=[]>
[4] pry(main)> parse.send(:words)
=> []
[5] pry(main)> 

I initially thought it was -f /usr/share/FreeLing/config/en.cfg flag as this path doesn't exist. The path is /usr/share/freeling/config/en.cfg. So I'm symlinked /usr/share/FreeLing to /usr/share/freeling but it made no difference.

If I run it on the command line it works analyze seems to return something sensible (I've never used FreeLing before):

echo "I like melons." | analyze --locale=en_AU.UTF-8 -f /usr/share/FreeLing/config/en.cfg --flush --nonumb --nodate
I i PRP 1
like like IN 0.867653
melons melon NNS 1
. . Fp 1

I wondered if it might be the locale option (I don't have en_US.UTF-8 installed) and analyze fails without it but after adding locale it still returned an empty array.

Kimtaro commented 9 years ago

Hmm, it might be that something has changed in FreeLing 3.1. I think it was on version 2.9 when I wrote this code. I'll see if I can reproduce locally with 3.1 and fix.

Kimtaro commented 9 years ago

I have tried with Freeling 3.1 and it's working for me.

It's odd that your Ve is trying to use /usr/share/FreeLing when it should use /usr/share/freeling though.

Try forcing the provider class to use the correct path by running this right after require 've': Ve::Manager.set_default_config_for(Ve::Provider::FreelingEn, {:flags => "-f /usr/local/share/freeling/config/en.cfg --flush --nonumb --nodate"})

Then try

provider = Ve::Manager.provider_for(:en, :words)
parse = provider.parse('I like melons.')

Here's the output I get:

$ bundle exec irb -Ilib
irb(main):001:0> require 've'
=> true
irb(main):002:0> Ve::Manager.set_default_config_for(Ve::Provider::FreelingEn, {:flags => "-f /usr/local/share/freeling/config/en.cfg --flush --nonumb --nodate"})
=> {:flags=>"-f /usr/local/share/freeling/config/en.cfg --flush --nonumb --nodate"}
irb(main):003:0> provider = Ve::Manager.provider_for(:en, :words)
=> #<Ve::Provider::FreelingEn:0x007f0fad04c6d0 @config={:app=>"/usr/local/bin/analyzer", :path=>"", :flags=>"-f /usr/local/share/freeling/config/en.cfg --flush --nonumb --nodate", :freeling_dir_name=>"freeling"}, @stdin=#<IO:fd 10>, @stdout=#<IO:fd 11>, @stderr=#<IO:fd 13>>
irb(main):004:0> parse = provider.parse('I like melons.')
I i PRP 1
like like IN 0.867653
melons melon NNS 1
. . Fp 1

VeEnd veend NP 1
=> #<Ve::Parse::FreelingEn:0x007f0fad03e918 @tokens=[{:raw=>"I i PRP 1", :type=>:parsed, :literal=>"I", :lemma=>"i", :pos=>"PRP", :accuracy=>"1", :characters=>0..0}, {:type=>:unparsed, :literal=>" ", :characters=>1..1}, {:raw=>"like like IN 0.867653", :type=>:parsed, :literal=>"like", :lemma=>"like", :pos=>"IN", :accuracy=>"0.867653", :characters=>2..5}, {:type=>:unparsed, :literal=>" ", :characters=>6..6}, {:raw=>"melons melon NNS 1", :type=>:parsed, :literal=>"melons", :lemma=>"melon", :pos=>"NNS", :accuracy=>"1", :characters=>7..12}, {:raw=>". . Fp 1", :type=>:parsed, :literal=>".", :lemma=>".", :pos=>"Fp", :accuracy=>"1", :characters=>13..13}, {:raw=>"", :type=>:sentence_split, :literal=>""}], @text="I like melons.">
irb(main):005:0>