bundler/setup error - Mac OS 10.13.6, ruby 2.3.7p456

[orr721]

hi,

wanted to try it out, but i'm not able to run your demo program (I have put it in sm.rb file):

ruby sm.rb 
/System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- bundler/setup (LoadError)
    from /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    from /Library/Ruby/Gems/2.3.0/gems/porter2stemmer-1.0.1/lib/porter2stemmer.rb:2:in `<top (required)>'
    from /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    from /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    from /Library/Ruby/Gems/2.3.0/gems/smalltext-0.1.0/lib/smalltext.rb:1:in `<top (required)>'
    from /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:127:in `require'
    from /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:127:in `rescue in require'
    from /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:39:in `require'
    from sm.rb:1:in `<main>'

gem install smalltext completed successfully.

ruby -v
ruby 2.3.7p456 (2018-03-28 revision 63024) [universal.x86_64-darwin17]

any ideas what is wrong?

hey, and while I am at it, can you please consider support for using other tokenizers as well? I myself prefer the pragmatic_tokenizer as it supports many languages, has stopwords, cleanup methods, etc.. thank you.

[arjunmenon]

Hey This looks like a missing bundler issue.

Do you have bundler installed? What does gem which bundler say? If it is empty, please install bundler, gem i bundler

And do you have multiple versions of ruby installed in your system? If thats the case I highly recommend using a version manager like rbenv.

[orr721]

yep, that was the issue, gem install bundler fixed the error and the demo runs fine. bundler isn't listed as one of the run-time dependencies though..

any thoughts about the tokenizer question? thx.

[arjunmenon]

Bundler is generally part of a standard Ruby setup.

While building this library I had evaluated many tokenizers, including pragmatic_tokenizer. What I remember is it could not split some types of sentences the way I was looking for but the tokenizer gem, which I have included, did.

The preprocessor for Smalltext can accept your list of stopwords, I simply have not exposed that config yet. As far as other languages are concerned I can surely include an option to have the preprocessor consider that. Give me some time.