brianmario / yajl-ruby

A streaming JSON parsing and encoding library for Ruby (C bindings to yajl)
http://rdoc.info/projects/brianmario/yajl-ruby
MIT License
1.48k stars 169 forks source link

Add support for Ruby 2.0 #116

Open jarmo opened 11 years ago

jarmo commented 11 years ago

On Ruby 2.0 yajl does not work (at least on Windows):

require "yajl"
LoadError Exception: cannot load such file -- yajl/2.0/yajl
shellscape commented 11 years ago

Seconded for Ruby 2.0 on Windows. Same exact error.

luislavena commented 11 years ago

@jarmo and @shellscape please see RubyInstaller announcement for Ruby 2.0.0 release:

https://groups.google.com/d/msg/rubyinstaller/mg5ailNICvM/QbBfNByec-0J

* Existing pre-compiled gems are not Ruby 2.0 compatible 

Ruby 2.0 introduces ABI breakage which means compiled C extensions with previous 
1.9.3 will work with Ruby 2.0. 

DO NOT install Ruby 2.0 on top of existing Ruby 1.9.3, or try to use compiled 
extensions with it. 

You will be required to force compilation of those gems: 

    gem install <name> --platform=ruby 

This will require you have the extra dependencies required for that gem to 
compile. Look at the gem documentation for the requirements. 

You can use that to install yajl-ruby without issues:

C:\Users\Luis>ruby -v
ruby 2.0.0p0 (2013-02-24) [i386-mingw32]

C:\Users\Luis>gem --version
2.0.3

C:\Users\Luis>gem install yajl-ruby --platform=ruby
Fetching: yajl-ruby-1.1.0.gem (100%)
Temporarily enhancing PATH to include DevKit...
Building native extensions.  This could take a while...
Successfully installed yajl-ruby-1.1.0
1 gem installed

C:\Users\Luis>irb
DL is deprecated, please use Fiddle
irb(main):001:0> require "yajl"
=> true
irb(main):002:0> Yajl.load '{"foo":1}'
=> {"foo"=>1}
irb(main):003:0>
brianmario commented 11 years ago

@luislavena thank you for that!

jarmo commented 11 years ago

@luislavena thanks for the information. To be honest, i did know that trick.

However, the problem is that i'm not aware of the possibility to specify that platform option for Bundler. There is a platform option available in Gemfile, but it means a different thing - the gem will be installed only on specified platform.

This means that Bundler will install and load always the precompiled version of the yajl gem even if i have it installed prior with --platform=ruby.

The problem appears like this:

C:\users\Jarmo>ruby -v
ruby 2.0.0p0 (2013-02-24) [i386-mingw32]

C:\users\Jarmo>gem install yajl-ruby --platform=ruby
Fetching: yajl-ruby-1.1.0.gem (100%)
Temporarily enhancing PATH to include DevKit...
Building native extensions.  This could take a while...
Successfully installed yajl-ruby-1.1.0
1 gem installed

C:\users\Jarmo>irb
DL is deprecated, please use Fiddle
irb(main):001:0> require "yajl"
=> true
irb(main):002:0> Yajl.load '{"foo":1}'
=> {"foo"=>1}
irb(main):003:0> exit

C:\users\Jarmo>bundle init
Writing new Gemfile to C:/users/Jarmo/Gemfile

C:\users\Jarmo>echo gem "yajl-ruby" >> Gemfile

C:\users\Jarmo>cat Gemfile
# A sample Gemfile
source "https://rubygems.org"

# gem "rails"
gem "yajl-ruby"

C:\users\Jarmo>bundle install
Fetching gem metadata from https://rubygems.org/..
Resolving dependencies...
Installing yajl-ruby (1.1.0)
Using bundler (1.3.4)
Your bundle is complete!
Use `bundle show [gemname]` to see where a bundled gem is installed.

C:\users\Jarmo>bundle show
Gems included by the bundle:
  * bundler (1.3.4)
  * yajl-ruby (1.1.0)

C:\users\Jarmo>bundle exec ruby -e "require 'yajl'"
C:/bin/Ruby200-p0/lib/ruby/gems/2.0.0/gems/yajl-ruby-1.1.0-x86-mingw32/lib/yajl/yajl.rb:2:in `require': cannot load such file -- yajl/2.0/yajl (LoadError)
        from C:/bin/Ruby200-p0/lib/ruby/gems/2.0.0/gems/yajl-ruby-1.1.0-x86-mingw32/lib/yajl/yajl.rb:2:in `<top (required)>'
        from C:/bin/Ruby200-p0/lib/ruby/gems/2.0.0/gems/yajl-ruby-1.1.0-x86-mingw32/lib/yajl.rb:1:in `require'
        from C:/bin/Ruby200-p0/lib/ruby/gems/2.0.0/gems/yajl-ruby-1.1.0-x86-mingw32/lib/yajl.rb:1:in `<top (required)>'
        from -e:1:in `require'
        from -e:1:in `<main>'

C:\users\Jarmo>gem list yajl
*** LOCAL GEMS ***
yajl-ruby (1.1.0 ruby x86-mingw32)

As you can see then Bundler detected that there is a precompiled yajl-ruby for my platform available on rubygems and installed it even if i had already installed a working version of the gem on my system.

Please let me know if there's a workaround for such a situation.

If there's not, then please reopen this issue and: 1) Release a new version of yajl-ruby without precompiled binaries so Ruby would compile these itself; 2) or release a new version of yajl-ruby with precompiled binaries as it is with current version of the gem.

brianmario commented 11 years ago

I know it's kindof a pain in comparison, but you should be able to download the .gem file itself (not the pre-compiled one) and stick that in your bundle's cache directory and have it use that instead. I haven't verified that on Windows, but that's how it works everywhere else I know of.

I'll reopen this until I get a chance to push out another cross-compiled build with 1.8, 1.9 and 2.0 versions inside. What a pain...

luislavena commented 11 years ago

Bundler used to detect already installed gem and use that skipping installation of the one from rubygems.org.

Perhaps you have isolation enabled?

Sorry for top posting. Sent from mobile. On Mar 17, 2013 5:08 AM, "Jarmo Pertman" notifications@github.com wrote:

@luislavena https://github.com/luislavena thanks for the information. To be honest, i did know that trick.

However, the problem is that i'm not aware of the possibility to specify that platform option for Bundler. There is a platform option available in Gemfile, but it means a different thing - the gem will be installed only on specified platform.

This means that Bundler will install and load always the precompiled version of the yajl gem even if i have it installed prior with --platform=ruby.

The problem appears like this:

C:\users\Jarmo>ruby -v ruby 2.0.0p0 (2013-02-24) [i386-mingw32]

C:\users\Jarmo>gem install yajl-ruby --platform=ruby Fetching: yajl-ruby-1.1.0.gem (100%) Temporarily enhancing PATH to include DevKit... Building native extensions. This could take a while... Successfully installed yajl-ruby-1.1.0 1 gem installed

C:\users\Jarmo>irb DL is deprecated, please use Fiddle irb(main):001:0> require "yajl" => true irb(main):002:0> Yajl.load '{"foo":1}' => {"foo"=>1} irb(main):003:0> exit

C:\users\Jarmo>bundle init Writing new Gemfile to C:/users/Jarmo/Gemfile

C:\users\Jarmo>echo gem "yajl-ruby" >> Gemfile

C:\users\Jarmo>cat Gemfile

A sample Gemfile

source "https://rubygems.org"

gem "rails"

gem "yajl-ruby"

C:\users\Jarmo>bundle install Fetching gem metadata from https://rubygems.org/.. Resolving dependencies... Installing yajl-ruby (1.1.0) Using bundler (1.3.4) Your bundle is complete! Use bundle show [gemname] to see where a bundled gem is installed.

C:\users\Jarmo>bundle show Gems included by the bundle:

  • bundler (1.3.4)
  • yajl-ruby (1.1.0)

C:\users\Jarmo>bundle exec ruby -e "require 'yajl'" C:/bin/Ruby200-p0/lib/ruby/gems/2.0.0/gems/yajl-ruby-1.1.0-x86-mingw32/lib/yajl/yajl.rb:2:in require': cannot load such file -- yajl/2.0/yajl (LoadError) from C:/bin/Ruby200-p0/lib/ruby/gems/2.0.0/gems/yajl-ruby-1.1.0-x86-mingw32/lib/yajl/yajl.rb:2:in<top (required)>' from C:/bin/Ruby200-p0/lib/ruby/gems/2.0.0/gems/yajl-ruby-1.1.0-x86-mingw32/lib/yajl.rb:1:in require' from C:/bin/Ruby200-p0/lib/ruby/gems/2.0.0/gems/yajl-ruby-1.1.0-x86-mingw32/lib/yajl.rb:1:in<top (required)>' from -e:1:in require' from -e:1:in

'

C:\users\Jarmo>gem list yajl * LOCAL GEMS * yajl-ruby (1.1.0 ruby x86-mingw32)

As you can see then Bundler detected that there is a precompiled yajl-rubyfor my platform available on rubygems and installed it even if i had already installed a working version of the gem on my system.

Please let me know if there's a workaround for such a situation.

If there's not, then please reopen this issue and: 1) Release a new version of yajl-ruby without precompiled binaries so Ruby would compile these itself; 2) or release a new version of yajl-ruby with precompiled binaries as it is with current version of the gem.

— Reply to this email directly or view it on GitHubhttps://github.com/brianmario/yajl-ruby/issues/116#issuecomment-15019042 .

jarmo commented 11 years ago

What do you mean by "isolation enabled"?

luislavena commented 11 years ago

There is a setting in bundler config that disables usage of system wide installed gems.

I used to manually install the gem (with compilation) and then simply do "bundle check" and not install.

What is the output of "bundle config" inside the directory of your gemfile?

Sorry for top posting. Sent from mobile. On Mar 17, 2013 10:09 AM, "Jarmo Pertman" notifications@github.com wrote:

What do you mean by "isolation enabled"?

— Reply to this email directly or view it on GitHubhttps://github.com/brianmario/yajl-ruby/issues/116#issuecomment-15022409 .

jarmo commented 11 years ago

I do not seem to have any specific settings set for Bundler:

C:\users\Jarmo>bundle config
Settings are listed in order of priority. The top value will be used.
AnilMaktala commented 11 years ago

$ Gem install yajl-ruby Building native extensions. This could take a while ... Successfully installed yajl-ruby-1.1.0 Parsing documentation for yajl-ruby-1.1.0 unable to convert "\ x90" from ASCII-8BIT to UTF-8 for lib / yajl / yajl.so, skipping Done installing documentation for yajl-ruby (0 sec). 1 gem installed

facing below error when i ran this command : jekyll build --source "templates" --destination "dev"

c :/ Local/Ruby/ruby-2.0.0-p0-i386-mingw32/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb: 45: in require ': cannot load such file - yajl / 2.0/yajl (LoadError) from c :/ Local/Ruby/ruby-2.0.0-p0-i386-mingw32/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb: 45: inrequire ' from c :/ Local/Ruby/ruby-2.0.0-p0-i386-mingw32/lib/ruby/gems/2.0.0/gems/yajl-ruby-1.1.0-x86-mingw32/lib/yajl/yajl. rb: 2: in <top (required)> ' from c :/ Local/Ruby/ruby-2.0.0-p0-i386-mingw32/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb: 45: inrequire ' from c :/ Local/Ruby/ruby-2.0.0-p0-i386-mingw32/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb: 45: in require ' from c :/ Local/Ruby/ruby-2.0.0-p0-i386-mingw32/lib/ruby/gems/2.0.0/gems/yajl-ruby-1.1.0-x86-mingw32/lib/yajl.rb: 1: in<top (required)> ' from c :/ Local/Ruby/ruby-2.0.0-p0-i386-mingw32/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb: 110: in require ' from c :/ Local/Ruby/ruby-2.0.0-p0-i386-mingw32/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb: 110: inrescue in require ' from c :/ Local/Ruby/ruby-2.0.0-p0-i386-mingw32/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb: 35: in require ' from c :/ Local/Ruby/ruby-2.0.0-p0-i386-mingw32/lib/ruby/gems/2.0.0/gems/pygments.rb-0.3.7/lib/pygments/popen.rb: 3: in<top (required)> ' from c :/ Local/Ruby/ruby-2.0.0-p0-i386-mingw32/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb: 106: in require ' from c :/ Local/Ruby/ruby-2.0.0-p0-i386-mingw32/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb: 106: inrequire ' from c :/ Local/Ruby/ruby-2.0.0-p0-i386-mingw32/lib/ruby/gems/2.0.0/gems/pygments.rb-0.3.7/lib/pygments.rb: 1: in <top (required)> ' from c :/ Local/Ruby/ruby-2.0.0-p0-i386-mingw32/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb: 58: inrequire ' from c :/ Local/Ruby/ruby-2.0.0-p0-i386-mingw32/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb: 58: in require ' from c :/ Local/Ruby/ruby-2.0.0-p0-i386-mingw32/lib/ruby/gems/2.0.0/gems/jekyll-0.12.1/lib/jekyll.rb: 27: in<top (required)> ' from c :/ Local/Ruby/ruby-2.0.0-p0-i386-mingw32/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb: 58: in require ' from c :/ Local/Ruby/ruby-2.0.0-p0-i386-mingw32/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb: 58: inrequire ' from c :/ Local/Ruby/ruby-2.0.0-p0-i386-mingw32/lib/ruby/gems/2.0.0/gems/jekyll-0.12.1/bin/jekyll: 20: in <top (required )> ' from c :/ Local/Ruby/ruby-2.0.0-p0-i386-mingw32/bin/jekyll: 23: inload ' from c :/ Local/Ruby/ruby-2.0.0-p0-i386-mingw32/bin/jekyll: 23: in `

'

luislavena commented 11 years ago

$ Gem install yajl-ruby Building native extensions. This could take a while ... Successfully installed yajl-ruby-1.1.0 Parsing documentation for yajl-ruby-1.1.0 unable to convert "\ x90" from ASCII-8BIT to UTF-8 for lib / yajl / yajl.so, skipping Done installing documentation for yajl-ruby (0 sec). 1 gem installed

If you see lower in the backtrace: Local/Ruby/ruby-2.0.0-p0-i386-mingw32/lib/ruby/gems/2.0.0/gems/yajl-ruby-1.1.0-x86-mingw32/lib/yajl/yajl. rb

You have x86-mingw32 gem installed.

As mentioned in the RubyInstaller release notes:

https://groups.google.com/d/msg/rubyinstaller/mg5ailNICvM/QbBfNByec-0J

* Existing pre-compiled gems are not Ruby 2.0 compatible 

Ruby 2.0 introduces ABI breakage which means compiled C extensions with previous 
1.9.3 will work with Ruby 2.0. 

DO NOT install Ruby 2.0 on top of existing Ruby 1.9.3, or try to use compiled 
extensions with it. 

You will be required to force compilation of those gems: 

    gem install <name> --platform=ruby 

This will require you have the extra dependencies required for that gem to 
compile. Look at the gem documentation for the requirements. 

So:

Bundler will keep attempting to install x86-mingw32, so you will need to be careful when doing bundle install or update.

roberts1000 commented 10 years ago

One more piece of the puzzle. It turns out bundler, prior to version 1.4, doesn't recognize 64 bit gems on Windows https://github.com/bundler/bundler/issues/2658 which causes the problems noted above.

A full workaround appears to be:

  1. Upgrade to bundler 1.4 or greater (currently using 1.5.2 which worked for me)
  2. (Optional) Manually install yajl-ruby via, gem install yajl-ruby --platform=ruby
  3. Delete your Gemfile.lock and run bundle again

A full article can be found here.

jf647 commented 10 years ago

I was able to use gem-compiler to get a binary gem for v1.2.1 for Ruby 2.0 on Windows that works built using gem-compiler.

The gem can be downloaded here (depending on the level of trust you have in me):

https://s3-us-west-1.amazonaws.com/nadtshare/yajl-ruby-1.2.1-x86-mingw32.gem

Or you can build it yourself by installing rubyinstaller, the devkit and running

gem install gem-compiler
gem fetch yajl-ruby --platform=ruby
gem compile yajl-ruby-1.2.1.gem

To use the binary gem:

gem uninstall yajl-ruby
gem install yajl-ruby-1.2.1-x86-mingw32.gem

However, what led me to this problem in the first place (foodcritic from Chef - see [https://github.com/opscode/chef-dk/issues/124](this issue)) still has a problem because it's gemspec puts a restriction of '~> 1.1.0'.

That was the only safe way to cover Windows and Unix on Ruby 1.9, but in order to change it to '~> 1.2.1', rubygems.org has to host a binary version of the 1.2.1 gem. Otherwise your app will work on Unix and fail on Windows.

So what is really needed here is for Brian or Lloyd to upload a Windows binary gem for v1.2.1 to rubygems, which will let people who consume yajl-ruby up their spec version to v1.2.1 and everyone should be happy.

lozandier commented 10 years ago

@luislavena @jf647 For yajl-ruby I've always have had to remove yajl-ruby-1.2.1-x86-mingw32.gem to get anything working like bundle exec rails s.

XhmikosR commented 9 years ago

Bump this is a pretty serious issue on Windows with Ruby >= 2.0 and breaks many other gems.

luislavena commented 9 years ago

Unsure who is maintainer of this gem, but: you can use rake-compiler-dev-box to cross-compile yajl-ruby within OSX/Linux to Windows:

https://github.com/tjschuck/rake-compiler-dev-box

All instructions on usage there. You can also include support for 2.1.x since is enabled by default.

Hope this helps.

XhmikosR commented 9 years ago

The problem is the gem that is published on rubygems.org.

/CC @brianmario @sferik

luislavena commented 9 years ago

@XhmikosR you cannot fix by replacing published gems, the fix can be done moving forward, not backwards.

XhmikosR commented 9 years ago

@luislavena: well, we have this given situation. yajl-ruby breaks pygments.rb which in turn breaks Jekyll on Windows with Ruby >= 2.0. This is my problem I'm trying to find a solution once and for all.

I don't have a lot of experience with Ruby, but we really need to get this fixed. Not for me, but for everyone's sake. Otherwise we will be stuck with Ruby 1.9.3 for a lot more. Which wouldn't be so much trouble but the next Jekyll version will drop 1.9.3 support.

I hope you see my point of view here. :/

luislavena commented 9 years ago

@XhmikosR pygments is no longer mandatory for Jekyll, you can use Rouge instead.

Since you're not familiar with Ruby, crash course:

That is the process, so I leave to the gem maintainer to decide what or how to do it.

I feel your pain that this is a problem, needs to be solved, etc, etc. But fixes to projects cannot happen magically or by continuous me too reports on issues. Fixes happens when people contribute or provide help or assistance.

Either Jekyll maintainers or contributors should step up and take the issue on their side, since they are the ones going to drop support on 1.9.3, not me, neither yajl-ruby developers.

You, me or zillion more folks being annoyed by this will not get this fixed.

Hope you see my point of view.

XhmikosR commented 9 years ago

@luislavena: I'm completely aware of the rouge alternative. I don't manage GitHub pages though, and they use pygments.

Also, give me a break, I didn't know I wasn't allowed to express myself when something like this big affects me.

There's clearly an issue, and it needs to be fixed. Backwards or not, from whoever responsible.

luislavena commented 9 years ago

Also, give me a break, I didn't know I wasn't allowed to express myself when something like this big affects me.

Sure, take a break, relax, have fun.

Sure gem maintainers or contributors will carve some of their free time (you know, that one that happens between paid work and family and friends) to come and work on projects just to fix problems of others that do not care at all about their effort put free on projects, simply because you think you deserve or entitled to get a fix, because you know, all the money you have put on this project.

As you express yourself, sure I believe I'm allowed to express myself :smile:

XhmikosR commented 9 years ago

/trolling on Right, please, teach me how open source works. I'm new in this area and I need a master. /trolling off

Anyway, this leads nowhere. Either provide a solution or leave the people who are affected to express their problem.

sferik commented 9 years ago

@XhmikosR I am not an owner of this gem, only @brianmario is, so he is the only one able to push a fix to rubygems.org.

luislavena commented 9 years ago

Either provide a solution

Right on:

yajl-ruby uses rake-compiler:

To compile things.

rake-compiler:

https://github.com/rake-compiler/rake-compiler

Was created by me 6 years ago precisely to provide binaries for Windows users so they shouldn't be suffering by installation problems (aka: you)

Last year, to make things more simpler, another developer, from frustration wanting to release bcrypt gem for Windows users created rake-compiler-dev-box:

https://github.com/tjschuck/rake-compiler-dev-box

Which is another tool for gem authors to make it more easy to release gems for Windows users so they don't suffer (aka: you don't suffer).

So sure, vent out all your frustration since open-source developers are also your therapists that need to listen to your complains while also fix your problems.

The solution was provided, but perhaps gem author is busy or have other things to attend, have you considered that?

Instead of saying how affected by this you and others are, perhaps any of the Jekyll developers can step up and offer help instead of complaining?

I believe there are other Ruby developers that contribute to Jekyll that care enough for Windows users that can help out? (At least the documentation of Jekyll says that: http://jekyllrb.com/docs/windows/#installation)

You see, the attitude you present goes along the lines of other users who believe that because someone's creation was shared and made freely available for others to benefit is enough reason for them to become your support line, therapist and repairman.

You stated that you didn't know (or care) on how Ruby and gems work (or were released). The comment I gave previously aimed to provide guidance to the gem author and/or maintainer to ease them the process to release those gems.

My comment wasn't directed at you, or to you, but you decided that the information I provided didn't solve your problem and that was good enough to keep complaining about it.

If you think I trolled you by my statements, please accept my apologies.

But if after reading all this you still believe gem author or maintainer owes you something, please go see an specialist, or give that money to any developer that is willing to help out solve this, but do something that doesn't involve complaining.

pkerpedjiev commented 6 years ago

Sorry to wade into this old issue but I'm getting a similar problem on OS X:

(cenv3) pete@twok:~/projects/emptypipes [develop|?P]$ gem install yajl-ruby --platform=ruby
Fetching: yajl-ruby-1.4.1.gem (100%)
Building native extensions. This could take a while...
Successfully installed yajl-ruby-1.4.1
Parsing documentation for yajl-ruby-1.4.1
Installing ri documentation for yajl-ruby-1.4.1
Done installing documentation for yajl-ruby after 0 seconds
1 gem installed
(cenv3) pete@twok:~/projects/emptypipes [develop|?P]$ bundle install
Fetching gem metadata from https://rubygems.org/...............

...

yajl_ext.c:881:22: error: use of undeclared identifier 'rb_cFixnum'
    rb_define_method(rb_cFixnum, "to_json", rb_yajl_json_ext_fixnum_to_json, -1);
                     ^
17 warnings and 1 error generated.
make: *** [yajl_ext.o] Error 1

make failed, exit code 2

Gem files will remain installed in /usr/local/lib/ruby/gems/2.4.0/gems/yajl-ruby-1.2.1 for inspection.
Results logged to /usr/local/lib/ruby/gems/2.4.0/extensions/x86_64-darwin-16/2.4.0/yajl-ruby-1.2.1/gem_make.out

An error occurred while installing yajl-ruby (1.2.1), and Bundler cannot continue.
Make sure that `gem install yajl-ruby -v '1.2.1' --source 'https://rubygems.org/'` succeeds before bundling.

In Gemfile:
  jekyll was resolved to 2.5.1, which depends on
    pygments.rb was resolved to 0.6.3, which depends on
      yajl-ruby

I'm completely clueless in the world of gems and ruby and am hoping that maybe somebody here can quickly spot what I'm doing wrong.

luislavena commented 6 years ago

Hello @pkerpedjiev, the error is indicating that the version that fails installation is 1.2.1, not the latest one.

I will recommend update to latest yajl-ruby:

$ bundle update yajl-ruby
cooljeanius commented 2 months ago

Note that this issue prevents the use of the rubocop codescanning action with GitHub Actions: https://github.com/arthurnn/code-scanning-rubocop