search

phusion / traveling-ruby

Self-contained Ruby binaries that can run on any Linux distribution and any macOS machine.
http://FooBarWidget.github.io/traveling-ruby
MIT License
2.1k stars 122 forks source link

Which native extensions should we package? #6

Open FooBarWidget opened 9 years ago

FooBarWidget commented 9 years ago

In issue #3 I have described how I envision support for native extensions to work. Since requiring developers to build native extensions themselves is too much hassle, I proposed providing precompiled versions of the most popular native extensions, that developers can just easily drop into their package.

But this raises the question: which native extensions? We can't automatically package them all. For reasons explained in #3, packaging native extensions requires a lot of manual work. So we have to choose.

I plan on packaging these gems:

Because they're popular:

Because Rails apps likely need these:

I've opened a poll for more native extensions. Please add your input here: https://docs.google.com/spreadsheets/d/1oNgrwHyV9UaMGFTttsfA9QPvL461NWLfHeeAap3Az_A/edit?usp=sharing

antoinelyset commented 9 years ago

This gist seems interesting to list all your gems with native extensions : 

require 'rubygems'

specs_with_extensions = Gem::Specification.each.select { |spec| spec.extensions.any?  }
specs_with_extensions.each do |spec|
  puts "#{spec.name} (extensions: #{spec.extensions.inspect})"
end

(source https://gist.github.com/aelesbao/1414b169a79162b1d795)

P.S: I don't know if this is wanted, but your Google Doc is restricted.

FooBarWidget commented 9 years ago

The doc is supposed to be public. I've fixed that and updated the link.

I know I can lookup all gems with native extensions, but I cannot package all gems because it's too much work. The best thing I can do is package the most popular ones, hence the poll.

sheerun commented 9 years ago

@FooBarWidget Can't packaging be automated for any gem? Do you need to choose?

FooBarWidget commented 9 years ago

No it cannot be automated. See https://github.com/phusion/traveling-ruby/issues/3#issuecomment-66348237 where I said:

I think it is impossible to generalize and to automate this. The current Traveling Ruby binaries are hand-tweaked to ensure that they are portable and have no dependencies on non-system libraries. I think it is impossible to guarantee this for gems: to make the resulting binary portable would probably require a lot of manual work.

Packaging nokogiri and sqlite already took me a lot of manual work. I had to install libxml, libxslt and libsqlite3 from source, compile them with very specific settings, edit Makefiles everywhere to ensure correct linking flags, and do this twice (once for Linux, once for OS X).

So yes, I have to choose.

drnic commented 9 years ago

The set above includes my requirements for BOSH CLI https://github.com/cloudfoundry/bosh/blob/master/Gemfile.lock thanks!

FooBarWidget commented 9 years ago

@drnic Thanks, but can you help me by filtering out the gem names that contain native extensions, and modifying the spreadsheet? :)

sheerun commented 9 years ago

I'd vote for porting https://github.com/ffi/ffi

FooBarWidget commented 9 years ago

@sheerun Ruby 2.1 has Fiddle: http://ruby-doc.org/stdlib-2.1.0/libdoc/fiddle/rdoc/Fiddle.html Doesn't that make ffi obsolete?

sheerun commented 9 years ago

There are many many existing gems that use ffi/ffi

drnic commented 9 years ago

I was linking for completeness. I think nokogiri, mysql2, sqlite, pg are the native gems

On Wed, Dec 10, 2014 at 8:48 AM, Hongli Lai notifications@github.com wrote:

@drnic Thanks, but can you help me by filtering out the gem names that contain native extensions, and modifying the spreadsheet? :)

Reply to this email directly or view it on GitHub: https://github.com/phusion/traveling-ruby/issues/6#issuecomment-66482464

noahgibbs commented 9 years ago

Also, gems that want to be compatible with Ruby versions before 2.1 still have to use FFI for awhile. I also have no idea where JRuby is on implementing Fiddle, which may matter for compatibility as well.

FooBarWidget commented 9 years ago

ffi, mysql2, pg and bcrypt have now been packaged.

youngbrioche commented 9 years ago

Nokogiri should definitely be on the list.

FooBarWidget commented 9 years ago

Nokogiri is already packaged. See spreadsheet.

drnic commented 9 years ago

And sqlite3 please?

FooBarWidget commented 9 years ago

Already packaged. See spreadsheet.

Nowaker commented 9 years ago

Great work guys. Traveling Ruby saves us working on VirtKick a lot of work. :-) Thanks for this!

FooBarWidget commented 9 years ago

@Nowaker Glad you like it. :) I'm compiling a list of Traveling Ruby users. Could you write a short ~50 words introduction about Virtkick, and could you write another ~50 words paragraph in which you describe how Traveling Ruby has helped you?

Nowaker commented 9 years ago

@FooBarWidget Sure, no problem!

VirtKick in 50 words: VirtKick is a self-hosted cloud panel, similar to DigitalOcean. With privacy in mind we simplify creating, managing, hosting and providing virtual servers. Forget about installing packages or editing configuration files - all is set up automatically and ready to use. 100% open source.

VirtKick + Traveling Ruby: @Rush will provide a nice text, he did the hacking. :)

drnic commented 9 years ago

@FooBarWidget https://github.com/cloudfoundry-community/traveling-bosh is using it too

BOSH is a Cloud Foundry project for release engineering, deployment, and lifecycle management of large-scale cloud software. BOSH can provision and deploy software over hundreds of VMs and performs monitoring, failure recovery and software updates with zero-to-minimal downtime.

Rush commented 9 years ago

@FooBarWidget: Here is how traveling-ruby helps VirtKick:

VirtKick's goal is a one click installation via standalone installer and/or a distro package. We use rails and other dependencies, many of which we have at bleeding-edge versions from git. traveling-ruby bundles it all. Without it, the installation process was long, required a network connection and lots of hard dependencies.

FooBarWidget commented 9 years ago

@Rush @drnic Thanks. :) Could you also provide photos, icons or pictures that I can use? Should be at least 256x256 pixels.

Nowaker commented 9 years ago

@FooBarWidget Some logos are here: https://press.virtkick.io/presskit. Let me know if it's enough, or you need something more.

FooBarWidget commented 9 years ago

@Nowaker Looks good. Should I quote you or @Rush? What are your positions within your organization?

FooBarWidget commented 9 years ago

I've added your testimonials to the website, but I've edited them a little bit: http://phusion.github.io/traveling-ruby/ Is this ok?

Nowaker commented 9 years ago

@FooBarWidget Looks good, thanks. :-)

drnic commented 9 years ago

Is it possible to include https://github.com/brianmario/yajl-ruby?

Sorry I didn't spot it in https://github.com/cloudfoundry/bosh/blob/master/Gemfile.lock#L31 before as a native extension.

drnic commented 9 years ago

Specifically yajl-ruby (~> 1.1.0) (perhaps in addition to 1.2.1)

See other issues - I'm struggling to build this myself on both osx & linux - sorry.

FooBarWidget commented 9 years ago

I can only package a single version. Maintaining two versions is too much of a burden.

Sent from my Android phone.

drnic commented 9 years ago

@FooBarWidget ok, the latest yajl-ruby is ok if that's all that's possible. I'll try to work it in upstream. TIA.

FooBarWidget commented 9 years ago

Could you tell me why you need yajl-ruby? Is the regular json gem not good enough?

drnic commented 9 years ago

I'm the middleman - trying to package the BOSH CLI which is developed by a team at Pivotal; so I'm not sure if there's history around the choice of JSON gem /cc @cppforlife (BOSH product manager).

On Wed, Dec 17, 2014 at 3:46 PM, Hongli Lai notifications@github.com wrote:

Could you tell me why you need yajl-ruby? Is the regular json gem not good enough?

Reply to this email directly or view it on GitHub: https://github.com/phusion/traveling-ruby/issues/6#issuecomment-67418527

drnic commented 9 years ago

Unsure still about switching to pure JSON gem, but BOSH CLI has upgraded to latest yajl-ruby version https://github.com/cloudfoundry/bosh/pull/715#issuecomment-67543637

drnic commented 9 years ago

@FooBarWidget is it ok to include yajl-ruby please? Its my last dependency (afaict) to ship the the bosh-bootstrap experience into traveling-bosh https://github.com/cloudfoundry-community/bosh-bootstrap/blob/master/ChangeLog.md#v015

FooBarWidget commented 9 years ago

@drnic Done. Will be part of the next version.

drnic commented 9 years ago

Thanks!!

On Fri, Dec 19, 2014 at 11:18 AM, Hongli Lai notifications@github.com wrote:

@drnic Done. Will be part of the next version.

Reply to this email directly or view it on GitHub: https://github.com/phusion/traveling-ruby/issues/6#issuecomment-67683657

FooBarWidget commented 9 years ago

And 20141219 is released.

drnic commented 9 years ago

Ahh, damn (for me). The whole bunch of the BOSH dependency structure requires nokogiri ~> 1.5. Started PRing the dependencies and created parent issue. https://github.com/cloudfoundry/bosh/issues/719

(Just an FYI for the challenges of us only having one native C version of a gem)

drnic commented 9 years ago

Ooh I found a hack for the moment:

def create_package(target)
...
  # HACK: Backporting the pre-built nokogiri.bundle to the bundled 1.5.11
  extn = Dir["#{package_dir}/lib/vendor/ruby/2.1.0/extensions/*"].first
  sh "mv #{extn}/2.1.0-static/nokogiri-{1.6.5,1.5.11}"

  if !ENV['DIR_ONLY']
...
end
drnic commented 9 years ago

Could we add eventmachine 1.0.4 (latest) please?

drnic commented 9 years ago

And thin please (its use is embedded in a bosh CLI process; I can't just switch out to webrick).

FooBarWidget commented 9 years ago

I'm working on support for nokogiri 1.5, but it's rather old doesn't have a bundled libxml, so I need to compile that.

FooBarWidget commented 9 years ago

It turns out to be harder than I thought. It's best if you upgrade to Nokogiri 1.6.

drnic commented 9 years ago

I have a functioning workaround for nokogiri for the moment whilst I'm trying to get everyone to upgrade. Thin & eventmachine are the newly discovered deps.

On Sun, Dec 21, 2014 at 2:41 AM, Hongli Lai notifications@github.com wrote:

It turns out to be harder than I thought. It's best if you upgrade to Nokogiri 1.6.

Reply to this email directly or view it on GitHub: https://github.com/phusion/traveling-ruby/issues/6#issuecomment-67766284

drnic commented 9 years ago

BTW, new workaround for using latest nokogiri and latest thin/eventmachine (when they are released) is Gemfile + github branches; whilst waiting for PRs & gems to be cut.

FooBarWidget commented 9 years ago

Eventmachine and Thin have been added.

drnic commented 9 years ago

Thanks mate!

On Sun, Dec 21, 2014 at 12:27 PM, Hongli Lai notifications@github.com wrote:

Eventmachine and Thin have been added.

Reply to this email directly or view it on GitHub: https://github.com/phusion/traveling-ruby/issues/6#issuecomment-67783376

drnic commented 9 years ago

@FooBarWidget can you bump this ticket again when you have a chance to cut new releases? thanks! http://traveling-ruby.s3-us-west-2.amazonaws.com/list.html

drnic commented 9 years ago

Thin/Eventmachine released as an xmas present? :D

FooBarWidget commented 9 years ago

@drnic Done. 20141224 is released.