search

jordansissel / fpm

Effing package management! Build packages for multiple platforms (deb, rpm, etc) with great ease and sanity.
http://fpm.readthedocs.io/en/latest/
Other
11.14k stars 1.07k forks source link

Proposal: Rewrite in Go? #669

Closed jordansissel closed 9 years ago

jordansissel commented 10 years ago

@evanphx proposed possibly writing fpm in Go to solve some of his pains. Like all things, I am open to such proposals and am filing this issue as a means to further discussion.

Fundamentally, the goal of fpm is solving problems, and ruby is becoming a problem because versioning changes so much of the internals. I want installing fpm to be an easy process. I want using fpm to be easy.

There are ways to solve the current problems fpm has with ruby, but before we talk about that, let's try to list subjectively the pros and cons of being in ruby and being in go.

I've tried to focus the items listed below as long-term items, not short-term things.

Invariants

jordansissel commented 10 years ago

If you have thoughts on this, please let me know. I am most specifically interested in the benefits of ruby vs go for users. For developers, we can solve any problems with code, so I am not worried there.

jordansissel commented 10 years ago

Replacing Clamp's functionality in Go is gonna be a pain, probably. Go's flags library doesn't even come close.

r4um commented 10 years ago

I would have voted go until I came across rust :), its still evolving though and would be better to wait for 1.0.

User pain with install/setup is primary problem with ruby/rubygems and anything that replaces it with single easily deployable/usable binary is a huge win. That being said has the path of bundling fpm as a single jar been considered since JRuby is supported ? Slow startup time might be a problem(?) though.

Replacing Clamp's functionality in Go is gonna be a pain, probably. Go's flags library doesn't even come close.

Very true, also same for many other dependencies, though that gives a chance to write/contribute more go code :).

torrancew commented 10 years ago

Still thinking this through further, but I would fully support a Go rewrite under the following conditions (disclaimer: my primary use of FPM is via the Ruby API, and I use it very heavily):

jordansissel commented 10 years ago

@r4um I like what I read about Rust, but Mozilla's leadership of the project is terribly confusing as it continues to produce new projects that use nodejs and go, not rust. If the apparent owner of the project doesn't seem to value it, it makes me feel like the project has no future, so I'm not currently interested in it. Contrast this with Google which has that gopher team taking over internal projects and implementing them in Go.

jordansissel commented 10 years ago

@torrancew fpm's principles, features, and behaviors should stay exactly the same regardless of the underlying implementation. The danger for you obviously is API problems, but I think we can solve that. I mean, we could ship, for now, a rubygem 'fpm' wrapper that invokes whatever programmatic interface to the 'new fpm' sanely.

r4um commented 10 years ago

@r4um I like what I read about Rust, but Mozilla's leadership of the project is terribly confusing as it continues to produce new projects that use nodejs and go, not rust. If the apparent owner of the project doesn't seem to value it, it makes me feel like the project has no future, so I'm not currently interested in it. Contrast this with Google which has that gopher team taking over internal projects and implementing them in Go.

Go it is then :+1:. +1 for a rewrite.

Still thinking this through further, but I would fully support a Go rewrite under the following conditions (disclaimer: my primary use of FPM is via the Ruby API, and I use it very heavily):

Can you share some examples/links to the code ? It will help.

FPM should remain easy to extend, for some definition of "extend" - I consume the API to customize packaging for things like apps of a specific framework, and I do some very useful things in this space (am still trying to get this code cleared for OSS release, so not available to reference at this time) FPM's guiding principles should remain the same - FPM has been insanely easy to contribute to, from both a code and "policy" perspective

As long as fpm provides and can consume a language agnostic interface (via a config, spec etc.) this should not be a problem.

evanphx commented 10 years ago

I spiked some code to show some basics, basically convert a dir to a deb. It's just so show a bit how we'd progress: https://github.com/evanphx/fpm/tree/master/go

jordansissel commented 10 years ago

Also worth concerning facts like omnibus and fpm-cookery using fpm very heavily. Don't want to mess that up.

rouge8 commented 10 years ago

As a non-rubyist user, :+1:. The most frustrating part of building packages with fpm is by far getting it to install on random platforms we have to make packages for.

torrancew commented 10 years ago

@jordansissel exactly - not terribly concerned on that front. When I referenced extension points and hooks earlier, I am essentially referring to the ability to create tools like omnibus, fpm-cookery, custom build tools, etc on top of fpm.

torrancew commented 10 years ago

For the record, :+1: here.

neoice commented 10 years ago

is there any concern about limiting the developer base by using Go over Ruby? while the vast majority of fpm is written by just a few people, there's a few dozen other contributors.

jordansissel commented 10 years ago

I have concerns about this, but anecdotally I find go fairly easy to read and would consider it a teaching opportunity (as I do now with ruby) for folks interested in contributing code but without that skill or confidence to do so.

Thoughts? We can be vocal about this "we welcome you to learn with us" type of thing for new coders.

On Monday, April 7, 2014, neoice notifications@github.com wrote:

is there any concern about limiting the developer base by using Go over Ruby? while the vast majority of fpm is written by just a few people, there's a few dozen other contributors.

Reply to this email directly or view it on GitHubhttps://github.com/jordansissel/fpm/issues/669#issuecomment-39749006 .

jordansissel commented 10 years ago

I also note that Go has no Solaris releases available. Packer seems to get away with not supporting Solaris as a runtime environment. I can't tell is Go+Solaris support is on the roadmap (quick googling finds ancient blog/mail threads talking about it).

Does having Solaris support as a runtime environment matter? We could still support solaris packages, just not running fpm on solaris itself.

r4um commented 10 years ago

Experimental support for solaris added in 1.3

Go 1.3 now includes experimental support for Solaris on the amd64 (64-bit x86) architecture.

http://tip.golang.org/doc/go1.3

jordansissel commented 10 years ago

Looked around at golang's libraries for flags, most powerful one I found is https://github.com/jessevdk/go-flags but it doesn't seem amenable to replacing the per-package flags fpm supports today, I mean, not without some bizarre hacks or other weirdness.

evanphx commented 10 years ago

Well, because go doesn't support dynamic loading of code, it seems like all the flags could be in one place.

evanphx commented 10 years ago

But if they need to be broken up, it's doable.

jordansissel commented 10 years ago

@evanphx I forgot that about struct embedding, so I think we can achieve modularity (and ease of maintenance) while still achieving the flags interface fpm has today:

type DebOptions struct { ... }
type RPMOptions struct { ... }
type SolarisOptions struct { ... }
type  AllOptions struct  {
  DebOptions
  RPMOptions
  SolarisOptions
  // ...
}

Then do flags testing on that. I've tested it and this strategy works (as expected) with go-flags.

jordansissel commented 10 years ago

Reviewing known projects that use fpm directly:

opscode/omnibus-ruby uses fpm currently in two ways:

bernd/fpm-cookery uses it in several places, via API only, best I can tell:

Macintosh(~/build/fpm-cookery) % ack FPM::P
lib/fpm/cookery/package/dir.rb
9:          FPM::Package::Dir.new

lib/fpm/cookery/package/gem.rb
9:          FPM::Package::Gem.new

lib/fpm/cookery/package/npm.rb
9:          FPM::Package::NPM.new

lib/fpm/cookery/package/python.rb
9:          FPM::Package::Python.new

lib/fpm/cookery/packager.rb
161:          output_class = FPM::Package.types[@target]
167:          rescue FPM::Package::FileAlreadyExists

spec/package_spec.rb
23:        FPM::Package::Dir.new
torrancew commented 10 years ago

@jordansissel did you evaluate https://github.com/spf13/cobra, as well? Reminds me a bit of clamp.

ringods commented 10 years ago

Putting the specific programming language aside, I want to ask another question: will fpm become a tool that doesn't need underlying tools like ar, unzip and/or tar?

Before I found this ticket, I was thinking of proposing to make fpm pure Ruby to let it work on Windows too, for every package type.

jordansissel commented 10 years ago

Ideally, yes. Haven't had energy for doing that, though. ;)

On Wednesday, May 21, 2014, Ringo De Smet notifications@github.com wrote:

Putting the specific programming language aside, I want to ask another question: will fpm become a tool that doesn't need underlying tools like ar, unzip and/or tar?

Before I found this ticket, I was thinking of proposing to make fpm pure Ruby to let it work on Windows too, for every package type.

— Reply to this email directly or view it on GitHubhttps://github.com/jordansissel/fpm/issues/669#issuecomment-43726983 .

djhaskin987 commented 10 years ago

If we switch languages, we'll have to find quirks to get around in that language, too. The advantage to sticking with ruby is that we already have found ruby's quirks. The work is done.

As to installing, it has been a bit of a pain to install fpm. However, I have found a way to install fpm as a "standalone" by downloading jruby and using jruby to gem install fpm, essentially packaging ruby with fpm, thereby making an fpm jar. If we provide such an artifact or something similar on a download page somewhere, most users' experience of installing our product could be instantly improved.

djhaskin987 commented 10 years ago

If we switch languages, we'll have to find quirks to get around in that language, too. The advantage to sticking with ruby is that we already have found ruby's quirks. The work is done.

As to installing, it has been a bit of a pain to install fpm. However, I have found a way to install fpm as a "standalone" by downloading jruby and using jruby to gem install fpm, essentially packaging ruby with fpm, thereby making an fpm jar. If we provide such an artifact or something similar on a download page somewhere, most users' experience of installing our product could be instantly improved.

grepory commented 9 years ago

@jordansissel -- We use https://github.com/codegangsta/cli with some success. It reminds me of Thor for Go. Also, we are a SmartOS shop, so... Testing for you. And writing, btw.

adamonduty commented 9 years ago

I love Go. But I'd be a little hesitant to rewrite a mature tool like fpm in it.

There are so many unknowns with Go. Will you have all the libraries you need? Will you have the developer community available to contribute? Will you just give up halfway through because the task is so monumental? Go is less expressive than ruby.

For me, the sweet spot for Go is when I need more concurrency and/or more speed. fpm seems to need neither, so the primary benefit would be easier distribution.

But couldn't you solve that in ruby? The jar route mentioned by @djhaskin987 is one I've taken before myself, but comes with its own problems like supporting different versions of Java, differing JDK implementations, etc. Its better, but IMHO not best.

Instead of putting a large amount of effort into porting a single tool to another language, why not improve the distribution problem for ruby in general? Tools like ocra can already package ruby applications for Windows. We just need a similar tool for linux and other platforms. A small binary (even one written on Go) could unpack an archive with a bundled ruby interpreter and exec a script. And you'd be free to drop support for older ruby versions. Seems win-win to me. Those that want a packaging tool that just works can download the binary distribution, and ruby devs that already have the environment and knowledge won't skip a beat.

ringods commented 9 years ago

If you want a distributable fpm, including a tested Ruby, don't forget to look at Chef's omnibus installer.

jordansissel commented 9 years ago

Indeed -

Most of the user problems I think we have these days are related to distribution. We can provide standalone packages for users to install most easily and probably solve this.

For accessing from Go, we can add an JSON-RPC interface that allows invoking FPM from any language, and once we solve packaging, this becomes pretty compelling.

djhaskin987 commented 9 years ago

I change my mind. Go has much better distribution experience. As far as that goes, I'm all for switching languages. Ruby is a nightmare when it comes to distributing software. Go is not, from what I've heard you get a single executable for the target platform. What a relief that would be! I wouldn't have to -- ironically -- find some wierd way to package a tool meant to make packaging easy.

What could fpm be like in go? Have a look at aptly, a go project designed to make managing package repositories easier. It's built to manage deb repos, but I can easily get it for CentOS as a single binary executable, and it just works.

rholder commented 9 years ago

As for the jar packaging route, I assembled a JRuby doodad similar to what @djhaskin987 described about a year ago that we've been using for all of our production builds (and just recently updated it for 1.3.3). Distributing FPM as a single binary among my team made it a really easy sell and getting it integrated with Jenkins was a snap. I am very much in favor of single standalone binaries and would welcome a Go version of this fine tool as well. :+1: on continuing to make an excellent project even better.

jbcraig commented 9 years ago

I would be concerned that go will kill Solaris 10/11 SPARC support. The go documentation says it has experimental support for illumos / Solaris 11 on x86. Not sure what this means for native Solaris 10 on x86 or SPARC. Lack of SPARC support would kill 50% of fpm's value proposition for me.

jordansissel commented 9 years ago

@jbcraig noted!

As this ticket was mostly a thought experiment, I'm happy keeping fpm in ruby. Rewriting in another language will significantly derail stability (or improvements) in order to keep backwards compatibility and whatnot.

I'm content keeping it in Ruby :)

jordansissel commented 9 years ago

Going to close this. Thanks to everyone who helped provide feedback and ideas :)