Recursive package building

[tabletcorry]

It would be really nice if fpm followed the dependency chain that it reads from the python package. If I forget to check the resulting package for its requirements I can end up deploying a package that will not install...

Perhaps a "--recursion" mode?

Edit: Removed python tag, as this is a general feature. Examples are fpm(ruby) and fabric(python)

[jaybuff]

I wrote a script that uses fpm to spit out all the dependencies of a given package. For example:

require 'fpm'

def deps_for(name)
  pkg = FPM::Package::Python.new
  pkg.input(name)

  deps = []
  pkg.dependencies.each do |dep|
    name, op, version = dep.split(/\s+/)
    op = "==" if op == "="
    deps.push "#{name}#{op}#{version}"
    deps.push deps_for("#{name}#{op}#{version}")
  end

  deps
end

deps_for("coverage>=3.4").each do |dep|
  puts dep
end

Then you can just use xargs on that output to pipe it to fpm -s python -t rpm

[jordansissel]

I definitely want this (and certainly many people ask about it).

Thinking out loud -

Having automatic dependencies allow for objects instead of strings could be useful. Having gem add a dependency of FPM::Package::Gem("rails", :version => "1.0") or something, would let us scan dependencies to additionally build. Making these valid dependencies in debs would just require a way to emit a package (gem, etc) object as a dependency string somehow.

Still needs investigation as to how to perfectly solve dependency conditions though, but it's maybe a start.

[mblair]

Yea, I wrote some code (internally) atop FPM's internal API to solve this very problem, amongst others. I ended up using RubyParser to parse Gemfiles, wrote a requirements.txt parser and wrote fetchers and caching for hg/git/bzr, the works. It got hairy and other projects got important, so I haven't touched it in half a year, but I can jot down some notes.

[anjo]

This seems to work halfway, based on the code above. Note that I'm not that familiar with ruby... the important thing for me is that the code also emmits the order of dependencies (i hope, havent tested it), so I can put those in chef in the correct order.

That I currently can't figure out is how to resolve "~>" like stuff. Also == should use the same version.

#!/usr/bin/ruby
require 'rubygems'
require 'fpm'
@deps = []
@level = 0
@indent = "                          "

def deps_for(input)
  pkg = FPM::Package::Gem.new
  name, op, version = input.split(/\s+/)
  pkg.input(name)

  puts "# #{@indent[0..@level]}Loading #{name} #{pkg.version} (#{input})"

  @deps.unshift pkg

  pkg.dependencies.each do |dep|
    name, op, version = dep.split(/\s+/)
    op = "==" if op == "="
    actual = "#{name} #{op} #{version}"
    @level = @level + 1
    deps_for(actual)
    @level = @level - 1
  end
end

deps_for(ARGV[0])

@deps.each do |dep|
  puts "fpm -s gem -t deb -v #{dep.version} #{dep.name}"
end

Output looks like:

redis:logstash-cli ak$ ruby deps.rb 'logstash-cli == 0.0.8'
#  Loading logstash-cli 0.0.8 (logstash-cli == 0.0.8)
#   Loading tire 0.5.8 (tire >= 0)
#    Loading rake 10.0.4 (rake >= 0)
#    Loading rest-client 1.6.7 (rest-client ~> 1.6)
#     Loading mime-types 1.23.0 (mime-types >= 1.16)
#    Loading multi_json 1.7.4 (multi_json ~> 1.3)
#    Loading activemodel 3.2.13 (activemodel >= 3.0)
#     Loading activesupport 3.2.13 (activesupport == 3.2.13)
#      Loading i18n 0.6.4 (i18n == 0.6.1)
#      Loading multi_json 1.7.4 (multi_json ~> 1.0)
#     Loading builder 3.2.0 (builder ~> 3.0.0)
#    Loading hashr 0.0.22 (hashr ~> 0.0.19)
#    Loading activesupport 3.2.13 (activesupport >= 0)
#     Loading i18n 0.6.4 (i18n == 0.6.1)
#     Loading multi_json 1.7.4 (multi_json ~> 1.0)
#    Loading ansi 1.4.3 (ansi >= 0)
#   Loading thor 0.18.1 (thor >= 0)
#   Loading amqp 1.0.2 (amqp >= 0)
#    Loading eventmachine 1.0.3 (eventmachine >= 0)
#    Loading amq-client 1.0.2 (amq-client ~> 1.0.2)
#     Loading eventmachine 1.0.3 (eventmachine >= 0)
#     Loading amq-protocol 1.6.0 (amq-protocol >= 1.2.0)
#    Loading amq-protocol 1.6.0 (amq-protocol >= 1.3.0)
#   Loading rack 1.5.2 (rack >= 0)
#   Loading yajl-ruby 1.1.0 (yajl-ruby >= 0)
#   Loading fastercsv 1.5.5 (fastercsv >= 0)
#   Loading json 1.8.0 (json >= 0)
fpm -s gem -t deb -v 1.8.0 json
fpm -s gem -t deb -v 1.5.5 fastercsv
fpm -s gem -t deb -v 1.1.0 yajl-ruby
fpm -s gem -t deb -v 1.5.2 rack
fpm -s gem -t deb -v 1.6.0 amq-protocol
fpm -s gem -t deb -v 1.6.0 amq-protocol
fpm -s gem -t deb -v 1.0.3 eventmachine
fpm -s gem -t deb -v 1.0.2 amq-client
fpm -s gem -t deb -v 1.0.3 eventmachine
fpm -s gem -t deb -v 1.0.2 amqp
fpm -s gem -t deb -v 0.18.1 thor
fpm -s gem -t deb -v 1.4.3 ansi
fpm -s gem -t deb -v 1.7.4 multi_json
fpm -s gem -t deb -v 0.6.4 i18n
fpm -s gem -t deb -v 3.2.13 activesupport
fpm -s gem -t deb -v 0.0.22 hashr
fpm -s gem -t deb -v 3.2.0 builder
fpm -s gem -t deb -v 1.7.4 multi_json
fpm -s gem -t deb -v 0.6.4 i18n
fpm -s gem -t deb -v 3.2.13 activesupport
fpm -s gem -t deb -v 3.2.13 activemodel
fpm -s gem -t deb -v 1.7.4 multi_json
fpm -s gem -t deb -v 1.23.0 mime-types
fpm -s gem -t deb -v 1.6.7 rest-client
fpm -s gem -t deb -v 10.0.4 rake
fpm -s gem -t deb -v 0.5.8 tire
fpm -s gem -t deb -v 0.0.8 logstash-cli

[anjo]

Jesus, what a mess. So some gems don't actually advertise their deps.

What would help:

• how do I get a specific version?
• how do I match the requested pattern to the repo?

There seems to be no way to "fpm --version" a specific version reliably. Also, I don't know how to actually get at ~> x.y like deps.

[blast-hardcheese]

Even if this can't be solved in a generic fashion, it would still be nice to have something that worked for simple cases. Have there been any improvements or thoughts in the past 9 months?

[jordansissel]

We have a plan to solve this generally; @torrancew and I have discussed ways to do it. We have a solution that'll work, just needs implementing.

[blast-hardcheese]

Great! Looking forward to it! I'm looking to recursively packaging things from gem, pypi, and npm into something that can be dropped into a self-hosted apt repo so I don't have to deal with a multitude of package managers. Thanks again for the hard work, really enjoyed seeing your talk yesterday.

[neoice]

it sounds like @blast-hardcheese wants to do the same kind of thing as me. all Ruby and Python at work and home gets installed via fpm (including fpm itself).

[guilhem]

@jordansissel any news about this ticket?

We are looking for a recursive python -> deb.

If you are thinking about a way, can you share it? Maybe we can work on it.

[vStone]

I'm not convinced this should be part of FPM. A far cleaner solution would be a contrib folder with scripts that wrap around fpm.

I use a small script that uses bundler to install all dependencies for a certain package and then loop package those in a loop.

[fruch]

+1,

I really think this is a killer feature for fpm, classically for python packages

[jordansissel]

@fruch It's still on my todo list to do this. I'll make time eventually.

[jordansissel]

I hacked on a proof-of-concept of this:

% bin/fpm -s gem -t deb --recursive rails
Created package {:path=>"rubygem-thread-safe_~> 0.1_all.deb"}
Created package {:path=>"rubygem-tzinfo_~> 1.1_all.deb"}
Created package {:path=>"rubygem-minitest_~> 5.1_all.deb"}
Created package {:path=>"rubygem-thread-safe_>= 0.3.4,~> 0.3_all.deb"}
Created package {:path=>"rubygem-activesupport_= 4.2.1_all.deb"}
Created package {:path=>"rubygem-rack_~> 1.6_all.deb"}
Created package {:path=>"rubygem-rack_>= 1.0_all.deb"}
Created package {:path=>"rubygem-rack-test_~> 0.6.2_all.deb"}
Created package {:path=>"rubygem-loofah_~> 2.0_all.deb"}
Created package {:path=>"rubygem-rails-html-sanitizer_>= 1.0.1,~> 1.0_all.deb"}
Created package {:path=>"rubygem-activesupport_< 5.0,>= 4.2.0.beta_all.deb"}
Created package {:path=>"rubygem-activesupport_>= 4.2.0.alpha_all.deb"}
Created package {:path=>"rubygem-rails-deprecated-sanitizer_>= 1.0.1_all.deb"}
Created package {:path=>"rubygem-rails-dom-testing_>= 1.0.5,~> 1.0_all.deb"}
Created package {:path=>"rubygem-builder_~> 3.1_all.deb"}
Created package {:path=>"rubygem-erubis_~> 2.7.0_all.deb"}
Created package {:path=>"rubygem-actionview_= 4.2.1_all.deb"}
Created package {:path=>"rubygem-actionpack_= 4.2.1_all.deb"}
...

The file names are funky because it's a bad PoC (the "version" is "= 4.2.1" (equal, space, 4.2.1) heh.

This shows what I'm looking to implement, though. It also helped me figure out where to expect some pain :P

[anjo]

Without thinking too hard about this... say you generate two packages, one today and one in a month.

If you build the names like you do above, then the packages might have very different content with the same name, depending in if matching sub versions have been created in the meantime, or wouldn't they?

[jordansissel]

If you build the names like you do above

The versions in the packages like rubygem-activesupport_< 5.0,>= 4.2.0.beta_all.deb are a bug, not a feature, in this poc. It should have a single specific version, not the < 5.0,>= 4.2.0.beta text. It's a bug - sorry for the confusion :P This POC is totally incomplete but shows what I'm expecting the interface for users to be and the rough behavior (--recursive flag, and fpm building tons of packages based on the dependencies)

[fruch]

I took a look at poc/recursive-packaging, and it's seem quite usable already, (I didn't took it yet to a test drive, will try soon)

[fruch]

It's not yet working :), at least for python packages.

(venv)ifruchte@hyrcanus:~/pyrecman$ fpm -s python -t rpm --recursive setup.py Working on dependency {:name=>"python-motor", :conditions=>"= 0.4", :level=>:warn} Process failed: easy_install failed (exit code 1). Full command was:["easy_install", "-i", "https://pypi.python.org/simple", "--editable", "-U", "--build-directory", "/tmp/user/1002/package-python-build20150509-29167-dhg6jc/python-motor", "python-motor=== 0.4"] {:level=>:error} Working on dependency {:name=>"python-tornado", :conditions=>"= 4.1", :level=>:warn} Process failed: easy_install failed (exit code 1). Full command was:["easy_install", "-i", "https://pypi.python.org/simple", "--editable", "-U", "--build-directory", "/tmp/user/1002/package-python-build20150509-29167-g5l94g/python-tornado", "python-tornado=== 4.1"] {:level=>:error} Working on dependency {:name=>"python-lxml", :conditions=>"= 3.4.2", :level=>:warn} Process failed: easy_install failed (exit code 1). Full command was:["easy_install", "-i", "https://pypi.python.org/simple", "--editable", "-U", "--build-directory", "/tmp/user/1002/package-python-build20150509-29167-1i5b9pg/python-lxml", "python-lxml=== 3.4.2"] {:level=>:error} Working on dependency {:name=>"python-tornado-celery", :conditions=>"= 0.3.4", :level=>:warn} Process failed: easy_install failed (exit code 1). Full command was:["easy_install", "-i", "https://pypi.python.org/simple", "--editable", "-U", "--build-directory", "/tmp/user/1002/package-python-build20150509-29167-16njoiq/python-tornado-celery", "python-tornado-celery=== 0.3.4"] {:level=>:error} Working on dependency {:name=>"python-requests", :conditions=>"= 2.6.0", :level=>:warn} Process failed: easy_install failed (exit code 1). Full command was:["easy_install", "-i", "https://pypi.python.org/simple", "--editable", "-U", "--build-directory", "/tmp/user/1002/package-python-build20150509-29167-xviekl/python-requests", "python-requests=== 2.6.0"] {:level=>:error} no value for epoch is set, defaulting to nil {:level=>:warn} no value for epoch is set, defaulting to nil {:level=>:warn} Created package {:path=>"python-pyrecman-1.0.0.post0.dev1+g58892c8-1.noarch.rpm"}

[fruch]

@jordansissel any news regarding this ?

[jordansissel]

Not much beyond the PoC I mentioned above (which I don't know where the code is anymore, oops, probably in a branch somewhere).

I really want it implemented, but I haven't thought of a good way to solve this problem, yet.

[fruch]

anthoer place I need building RPM for lots of python packages, any new on this one ?

[Comradin]

Hi, that feature would be a killer. Am struggling with missing dependencies at the moment. Any progress on this so far?

[morph027]

found https://github.com/wimglenn/johnnydep which gives me a (pinned) list to recurse on.

DEPS=$(johnnydep "${PACKAGE}"=="${VERSION}" --output-format pinned)
for DEP in $DEPS; do fpm -s python -t deb -v "${DEP##*==}" "${DEP%%==*}"; done