crohr / pkgr

Package any app into deb or rpm packages, using heroku buildpacks
http://crohr.me/pkgr/
MIT License
591 stars 66 forks source link

Fails to build ruby projects #146

Closed ccouzens closed 3 years ago

ccouzens commented 6 years ago

Hi,

Since Friday afternoon, I've been unable to build ruby apps using pkgr.

I've made a stub app to demonstrate the problem. Effectively, this app is nothing more than a Gemfile and Gemfile.lock. The only gem is pkgr@1.6.0. The version of Ruby is 2.5.1, but I've seen the same problem with 2.4.4, 2.3.7 and 2.3.0.

bundle exec pkgr package . --verbose --debug

This gives the following log output:

[2018-04-30T11:41:02+01:00] DEBUG: sh(git describe --tags --abbrev=0)
[2018-04-30T11:41:02+01:00] DEBUG: Initializing builder with the following config: #<Pkgr::Config verbose=true, debug=true, buildpacks_cache_dir="/home/chris/.pkgr/buildpacks", architecture="x86_64", category="none", iteration="20180430114102", dependencies=[], build_dependencies=[], auto=false, clean=true, edge=true, env=[], verify=true, data_dir="/home/chris/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/pkgr-1.6.0/data", version="1.0.0", name="pkgr-demo">
[2018-04-30T11:41:02+01:00] DEBUG: sh(tar xzf /tmp/pkgr-tarball20180430-1770-1nayjax.tar.gz -C /tmp/d20180430-1770-md4n32/opt/pkgr-demo)
[2018-04-30T11:41:02+01:00] DEBUG: sh(rpm -qa 'curl' | grep 'curl' > /dev/null 2>&1)
[2018-04-30T11:41:03+01:00] DEBUG: sh(rpm -qa 'gcc' | grep 'gcc' > /dev/null 2>&1)
[2018-04-30T11:41:05+01:00] DEBUG: sh(rpm -qa 'gcc-c++' | grep 'gcc-c++' > /dev/null 2>&1)
[2018-04-30T11:41:06+01:00] DEBUG: sh(rpm -qa 'kernel-devel' | grep 'kernel-devel' > /dev/null 2>&1)
[2018-04-30T11:41:07+01:00] DEBUG: sh(rpm -qa 'openssl-devel' | grep 'openssl-devel' > /dev/null 2>&1)
[2018-04-30T11:41:08+01:00] DEBUG: sh(rpm -qa 'readline-devel' | grep 'readline-devel' > /dev/null 2>&1)
[2018-04-30T11:41:09+01:00] DEBUG: sh(rpm -qa 'libxml2-devel' | grep 'libxml2-devel' > /dev/null 2>&1)
[2018-04-30T11:41:10+01:00] DEBUG: sh(rpm -qa 'libxslt-devel' | grep 'libxslt-devel' > /dev/null 2>&1)
[2018-04-30T11:41:12+01:00] DEBUG: sh(rpm -qa 'libevent-devel' | grep 'libevent-devel' > /dev/null 2>&1)
[2018-04-30T11:41:13+01:00] DEBUG: sh(rpm -qa 'postgresql-devel' | grep 'postgresql-devel' > /dev/null 2>&1)
[2018-04-30T11:41:14+01:00] DEBUG: sh(rpm -qa 'mysql-devel' | grep 'mysql-devel' > /dev/null 2>&1)
[2018-04-30T11:41:15+01:00] DEBUG: sh(rpm -qa 'sqlite-devel' | grep 'sqlite-devel' > /dev/null 2>&1)
[2018-04-30T11:41:16+01:00] WARN: Missing build dependencies detected. Run the following to fix: sudo yum -q check-update ; sudo yum install -y "mysql-devel"
[2018-04-30T11:41:16+01:00] WARN: Can't create "/opt/pkgr-demo", which may be needed by some buildpacks.
[2018-04-30T11:41:16+01:00] DEBUG: sh(git fetch origin && ( git reset --hard v164-1 || git reset --hard origin/v164-1 ) && chmod -f +x bin/detect && chmod -f +x bin/compile && chmod -f +x bin/release)
[2018-04-30T11:41:17+01:00] DEBUG: sh(find . -type f -not -path '*/.git/*' -print0 | xargs -0 perl -pi -e s,/app/,/opt/pkgr-demo/,g)
[2018-04-30T11:41:17+01:00] DEBUG: sh(env -i PATH="$PATH" TARGET="el:7" BUILDPACK_NODE_VERSION="0.6.8" /home/chris/.pkgr/buildpacks/builtin/719e4e8cb9227b4e7f6a118d958174bd6045fa93/heroku-buildpack-ruby/bin/detect "/tmp/d20180430-1770-md4n32/opt/pkgr-demo")
-----> Ruby app
[2018-04-30T11:41:17+01:00] DEBUG: Running "env -i PATH=\"$PATH\" TARGET=\"el:7\" BUILDPACK_NODE_VERSION=\"0.6.8\" /home/chris/.pkgr/buildpacks/builtin/719e4e8cb9227b4e7f6a118d958174bd6045fa93/heroku-buildpack-ruby/bin/compile \"/tmp/d20180430-1770-md4n32/opt/pkgr-demo\" \"/tmp/d20180430-1770-md4n32/opt/pkgr-demo/.git/cache\" \"/tmp/d20180430-1770-117oyih\" "
-----> Compiling Ruby
sh: command substitution: line 0: unexpected EOF while looking for matching `''
sh: command substitution: line 1: syntax error: unexpected end of file
 !
 !     There was an error parsing your Gemfile, we cannot continue
 !     /tmp/d20180430-1922-1bkp5d3/bundler-1.15.1/bin/bundle:22:in `<main>': undefined method `activate_bin_path' for Gem:Module (NoMethodError)
 !
[2018-04-30T11:41:25+01:00] DEBUG: RuntimeError : compile failed
[2018-04-30T11:41:25+01:00] DEBUG: /home/chris/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/pkgr-1.6.0/lib/pkgr/buildpack.rb:49:in `block in compile'
[2018-04-30T11:41:25+01:00] DEBUG: /home/chris/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/pkgr-1.6.0/lib/pkgr/buildpack.rb:42:in `chdir'
[2018-04-30T11:41:25+01:00] DEBUG: /home/chris/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/pkgr-1.6.0/lib/pkgr/buildpack.rb:42:in `compile'
[2018-04-30T11:41:25+01:00] DEBUG: /home/chris/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/pkgr-1.6.0/lib/pkgr/builder.rb:123:in `block in compile'
[2018-04-30T11:41:25+01:00] DEBUG: /home/chris/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/pkgr-1.6.0/lib/pkgr/builder.rb:121:in `each'
[2018-04-30T11:41:25+01:00] DEBUG: /home/chris/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/pkgr-1.6.0/lib/pkgr/builder.rb:121:in `compile'
[2018-04-30T11:41:25+01:00] DEBUG: /home/chris/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/pkgr-1.6.0/lib/pkgr/builder.rb:28:in `call'
[2018-04-30T11:41:25+01:00] DEBUG: /home/chris/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/pkgr-1.6.0/lib/pkgr/dispatcher.rb:33:in `call'
[2018-04-30T11:41:25+01:00] DEBUG: /home/chris/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/pkgr-1.6.0/lib/pkgr/cli.rb:170:in `package'
[2018-04-30T11:41:25+01:00] DEBUG: /home/chris/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/thor-0.20.0/lib/thor/command.rb:27:in `run'
[2018-04-30T11:41:25+01:00] DEBUG: /home/chris/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/thor-0.20.0/lib/thor/invocation.rb:126:in `invoke_command'
[2018-04-30T11:41:25+01:00] DEBUG: /home/chris/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/thor-0.20.0/lib/thor.rb:387:in `dispatch'
[2018-04-30T11:41:25+01:00] DEBUG: /home/chris/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/thor-0.20.0/lib/thor/base.rb:466:in `start'
[2018-04-30T11:41:25+01:00] DEBUG: /home/chris/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/pkgr-1.6.0/bin/pkgr:8:in `<top (required)>'
[2018-04-30T11:41:25+01:00] DEBUG: /home/chris/.rbenv/versions/2.5.1/bin/pkgr:23:in `load'
[2018-04-30T11:41:25+01:00] DEBUG: /home/chris/.rbenv/versions/2.5.1/bin/pkgr:23:in `<top (required)>'
[2018-04-30T11:41:25+01:00] DEBUG: /home/chris/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.1/lib/bundler/cli/exec.rb:75:in `load'
[2018-04-30T11:41:25+01:00] DEBUG: /home/chris/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.1/lib/bundler/cli/exec.rb:75:in `kernel_load'
[2018-04-30T11:41:25+01:00] DEBUG: /home/chris/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.1/lib/bundler/cli/exec.rb:28:in `run'
[2018-04-30T11:41:25+01:00] DEBUG: /home/chris/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.1/lib/bundler/cli.rb:424:in `exec'
[2018-04-30T11:41:25+01:00] DEBUG: /home/chris/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.1/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
[2018-04-30T11:41:25+01:00] DEBUG: /home/chris/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.1/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
[2018-04-30T11:41:25+01:00] DEBUG: /home/chris/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.1/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
[2018-04-30T11:41:25+01:00] DEBUG: /home/chris/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.1/lib/bundler/cli.rb:27:in `dispatch'
[2018-04-30T11:41:25+01:00] DEBUG: /home/chris/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.1/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
[2018-04-30T11:41:25+01:00] DEBUG: /home/chris/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.1/lib/bundler/cli.rb:18:in `start'
[2018-04-30T11:41:25+01:00] DEBUG: /home/chris/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.1/exe/bundle:30:in `block in <top (required)>'
[2018-04-30T11:41:25+01:00] DEBUG: /home/chris/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.1/lib/bundler/friendly_errors.rb:122:in `with_friendly_errors'
[2018-04-30T11:41:25+01:00] DEBUG: /home/chris/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.1/exe/bundle:22:in `<top (required)>'
[2018-04-30T11:41:25+01:00] DEBUG: /home/chris/.rbenv/versions/2.5.1/bin/bundle:23:in `load'
[2018-04-30T11:41:25+01:00] DEBUG: /home/chris/.rbenv/versions/2.5.1/bin/bundle:23:in `<main>'
! ERROR: compile failed

I'll update the issue with any insights I find.

Kind regards,

Chris

Jed-Giblin commented 6 years ago

Having the same issue and investigating as well on ruby2.4.4

ccouzens commented 6 years ago

My colleague @jimeh got this working by modifying the buildpack.

To use this, either add a .pkgr.yml file with these contents:

buildpack: https://github.com/jimeh/heroku-buildpack-ruby.git

or run pkgr with the buildpack flag:

pkgr package . --buildpack=https://github.com/jimeh/heroku-buildpack-ruby.git

I expect the solution will be to merge these changes into pkgr's ruby buildpack.

Jed-Giblin commented 6 years ago

@ccouzens @jimeh thanks very much.

ccouzens commented 6 years ago

@Jed-Giblin note that this doesn't solve it for the demo app. Depending on your version of ruby this may or may not work.

Jed-Giblin commented 6 years ago

@ccouzens With a combination of the changes by @jimeh and @obfuscoder in #142 I was able to get this resolved.

evsav commented 6 years ago

@ccouzens @jimeh I applied your fix, but no luck unfortunately. I'm packaging a Ruby on Rails project (ruby 2.1.5, rails 4.1.9 in a docker container (CI pipeline) and getting the following error:

/root/.pkgr/buildpacks/custom/872254ccf5ea3976c818aa8f736a5a7fe72775aa/heroku-buildpack-ruby/lib/language_pack/shell_helpers.rb:49:inrun!': Command: 'set -o pipefail; curl -L --get --fail --retry 3 buildcurl.com -d recipe=ruby -d version=2.3.4 -d target=$TARGET -s -o - | tar zxf - ' failed unexpectedly: (LanguagePack::Fetcher::FetchError)

gzip: stdin: unexpected end of file tar: Child returned status 1 tar: Error is not recoverable: exiting now from /root/.pkgr/buildpacks/custom/872254ccf5ea3976c818aa8f736a5a7fe72775aa/heroku-buildpack-ruby/lib/language_pack/fetcher.rb:24:in fetch_untar' from /root/.pkgr/buildpacks/custom/872254ccf5ea3976c818aa8f736a5a7fe72775aa/heroku-buildpack-ruby/lib/language_pack/installers/heroku_ruby_installer.rb:22:inblock in fetch_unpack' from /root/.pkgr/buildpacks/custom/872254ccf5ea3976c818aa8f736a5a7fe72775aa/heroku-buildpack-ruby/lib/language_pack/installers/heroku_ruby_installer.rb:16:in chdir' from /root/.pkgr/buildpacks/custom/872254ccf5ea3976c818aa8f736a5a7fe72775aa/heroku-buildpack-ruby/lib/language_pack/installers/heroku_ruby_installer.rb:16:infetch_unpack' from /root/.pkgr/buildpacks/custom/872254ccf5ea3976c818aa8f736a5a7fe72775aa/heroku-buildpack-ruby/bin/support/download_ruby:14:in <main>' /root/.pkgr/buildpacks/custom/872254ccf5ea3976c818aa8f736a5a7fe72775aa/heroku-buildpack-ruby/bin/compile: line 12: /tmp/tmp.t4BfRLB4Br/bin/ruby: No such file or directory /root/.pkgr/buildpacks/custom/872254ccf5ea3976c818aa8f736a5a7fe72775aa/heroku-buildpack-ruby/bin/compile: line 19: /tmp/tmp.t4BfRLB4Br/bin/ruby: No such file or directory ! ERROR: compile failed

Any suggestions?

Jed-Giblin commented 6 years ago

@evsav buildcurl.com seems to be down

http://www.isitdownrightnow.com/buildcurl.com.html

Which is why your package fails. I was able to download the buildcurl service from https://packager.io/gh/crohr/buildcurl, set it up on a network host and add an environmental for BUILDCURL_URL and that got it working

evsav commented 6 years ago

@Jed-Giblin thanks for the prompt reply! I'll try doing the same

evsav commented 6 years ago

@Jed-Giblin worked. I had to convert the buildcurl.deb into .rpm to install it in our centos7 container. Thanks a lot

jimeh commented 6 years ago

I took a different approach than @Jed-Giblin to buildcurl.com being down. Since buildcurl seems to just be a proxy to barebuild.com I decided to just cache the Ruby binary archives I care about here: https://github.com/jimeh/heroku-buildpack-ruby-binaries

Then I customized my buildpack to go off to my cache for the few things I have in the cache. Initially the cached binaries were extracted from existing packages I had at hand, but when buildcurl.com came back I replaced them with archives downloaded directly from buildcurl.com.

I have a PR open on my buildpack which I intend to merge tomorrow morning: https://github.com/jimeh/heroku-buildpack-ruby/pull/1

NOTE: Use my buildpack and cached binaries at your own risk. It's all geared around what I personally need to get some of our legacy projects to build, so I can't give any kind of guarantee it works or will keep working for anyone else.

That said, I intend to spend some time looking at this properly when the smoke has settled, and see if I can get some of the fixes merged upstream into https://github.com/pkgr/heroku-buildpack-ruby

ccouzens commented 6 years ago

The following commit happened 11 days ago which updates the buildpack for ruby fc83c0c62a9f37f9f858011dc36d08b2543eb5f4.

This isn't yet in any released version of pkgr, but it is in the master branch.

By making my Gemfile

gem 'pkgr', github: 'crohr/pkgr

I can package my test app. I'll investigate further tomorrow, but it looks like the build issues may now be officially solved.

crohr commented 3 years ago

I believe the issue has been fixed since a while now, sorry for not answering at the time. Let me know if you still experience issues.