Failed to compile charlock_holmes 0.7.3 in heroku cedar-14 buildpack-multi

[donnykurnia]

I followed http://tooky.co.uk/using-charklock_holmes-on-heroku/ with little modification for cedar-14 paths that use Ubuntu Trusty.

My last deploy on 2015/01/11 to heroku successed. Today I try to deploy the exact same code to another heroku apps, but failed.

Here is the error messages that I got:

          Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.

          /tmp/build_22b7afabc74cec876e6a4a369b12c2f5/vendor/ruby-2.0.0/bin/ruby extconf.rb --with-icu-lib=/tmp/build_22b7afabc74cec876e6a4a369b12c2f5/.apt/usr/lib/x86_64-linux-gnu --with-icu-include=/tmp/build_22b7afabc74cec876e6a4a369b12c2f5/.apt/usr/include/x86_64-linux-gnu --with-ldflags='-fPIC' --with-cflags='-fPIC' --with-cxxflags='-fPIC'
          checking for main() in -licui18n... yes
          checking for main() in -licui18n... yes
          checking for unicode/ucnv.h... yes
          checking for main() in -lz... yes
          checking for main() in -licuuc... yes
          checking for main() in -licudata... yes
          creating Makefile

          make "DESTDIR="
          compiling encoding_detector.c
          In file included from encoding_detector.c:2:0:
          common.h:14:14: warning: ‘charlock_new_enc_str’ defined but not used [-Wunused-function]
          static VALUE charlock_new_enc_str(const char *str, size_t len, void *encoding)
          ^
          compiling ext.c
          In file included from ext.c:1:0:
          common.h:14:14: warning: ‘charlock_new_enc_str’ defined but not used [-Wunused-function]
          static VALUE charlock_new_enc_str(const char *str, size_t len, void *encoding)
          ^
          common.h:23:14: warning: ‘charlock_new_str’ defined but not used [-Wunused-function]
          static VALUE charlock_new_str(const char *str, size_t len)
          ^
          common.h:32:14: warning: ‘charlock_new_str2’ defined but not used [-Wunused-function]
          static VALUE charlock_new_str2(const char *str)
          ^
          compiling converter.c
          In file included from converter.c:2:0:
          common.h:23:14: warning: ‘charlock_new_str’ defined but not used [-Wunused-function]
          static VALUE charlock_new_str(const char *str, size_t len)
          ^
          common.h:32:14: warning: ‘charlock_new_str2’ defined but not used [-Wunused-function]
          static VALUE charlock_new_str2(const char *str)
          ^
          compiling transliterator.cpp
          linking shared-object charlock_holmes/charlock_holmes.so
          /usr/bin/ld: /tmp/build_22b7afabc74cec876e6a4a369b12c2f5/.apt/usr/lib/x86_64-linux-gnu/libicui18n.a(smpdtfmt.ao): relocation R_X86_64_PC32 against symbol `_ZN6icu_5216SimpleDateFormat22isAfterNonNumericFieldERKNS_13UnicodeStringEi' can not be used when making a shared object; recompile with -fPIC
          /usr/bin/ld: final link failed: Bad value
          collect2: error: ld returned 1 exit status
          make: *** [charlock_holmes.so] Error 1

          Gem files will remain installed in /tmp/build_22b7afabc74cec876e6a4a369b12c2f5/vendor/bundle/ruby/2.0.0/gems/charlock_holmes-0.7.3 for inspection.
          Results logged to /tmp/build_22b7afabc74cec876e6a4a369b12c2f5/vendor/bundle/ruby/2.0.0/gems/charlock_holmes-0.7.3/ext/charlock_holmes/gem_make.out
          Installing nested_form 0.3.2
          Installing newrelic_rpm 3.8.1.221
          An error occurred while installing charlock_holmes (0.7.3), and Bundler cannot
          continue.
          Make sure that `gem install charlock_holmes -v '0.7.3'` succeeds before
          bundling.
    !
    !     Failed to install gems via Bundler.
    !

    !     Push rejected, failed to compile Multipack app

I have try various combination, but still got the same errors.

.buildpacks file:

https://github.com/ddollar/heroku-buildpack-apt
https://github.com/timolehto/heroku-bundle-config
https://github.com/heroku/heroku-buildpack-ruby#v129

.heroku-bundle/config file:

---
BUNDLE_FROZEN: '1'
BUNDLE_PATH: vendor/bundle
BUNDLE_BIN: vendor/bundle/bin
BUNDLE_JOBS: 4
BUNDLE_WITHOUT: development:test
BUNDLE_DISABLE_SHARED_GEMS: '1'
BUNDLE_BUILD__CHARLOCK_HOLMES: --with-icu-lib=/app/.apt/usr/lib/x86_64-linux-gnu --with-icu-include=/app/.apt/usr/include/x86_64-linux-gnu --with-ldflags="$LDFLAGS -fPIC" --with-cflags="$CFLAGS -fPIC" --with-cxxflags="$CXXFLAGS -fPIC"

In the previous app, I just use BUNDLE_BUILD__CHARLOCK_HOLMES: --with-icu-lib=/app/.apt/usr/lib/x86_64-linux-gnu --with-icu-include=/app/.apt/usr/include/x86_64-linux-gnu.

I have try googling with various keywords, only to find references about gentoo and old version of charlock_holmes. I have check and only see that heroku-buildpack-ruby recently updated to v130, that's why I try to use v129, but it's show the same error.

My Gemfile:

source 'https://rubygems.org'
ruby '2.0.0'

gem 'puma'
gem "rack-timeout"

gem 'rails', '4.0.5'
gem 'sass-rails', '~> 4.0.0'
gem 'uglifier', '>= 1.3.0'
gem 'coffee-rails', '~> 4.0.0'

gem 'mongoid'
gem 'devise'
gem 'jquery-rails'
gem "twitter-bootstrap-rails"
gem 'jbuilder', '~> 1.2'
gem 'kaminari'
gem 'bootstrap-modal-rails', github: 'vicentereig/bootstrap-modal-rails'
gem 'hashie'
gem 'descriptive_statistics'

# db related
# gem 'acts_as_list_mongoid'
gem 'public_activity'
gem 'mongoid-tree', require: 'mongoid/tree'

# Exception handling
gem 'honeybadger'

# Encoding handling
# gem 'charlock_holmes', github: 'brianmario/charlock_holmes', branch: "bundle-icu"
gem 'charlock_holmes'

# Assets
gem 'jquery-ui-rails'
gem "selectize-rails"
gem 'jquery-datatables-rails', '~> 2.2.3'
gem 'nprogress-rails'
gem 'tabulous'

# forms
gem "simple_form", "~> 3.0.0"
gem "country_select", "~> 1.2.0"
gem "nested_form"
gem 'carrierwave'
gem 'carrierwave-mongoid', :require => 'carrierwave/mongoid'
gem 'mongoid-grid_fs', github: 'ahoward/mongoid-grid_fs'
gem 'open_uri_redirections'
gem 'remotipart', '~> 1.2.1'

# View generation
gem 'builder', '~> 3.1'
gem 'prawn'
gem 'prawn-html5', github: 'cs/prawn-html5'
gem 'wisepdf'
gem 'pandoc-ruby'
gem 'htmltoword'

# External apis
gem 'nokogiri'

# Heroku related
group :production do
  gem 'rails_12factor'
  gem "wkhtmltopdf-heroku", "~> 1.0.0"
end

group :development, :test do
  gem 'hirb'
  gem "dotenv-rails"
  gem 'rspec-rails', :require => false
  gem 'capybara-webkit'
  gem 'capybara'
  gem 'cucumber-rails', '~> 1.3', :require => false
  # database_cleaner is not required, but highly recommended
  gem 'database_cleaner'
  gem 'quiet_assets'
  gem 'factory_girl_rails'
  gem 'faker'
  gem 'email_spec'
  gem 'guard-cucumber'
  gem 'guard-rspec'
  gem 'guard-rails'
  gem 'guard-spork'
  gem 'guard-sidekiq'
  gem "rb-readline", "~> 0.5.0"
  gem 'rb-fsevent'
  gem 'growl'
  gem "spork", github: "sporkrb/spork"
  gem 'spork-rails', github: 'sporkrb/spork-rails'
  gem 'mongoid-rspec'
  gem 'fakeweb'
  gem 'commands'
end

group :development do
  gem "bullet"
  gem 'better_errors'
  gem 'binding_of_caller'
  gem 'meta_request'
  gem "foreman"
end

# monitoring
gem 'newrelic_rpm'

# background job
gem 'sidekiq', '~> 3'
gem 'sidekiq-failures'
gem 'sinatra', '>= 1.3.0', :require => nil

# bug tracking
gem "hirefire-resource"
gem 'rollbar'
gem 'raygun4ruby'

[donnykurnia]

Additional info, the content of Aptfile:

libicu-dev

[donnykurnia]

Here is the result when I use master branch of charlock_holmes from github:

          Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.

          /tmp/build_c80c1c65b90527246de4f9aa16bdb833/vendor/ruby-2.0.0/bin/ruby extconf.rb --with-icu-lib=/tmp/build_c80c1c65b90527246de4f9aa16bdb833/.apt/usr/lib/x86_64-linux-gnu --with-icu-include=/tmp/build_c80c1c65b90527246de4f9aa16bdb833/.apt/usr/include/x86_64-linux-gnu --with-cflags="-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wall -fno-strict-aliasing -fPIC"
          checking for main() in -licui18n... yes
          checking for main() in -licui18n... yes
          checking for unicode/ucnv.h... yes
          checking for main() in -lz... yes
          checking for main() in -licuuc... yes
          checking for main() in -licudata... yes
          creating Makefile

          make "DESTDIR="
          compiling converter.c
          compiling encoding_detector.c
          compiling ext.c
          compiling transliterator.cpp
          linking shared-object charlock_holmes/charlock_holmes.so
          /usr/bin/ld: /tmp/build_c80c1c65b90527246de4f9aa16bdb833/.apt/usr/lib/x86_64-linux-gnu/libicui18n.a(smpdtfmt.ao): relocation R_X86_64_PC32 against symbol `_ZN6icu_5216SimpleDateFormat22isAfterNonNumericFieldERKNS_13UnicodeStringEi' can not be used when making a shared object; recompile with -fPIC
          /usr/bin/ld: final link failed: Bad value
          collect2: error: ld returned 1 exit status
          make: *** [charlock_holmes.so] Error 1

          Gem files will remain installed in /tmp/build_c80c1c65b90527246de4f9aa16bdb833/vendor/bundle/ruby/2.0.0/bundler/gems/charlock_holmes-31307f781d76 for inspection.
          Results logged to /tmp/build_c80c1c65b90527246de4f9aa16bdb833/vendor/bundle/ruby/2.0.0/bundler/gems/charlock_holmes-31307f781d76/ext/charlock_holmes/gem_make.out
          Installing orm_adapter 0.5.0
          Installing colored 1.2
          An error occurred while installing charlock_holmes (0.7.3), and Bundler cannot
          continue.
          Make sure that `gem install charlock_holmes -v '0.7.3'` succeeds before
          bundling.
    !
    !     Failed to install gems via Bundler.
    !

    !     Push rejected, failed to compile Multipack app

[donnykurnia]

I'm ended using https://github.com/webandtech/heroku-buildpack-ruby and the gem installed just fine.

I wonder what cause the above error. It's definitely the same heroku app, same gcc, same cflags. The only different is the library source, apt and tar.gz. In my other apps that deployed on December 2014, it also use the apt and all went just fine.

[jayzes]

I'm running into the same issue, for what it's worth. Still haven't found a solution, but at least you're not the only one seeing it.

[donnykurnia]

Hi @jayzes,

Previously I have problems with PATH. First time deploy will success, but next deploy will failed. This bugs have been fixed in the latest heroku-buildpack-apt. I'm ended up forking webandtech into my github and will try to kept it up to date with heroku buildpack release: https://github.com/donnykurnia/heroku-buildpack-ruby

[jayzes]

@donnykurnia which heroku-buildpack-apt are you using to fix it? doesn't look like the main repo has been touched in a few months.

[jayzes]

for what it's worth, Ubuntu upgraded the libicu version in Trusty on 12/31 from 4.8 to 5.2 (https://launchpad.net/ubuntu/trusty/amd64/libicu-dev). Seems like that's the difference. I'm not sure if it's a problem with the underlying libicu version or the package.

[hammady]

I am having the same problem. My app worked on cedar but when I tried upgrading to cedar-14 it started to happen, can't build charlock_holmes native extension.

[jayzes]

@hammady for what it's worth, pinning the 4.8 debs in my Aptfile did the trick for me on cedar-14. Just put these two lines in there:

http://launchpadlibrarian.net/155496722/libicu48_4.8.1.1-13%2Bnmu1ubuntu1_amd64.deb
http://launchpadlibrarian.net/155496724/libicu-dev_4.8.1.1-13%2Bnmu1ubuntu1_amd64.deb

[hammady]

I removed the gem totally and used icu4j inside a thrift server which is already part of my app. Charlock is really too much pain and needs a lot of hacks in buildpacks. I can post the Java wrapper code if anyone is interested. On Thu, Apr 30, 2015 at 4:35 PM Jay Zeschin notifications@github.com wrote:

@hammady https://github.com/hammady for what it's worth, pinning the 4.8 debs in my Aptfile did the trick for me on cedar-14. Just put these two lines in there:

http://launchpadlibrarian.net/155496722/libicu48_4.8.1.1-13%2Bnmu1ubuntu1_amd64.debhttp://launchpadlibrarian.net/155496724/libicu-dev_4.8.1.1-13%2Bnmu1ubuntu1_amd64.deb

— Reply to this email directly or view it on GitHub https://github.com/brianmario/charlock_holmes/issues/91#issuecomment-97791543 .

[brodock]

are there any fix for this problem?

[msanchez01]

sudp apt-get install libicu-dev fixed my problem.

[ckdileep]

I followed the steps described here - https://github.com/brianmario/charlock_holmes/wiki/Installing-charlock-holmes-and-libicu-dev-on-heroku The Aptfile package links mentioned don't work. It works when "ubuntu0.5" is changed to "ubuntu0.6", for example - when http://mirrors.kernel.org/ubuntu/pool/main/i/icu/libicu48_4.8.1.1-3ubuntu0.5_amd64.deb is replaced with http://mirrors.kernel.org/ubuntu/pool/main/i/icu/libicu48_4.8.1.1-3ubuntu0.6_amd64.deb. However on deploying to heroku, I get this error - LoadError: libicuuc.so.48: cannot open shared object file: No such file or directory - /tmp/build_6c5bab6b6b98520124fe77ce277ed67d/vendor/bundle/ruby/1.9.1/gems/charlock_holmes-0.7.3/lib/charlock_holmes/charlock_holmes.so Would appreciate any help on this!

[donnykurnia]

I kept an updated fork of heroku-buildpack-ruby with Icu4C installed from binaries https://s3.amazonaws.com/frederick-heroku-binaries/icu4c-52.1.0.tar.gz

You can use this build pack: https://github.com/donnykurnia/heroku-buildpack-ruby I'll try to kept it updated with the new releases (last update is v140).

[holoiii]

Not sure if this will work for anybody else, but I tried @ckdileep's suggestion and it worked for me.

[dankozlowski]

I've tried all recommendations here with no luck, and https://github.com/brianmario/charlock_holmes/wiki/Installing-charlock-holmes-and-libicu-dev-on-heroku doesn't work either. Anyone else had success?

[dankozlowski]

I'm getting this:

remote:        Gem::Ext::BuildError: ERROR: Failed to build gem native extension.
remote:
remote:        /tmp/build_a5d8d3d2179fba79edba9bb1a02ee3d2/vendor/ruby-2.2.4/bin/ruby -r
remote:        ./siteconf20160922-174-gbufcm.rb extconf.rb /app/icu
remote:        checking for main() in -licui18n... no
remote:        checking for main() in -licui18n... no
remote:
remote:
remote:        ***************************************************************************************
remote:        *********** icu required (brew install icu4c or apt-get install libicu-dev)
remote:        ***********
remote:        ***************************************************************************************
remote:        *** extconf.rb failed ***
remote:        Could not create Makefile due to some reason, probably lack of necessary
remote:        libraries and/or headers.  Check the mkmf.log file for more details.  You may
remote:        need configuration options.
remote:
remote:        Provided configuration options:
remote:        --with-opt-dir
remote:        --without-opt-dir
remote:        --with-opt-include
remote:        --without-opt-include=${opt-dir}/include
remote:        --with-opt-lib
remote:        --without-opt-lib=${opt-dir}/lib
remote:        --with-make-prog
remote:        --without-make-prog
remote:        --srcdir=.
remote:        --curdir
remote:        --ruby=/tmp/build_a5d8d3d2179fba79edba9bb1a02ee3d2/vendor/ruby-2.2.4/bin/$(RUBY_BASE_NAME)
remote:        --with-icu-dir
remote:        --without-icu-dir
remote:        --with-icu-include
remote:        --without-icu-include=${icu-dir}/include
remote:        --with-icu-lib
remote:        --without-icu-lib=${icu-dir}/lib
remote:        --with-icui18nlib
remote:        --without-icui18nlib
remote:        --with-icui18nlib
remote:        --without-icui18nlib
remote:
remote:        extconf failed, exit code 1

[donnykurnia]

@dankozlowski how about using my buildpack: https://github.com/donnykurnia/heroku-buildpack-ruby

I kept it up-to-date with heroku releases, current version is 146.

[jcstringer]

Hey folks FWIW I experienced this same issue. I was able to push to Heroku using the very latest 0.7.6 version of this gem by upgrading to the very latest Heroku stack, which is Cedar-16. You can accomplish this by adding/editing an app.json in the root of your app and specifying the stack there.

"stack": "heroku-16",

https://devcenter.heroku.com/articles/upgrading-to-the-latest-stack

Hope this helps others.