search

rubyjs / libv8

Ruby gem binary distribution of the V8 JavaScript engine
269 stars 121 forks source link

installation fails on armv7l (scaleway cloud) #226

Closed rriemann closed 8 years ago

rriemann commented 8 years ago

I want to use discourse/mini_racer on my armv7l ubuntu maschine (Scaleway hoster). Unfortunately, I have a problem with the libv8 dependency.

First, I wonder why I cannot use the binary built available for arm.

Then, I wonder why also the compilation files. :( 

Installing libv8 5.0.71.48.3 with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/libv8-5.0.71.48.3/ext/libv8
/usr/local/rvm/rubies/ruby-2.3.1/bin/ruby -r ./siteconf20161003-17426-16wqsxi.rb extconf.rb --platform arm-linux
creating Makefile
Running: gclient root
Running: gclient config --spec 'solutions = [
  {
    "managed": False,
    "name": "v8",
    "url": "https://chromium.googlesource.com/v8/v8.git",
    "custom_deps": {},
    "deps_file": "DEPS",
    "safesync_url": "",
  },
]
'
Running: gclient sync --with_branch_heads
Running: git submodule foreach 'git config -f $toplevel/.git/config submodule.$name.ignore all'
Running: git config --add remote.origin.fetch '+refs/tags/*:refs/tags/*'
Running: git config diff.ignoreSubmodules all

________ running '/usr/bin/python v8/build/landmines.py' in
'/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/libv8-5.0.71.48.3/vendor'
Clobbering due to:
--- old_landmines       Mon Oct  3 21:03:31 2016
+++ new_landmines       Mon Oct  3 21:07:21 2016
@@ -12,3 +11,0 @@
-Clobber after ICU roll.
-Clobber after Android NDK update.
-Clober to fix windows build problems.

________ running 'download_from_google_storage --no_resume --platform=win32 --no_auth --bucket chromium-clang-format -s
v8/buildtools/win/clang-format.exe.sha1' in '/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/libv8-5.0.71.48.3/vendor'
The current platform doesn't match "win32", skipping.

________ running 'download_from_google_storage --no_resume --platform=darwin --no_auth --bucket chromium-clang-format -s
v8/buildtools/mac/clang-format.sha1' in '/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/libv8-5.0.71.48.3/vendor'
The current platform doesn't match "darwin", skipping.

________ running 'download_from_google_storage --no_resume --platform=linux* --no_auth --bucket chromium-clang-format -s
v8/buildtools/linux64/clang-format.sha1' in '/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/libv8-5.0.71.48.3/vendor'
0> Downloading v8/buildtools/linux64/clang-format...
Success!
Downloading 1 files took 25.373686 second(s)
Hook 'download_from_google_storage --no_resume '--platform=linux*' --no_auth --bucket chromium-clang-format -s
v8/buildtools/linux64/clang-format.sha1' took 25.58 secs

________ running '/usr/bin/python v8/tools/gcmole/download_gcmole_tools.py' in
'/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/libv8-5.0.71.48.3/vendor'
Skipping gcmole download as gcmole is not set in gyp flags.

________ running '/usr/bin/python v8/tools/jsfunfuzz/download_jsfunfuzz.py' in
'/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/libv8-5.0.71.48.3/vendor'
Skipping jsfunfuzz download as jsfunfuzz is not set in gyp flags.

________ running 'download_from_google_storage --no_resume --platform=win32 --no_auth --bucket chromium-luci -d
v8/tools/luci-go/win64' in '/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/libv8-5.0.71.48.3/vendor'
The current platform doesn't match "win32", skipping.

________ running 'download_from_google_storage --no_resume --platform=darwin --no_auth --bucket chromium-luci -d
v8/tools/luci-go/mac64' in '/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/libv8-5.0.71.48.3/vendor'
The current platform doesn't match "darwin", skipping.

________ running 'download_from_google_storage --no_resume --platform=linux* --no_auth --bucket chromium-luci -d
v8/tools/luci-go/linux64' in '/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/libv8-5.0.71.48.3/vendor'
0> File v8/tools/luci-go/linux64/isolate exists and SHA1 matches. Skipping.
Success!
Downloading 1 files took 0.131099 second(s)

________ running 'download_from_google_storage --no_resume --platform=win32 --no_auth --bucket chromium-gn -s
v8/buildtools/win/gn.exe.sha1' in '/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/libv8-5.0.71.48.3/vendor'
The current platform doesn't match "win32", skipping.

________ running 'download_from_google_storage --no_resume --platform=darwin --no_auth --bucket chromium-gn -s
v8/buildtools/mac/gn.sha1' in '/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/libv8-5.0.71.48.3/vendor'
The current platform doesn't match "darwin", skipping.

________ running 'download_from_google_storage --no_resume --platform=linux* --no_auth --bucket chromium-gn -s
v8/buildtools/linux64/gn.sha1' in '/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/libv8-5.0.71.48.3/vendor'
0> Downloading v8/buildtools/linux64/gn...
Success!
Downloading 1 files took 25.238818 second(s)
Hook 'download_from_google_storage --no_resume '--platform=linux*' --no_auth --bucket chromium-gn -s
v8/buildtools/linux64/gn.sha1' took 25.45 secs

________ running '/usr/bin/python v8/build/vs_toolchain.py update' in
'/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/libv8-5.0.71.48.3/vendor'

________ running '/usr/bin/python v8/third_party/binutils/download.py' in
'/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/libv8-5.0.71.48.3/vendor'

________ running '/usr/bin/python v8/build/download_gold_plugin.py' in
'/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/libv8-5.0.71.48.3/vendor'
Skipping gold plugin download for non-cfi build.

________ running '/usr/bin/python v8/tools/clang/scripts/update.py --if-needed' in
'/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/libv8-5.0.71.48.3/vendor'
Updating Clang to 259396-1...
Downloading prebuilt clang
Downloading https://commondatastorage.googleapis.com/chromium-browser-clang/Linux_x64/clang-259396-1.tgz .......... Done.
Creating directory
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/libv8-5.0.71.48.3/vendor/v8/third_party/llvm-build/Release+Asserts
clang 259396-1 unpacked

________ running '/usr/bin/python v8/build/gyp_v8' in
'/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/libv8-5.0.71.48.3/vendor'
Updating projects from gyp files...
Hook '/usr/bin/python v8/build/gyp_v8' took 17.90 secs

WARNING: 'v8/third_party/instrumented_libraries' is no longer part of this client.  It is recommended that you manually remove
it.

WARNING: 'v8/tools/gyp' is no longer part of this client.  It is recommended that you manually remove it.

WARNING: 'v8/third_party/WebKit/Source/platform/inspector_protocol' is no longer part of this client.  It is recommended that
you manually remove it.

WARNING: 'v8/test/test262/harness' is no longer part of this client.  It is recommended that you manually remove it.

WARNING: 'v8/third_party/markupsafe' is no longer part of this client.  It is recommended that you manually remove it.

WARNING: 'v8/third_party/jinja2' is no longer part of this client.  It is recommended that you manually remove it.

Applying /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/libv8-5.0.71.48.3/patches/build-standalone-static-library.patch
Applying /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/libv8-5.0.71.48.3/patches/disable-building-tests.patch
Applying /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/libv8-5.0.71.48.3/patches/fPIC-for-static.patch
Unsupported target: armv7l
Compiling v8 for armv7l
Using python 2.7.9
Using compiler: g++ (GCC version 4.9.2)
Beginning compilation. This will take some time.
Unsupported target: armv7l
Building v8 with env CXX=g\+\+ LINK=g\+\+ /usr/bin/make armv7l.release i18nsupport=off werror=no
GYP_DEFINES="v8_use_external_startup_data=0 linux_use_bundled_gold=0 clang=0"
make: *** No rule to make target 'armv7l.release'.  Stop.
Unsupported target: armv7l
Unsupported target: armv7l
Unsupported target: armv7l
Unsupported target: armv7l
Unsupported target: armv7l
Unsupported target: armv7l
Unsupported target: armv7l
Unsupported target: armv7l
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/libv8-5.0.71.48.3/ext/libv8/location.rb:40:in `block in verify_installation!':
libv8 did not install properly, expected binary v8 archive
'/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/libv8-5.0.71.48.3/vendor/v8/out/armv7l.release/obj.target/tools/gyp/libv8_base.a'to
exist, but it was not found (Libv8::Location::Vendor::ArchiveNotFound)
        from /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/libv8-5.0.71.48.3/ext/libv8/location.rb:39:in `each'
        from /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/libv8-5.0.71.48.3/ext/libv8/location.rb:39:in `verify_installation!'
        from /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/libv8-5.0.71.48.3/ext/libv8/location.rb:26:in `install!'
        from extconf.rb:7:in `<main>'

extconf failed, exit code 1

Gem files will remain installed in /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/libv8-5.0.71.48.3 for inspection.
Results logged to /var/www/discourse/vendor/bundle/ruby/2.3.0/extensions/armv7l-linux/2.3.0/libv8-5.0.71.48.3/gem_make.out

An error occurred while installing libv8 (5.0.71.48.3), and Bundler cannot continue.
Make sure that `gem install libv8 -v '5.0.71.48.3'` succeeds before bundling.
ignisf commented 8 years ago

Hello,

Thank you for reporting this issue!

First, I wonder why I cannot use the binary built available for arm.

Because we don't have any trusted ARM systems to build binaries on.

Then, I wonder why also the compilation files. :(

Because we don't have any ARM systems to test on.

ignisf commented 8 years ago

Could you please test if the armv7l branch works for you?

Add this to your Gemfile:

gem "libv8", github: "cowboyd/libv8", submodules: true, branch: "armv7l"

Thanks!

ignisf commented 8 years ago

Honestly, I tried making a QEMU ARM VM... It takes more than a day to compile V8. It's just not practical :(

rriemann commented 8 years ago

Thanks for your quick response!

I'm compiling right in the cloud the branch you mentioned plus an other version based on libv8 v5.0.x. I will let you know about the result.

Compiling seems to take indeed a long while. Configuring bundler to use 4 jobs does not seem to be taken into account.

For both test, I created two separate Gemfiles in distinct folders and run bundle install --no-deployment --without test development --path vendor/bundle.

How can I copy over/package the binary gem to use it later in my production project? Is rsyncing vendor/bundle sufficient?

Especially because compiling does take so much time, it would be even more worth to have a binary version. Would a Rasberry Pi be a compatible, trusted platform? :smile:

ignisf commented 8 years ago

Compiling seems to take indeed a long while. Configuring bundler to use 4 jobs does not seem to be taken into account.

Running MAKEFLAGS+=-j4 bundle install should speed up the build.

How can I copy over/package the binary gem to use it later in my production project? Is rsyncing vendor/bundle sufficient?

It should be.

Would a Rasberry Pi be a compatible, trusted platform? 😄

Only if it's under my physical control.

rriemann commented 8 years ago

I try to install mini_racer and actually it seems to expect some files in armv7l.release, but the files are actually in arm.release.

g++: error: /usr/local/rvm/gems/ruby-2.3.1/gems/libv8-5.0.71.48.3-arm-linux/vendor/v8/out/armv7l.release/obj.target/tools/gyp/libv8_base.a: No such file or directory
ignisf commented 8 years ago
  1. Please make sure you've removed all versions of libv8
  2. Use the HEAD version of mini_racer (gem 'mini_racer', github: 'discourse/mini_racer') because the currently released one requires libv8 5.0 and the V8 version of the armv7l branch here is > 5.3.
rriemann commented 8 years ago

Sorry for the uninformed questions:

  1. How do you make sure to not have any version of libv8 installed? I'm using rvm and bundler. Is gem uninstall sufficient?
  2. I'm using your tag https://github.com/cowboyd/libv8/tree/v5.0.71.48.3 and applied your last arm patch from armv71 to it: https://github.com/rriemann/libv8/commit/153221587c7b58926a4b505e4402fb480d6db255

If I would apply the arm patch to your HEAD, will I still be able to compile with libv8 5.0?

ignisf commented 8 years ago

Is gem uninstall sufficient?

yes.

I'm using your tag https://github.com/cowboyd/libv8/tree/v5.0.71.48.3 and applied your last arm patch from armv71 to it: rriemann@1532215

Had not noticed that. Doing this should be OK.

rriemann commented 8 years ago

So that's what I did in the initial case and I still ran into the problem.

I'm still investigating an other problem. Apparently, mini_racer cannot find libv8. As compiling is slow, testing is rather difficult. I let you know once I tracked further down the issue.

ignisf commented 8 years ago

Indeed the issue I'm seeing is the same you noticed:

g++: error: /usr/local/rvm/gems/ruby-2.3.1/gems/libv8-5.0.71.48.3-arm-linux/vendor/v8/out/armv7l.release/obj.target/tools/gyp/libv8_base.a: No such file or directory
                                                   ------------ this should be arm --------^

When mini_racer is asking the libv8 gem for compiler parameters, the libv8 gem should construct the path to the .a files from Libv8::Arch.libv8_arch. Which should return arm after applying my patch, unless it doesn't work :D

rriemann commented 8 years ago

I'm not sure if I understand the subtlety when the arch is arm and when armv7l. Could you create an issue for mini_racer than if you think that's their department?

It seems to me that libv8 is always downloaded and compiled from scratch when I run bundle install and installation has failed at some point before. Compilation is indeed very slow… :/

ignisf commented 8 years ago

can you add me a user on an ARM system so I can take a look for myself? My ssh keys can be found here: https://petko.me/pubkey.txt

ignisf commented 8 years ago

Could you create an issue for mini_racer than if you think that's their department?

mini_racer's not at fault here

rriemann commented 8 years ago

I've sent you a mail.

last attempt with a Gemfile containing my libv8 branch (v5.0) and mini_racer

bundle package
The git source `git://github.com/rriemann/libv8.git` uses the `git` protocol, which transmits data without encryption. Disable this warning with `bundle config 
git.allow_insecure true`, or switch to the `https` protocol to keep your data secure.
Fetching git://github.com/rriemann/libv8.git  
Cloning into 'vendor/depot_tools'...
Fetching gem metadata from https://rubygems.org/..........
Fetching version metadata from https://rubygems.org/.
Resolving dependencies...
Using bundler 1.13.2
Using libv8 5.0.71.48.3 from git://github.com/rriemann/libv8.git (at patch-5.0.71.48.3-arm-linux@1532215)
Installing mini_racer 0.1.3 with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /home/discourse/libv8-testing-v5.0/vendor/bundle/ruby/2.3.0/gems/mini_racer-0.1.3/ext/mini_racer_extension
/usr/local/rvm/rubies/ruby-2.3.1/bin/ruby -r ./siteconf20161005-11956-166vwxb.rb extconf.rb
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
        --with-opt-dir
        --without-opt-dir
        --with-opt-include
        --without-opt-include=${opt-dir}/include
        --with-opt-lib
        --without-opt-lib=${opt-dir}/lib
        --with-make-prog
        --without-make-prog
        --srcdir=.
        --curdir
        --ruby=/usr/local/rvm/rubies/ruby-2.3.1/bin/$(RUBY_BASE_NAME)
/usr/local/rvm/rubies/ruby-2.3.1/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- libv8 (LoadError)
        from /usr/local/rvm/rubies/ruby-2.3.1/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from extconf.rb:2:in `<main>'

extconf failed, exit code 1

Gem files will remain installed in /home/discourse/libv8-testing-v5.0/vendor/bundle/ruby/2.3.0/gems/mini_racer-0.1.3 for inspection.
Results logged to /home/discourse/libv8-testing-v5.0/vendor/bundle/ruby/2.3.0/extensions/armv7l-linux/2.3.0/mini_racer-0.1.3/gem_make.out

An error occurred while installing mini_racer (0.1.3), and Bundler cannot continue.
Make sure that `gem install mini_racer -v '0.1.3'` succeeds before bundling.
ignisf commented 8 years ago

I've pushed a binary for ARM. I'm hoping somebody will donate an ARM box for us to do this in the future. Again, make sure you do a gem uninstall libv8 before trying to install mini_racer again.

ignisf commented 8 years ago

And boy does asset crunching run slow on ARM 😁😁😬😬😬

ignisf commented 8 years ago

@edouardb FYI Discourse should run fine on Scaleway now with mini_racer instead of TRR