search

rake-compiler / rake-compiler-dock

Easy to use and reliable cross compiler environment for building Windows, Linux, Mac and JRuby binary gems.
MIT License
78 stars 30 forks source link

Running rake-compiler-dock in docker #35

Closed alexbuyn closed 1 year ago

alexbuyn commented 4 years ago

Hi, I'm new to ruby, forgive me if that's a dumb question :)

I have a build server with ubuntu 18.04 and docker in it. I'd like to make a dockerfile to build my ruby gem which has native C code in it. Rake-compiler-dock seems perfect for the job. But I need to run it from a docker image, I don't want to install any build dependencies directly on build server. I plan to try and spawn sibling containers for builds instead of docker-in-docker.

My dockerfile for build environment looks like this for now:

FROM ruby 2.7
RUN curl -fsSLO https://download.docker.com/linux/static/stable/x86_64/docker-18.03.1-ce.tgz && tar xzvf docker-18.03.1-ce.tgz --strip 1 -C /usr/local/bin docker/docker && rm docker-18.03.1-ce.tgz

so I have ruby and docker.

then I do docker build . -t my-ruby-dev-env

So I run the docker environment:

docker run --rm -it --privileged=true -v //var/run/docker.sock:/var/run/docker.sock --mount src="$(pwd)",target=/usr/app,type=bind my-ruby-dev-env /bin/bash

Go into my app folder, and run bundle install. It succeeds.

This is my rake task:

` desc 'Compile native gems for distribution (Linux and Windows)' task 'gem:native' do require 'rake_compiler_dock' sh "rm -Rf pkg" # ensure clean package state sh "bundle package" # Avoid repeated downloads of gems by using gem files from the host. exttask.cross_platform.each do |plat| next if plat =~ /darwin/

Avoid conflicting declarations of gettimeofday: https://github.com/rake-compiler/rake-compiler-dock/issues/32

RakeCompilerDock.sh "find /usr/local/rake-compiler -name win32.h | while read f ; do sudo sed -i 's/gettimeofday/rb_gettimeofday/' $f ; done && bundle --local && rake clean && rake native:#{plat} gem RUBY_CC_VERSION=#{SUPPORTED_RUBY_VERSIONS}", platform: plat, verbose: true

end end`

When I run it I get the error:

bundle exec rake gem:native --trace fatal: not a git repository (or any parent up to mount point /usr) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). fatal: not a git repository (or any parent up to mount point /usr) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). ** Invoke gem:native (first_time) ** Invoke clean (first_time) ** Execute clean /usr/local/bundle/gems/rake-12.3.3/lib/rake/clean.rb:34: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call /usr/local/lib/ruby/2.7.0/fileutils.rb:622: warning: The called methodrm_r' is defined here rm -r tmp rm -r tmp/x86_64-linux/raygun_ext/2.7.1 rm -r tmp/x86_64-linux/stage Execute gem:native rm -Rf pkg bundle package fatal: not a git repository (or any parent up to mount point /usr) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). Your Gemfile contains path and git dependencies. If you want to cache them as well, please pass the --all flag. This will be the default on Bundler 3.0. Using rake 12.3.3 Using benchmark_driver 0.15.9 Using bundler 2.1.4 Using debase-ruby_core_source 0.10.9 Using unf_ext 0.0.7.7 Using unf 0.1.4 Using domain_name 0.5.20190701 Using excon 0.73.0 Using multipart-post 2.1.1 Using faraday 1.0.1 Using http-accept 1.7.0 Using http-cookie 1.0.3 Using mime-types-data 3.2020.0425 Using mime-types 3.3.1 Using multi_xml 0.6.0 Using httparty 0.18.0 Using httpclient 2.8.3 Using irb 1.0.0 Using minitest 5.13.0 Using netrc 0.11.0 Using rake-compiler 1.1.0 Using rake-compiler-dock 1.0.1 Using raygun-apm 1.0.49 from source at . Using rest-client 2.1.0 Updating files in vendor/cache Bundle complete! 15 Gemfile dependencies, 24 gems now installed. Use bundle info [gemname] to see where a bundled gem is installed. Updating files in vendor/cache rake-compiler-dock bash -c "find /usr/local/rake-compiler -name win32.h | while read f ; do sudo sed -i 's/gettimeofday/rb_gettimeofday/' $f ; done && bundle --local && rake clean && rake native:x86-mingw32 gem RUBY_CCVERSION=2.5.0:2.6.0:2.7.0" docker run -v /usr/app:/usr/app -e UID\=0 -e GID\=0 -e USER\= -e GROUP\=_root -e GEM_PRIVATE_KEY_PASSPHRASE -e ftp_proxy -e http_proxy -e https_proxy -e RCD_HOST_RUBY_PLATFORM\=x86_64-linux -e RCD_HOST_RUBY_VERSION\=2.7.1 -e RCD_IMAGE\=larskanis/rake-compiler-dock-mri-x86-mingw32:1.0.0 -w /usr/app --rm -i -t larskanis/rake-compiler-dock-mri-x86-mingw32:1.0.0 runas sigfw bash -c find\ /usr/local/rake-compiler\ -name\ win32.h\ |\ while\ read\ f\ \;\ do\ sudo\ sed\ -i\ \'s/gettimeofday/rb_gettimeofday/\'\ \$f\ \;\ done\ \&\&\ bundle\ --local\ \&\&\ rake\ clean\ \&\&\ rake\ native:x86-mingw32\ gem\ RUBY_CC_VERSION\=2.5.0:2.6.0:2.7.0 Don't run Bundler as root. Bundler can ask for sudo if it is needed, and installing your bundle as root will break this application for all non-root users on this machine. Could not locate Gemfile** rake aborted! Command failed with status (10): [docker run -v /usr/app:/usr/app -e UID\=0 -e GID\=0 -e USER\=_ -e GROUP\=_root -e GEM_PRIVATE_KEY_PASSPHRASE -e ftp_proxy -e http_proxy -e https_proxy -e RCD_HOST_RUBY_PLATFORM\=x86_64-linux -e RCD_HOST_RUBY_VERSION\=2.7.1 -e RCD_IMAGE\=larskanis/rake-compiler-dock-mri-x86-mingw32:1.0.0 -w /usr/app --rm -i -t larskanis/rake-compiler-dock-mri-x86-mingw32:1.0.0 runas sigfw bash -c find\ /usr/local/rake-compiler\ -name\ win32.h\ |\ while\ read\ f\ \;\ do\ sudo\ sed\ -i\ \'s/gettimeofday/rb_gettimeofday/\'\ \$f\ \;\ done\ \&\&\ bundle\ --local\ \&\&\ rake\ clean\ \&\&\ rake\ native:x86-mingw32\ gem\ RUBY_CC_VERSION\=2.5.0:2.6.0:2.7.0] /usr/local/bundle/gems/rake-compiler-dock-1.0.1/lib/rake_compiler_dock/starter.rb:96:in block in exec' /usr/local/bundle/gems/rake-compiler-dock-1.0.1/lib/rake_compiler_dock/starter.rb:46:ineach' /usr/local/bundle/gems/rake-compiler-dock-1.0.1/lib/rake_compiler_dock/starter.rb:46:in exec' /usr/local/bundle/gems/rake-compiler-dock-1.0.1/lib/rake_compiler_dock/starter.rb:14:insh' /usr/local/bundle/gems/rake-compiler-dock-1.0.1/lib/rake_compiler_dock.rb:46:in sh' /usr/app/Rakefile:41:inblock (2 levels) in <top (required)>' /usr/app/Rakefile:38:in each' /usr/app/Rakefile:38:inblock in <top (required)>' /usr/local/bundle/gems/rake-12.3.3/lib/rake/task.rb:273:in block in execute' /usr/local/bundle/gems/rake-12.3.3/lib/rake/task.rb:273:ineach' /usr/local/bundle/gems/rake-12.3.3/lib/rake/task.rb:273:in execute' /usr/local/bundle/gems/rake-12.3.3/lib/rake/task.rb:214:inblock in invoke_with_call_chain' /usr/local/bundle/gems/rake-12.3.3/lib/rake/task.rb:194:in synchronize' /usr/local/bundle/gems/rake-12.3.3/lib/rake/task.rb:194:ininvoke_with_call_chain' /usr/local/bundle/gems/rake-12.3.3/lib/rake/task.rb:183:in invoke' /usr/local/bundle/gems/rake-12.3.3/lib/rake/application.rb:160:ininvoke_task' /usr/local/bundle/gems/rake-12.3.3/lib/rake/application.rb:116:in block (2 levels) in top_level' /usr/local/bundle/gems/rake-12.3.3/lib/rake/application.rb:116:ineach' /usr/local/bundle/gems/rake-12.3.3/lib/rake/application.rb:116:in block in top_level' /usr/local/bundle/gems/rake-12.3.3/lib/rake/application.rb:125:inrun_with_threads' /usr/local/bundle/gems/rake-12.3.3/lib/rake/application.rb:110:in top_level' /usr/local/bundle/gems/rake-12.3.3/lib/rake/application.rb:83:inblock in run' /usr/local/bundle/gems/rake-12.3.3/lib/rake/application.rb:186:in standard_exception_handling' /usr/local/bundle/gems/rake-12.3.3/lib/rake/application.rb:80:inrun' /usr/local/bundle/gems/rake-12.3.3/exe/rake:27:in <top (required)>' /usr/local/bundle/bin/rake:23:inload' /usr/local/bundle/bin/rake:23:in <top (required)>' /usr/local/lib/ruby/2.7.0/bundler/cli/exec.rb:63:inload' /usr/local/lib/ruby/2.7.0/bundler/cli/exec.rb:63:in kernel_load' /usr/local/lib/ruby/2.7.0/bundler/cli/exec.rb:28:inrun' /usr/local/lib/ruby/2.7.0/bundler/cli.rb:476:in exec' /usr/local/lib/ruby/2.7.0/bundler/vendor/thor/lib/thor/command.rb:27:inrun' /usr/local/lib/ruby/2.7.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in invoke_command' /usr/local/lib/ruby/2.7.0/bundler/vendor/thor/lib/thor.rb:399:indispatch' /usr/local/lib/ruby/2.7.0/bundler/cli.rb:30:in dispatch' /usr/local/lib/ruby/2.7.0/bundler/vendor/thor/lib/thor/base.rb:476:instart' /usr/local/lib/ruby/2.7.0/bundler/cli.rb:24:in start' /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.1.4/libexec/bundle:46:inblock in <top (required)>' /usr/local/lib/ruby/2.7.0/bundler/friendly_errors.rb:123:in with_friendly_errors' /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.1.4/libexec/bundle:34:in<top (required)>' /usr/local/bin/bundle:23:in load' /usr/local/bin/bundle:23:in

' Tasks: TOP => gem:native`

In my local dev environment in docker for windows when I try and run the same script it mounts the directory from host to the directory in the container, trying /usr/app:/usr/app mount and fails. Maybe its similar and is some mount problem with sibling containers. Any help will be appreciated.

When I try and run docker run -v /usr/app:/usr/app -e UID\=0 -e GID\=0 -e USER\=_ -e GROUP\=_root -e GEM_PRIVATE_KEY_PASSPHRASE -e ftp_proxy -e http_proxy -e https_proxy -e RCD_HOST_RUBY_PLATFORM\=x86_64-linux -e RCD_HOST_RUBY_VERSION\=2.7.1 -e RCD_IMAGE\=larskanis/rake-compiler-dock-mri-x86-mingw32:1.0.0 -w /usr/app --rm -i -t larskanis/rake-compiler-dock-mri-x86-mingw32:1.0.0 runas sigfw bash the /usr/app directory is empty, that's why the error.

Thanks!

larskanis commented 2 years ago

rake-compiler-dock-1.2.0 has better support for container based CI due to PR #57 . Possibly this can solve your issue.