Building native extensions

[gouravkhunger]

Hi!

Context: I am trying to create installable ruby bootstraps for android architectures. I have altered the default bundled gems so that I can customize what gems are pre-installed. I wish to include the gem sass-embedded as well.

To do so I want to understand the build steps to set up this gem's native extensions.

During ruby's build, rbinstall.rb is responsible for building the a gem's extensions and then installing gems to the correct place. I patch it introduce a step to give preference from my build files as sometimes things need to be altered for android arch.

Based on the extension being built for a gem, the installation is assigned a builder and the :build_args from the installation options are passed to it.

builder.rb

  def builder_for(extension) # :nodoc:
    case extension
    when /extconf/ then
      Gem::Ext::ExtConfBuilder
    when /configure/ then
      Gem::Ext::ConfigureBuilder
    when /rakefile/i, /mkrf_conf/i then
      @ran_rake = true
      Gem::Ext::RakeBuilder
    when /CMakeLists.txt/ then
      Gem::Ext::CmakeBuilder
    when /Cargo.toml/ then
      Gem::Ext::CargoBuilder.new
    else
      build_error("No builder for extension '#{extension}'")
    end
  end

and since it'll be RakeBuilder for this gem, upon inspection I see the args are passed to the rake command. So I can run tasks like compile by passing them to the :build_args in options.

The catch here is that it assumes the Rakefile is always from the gem's extension(ext) folder. Now, from what I understand, the task install in ext/sass/Rakefile depends on task compile from Rakefile being pre-run to set up required files. Thus, I can't use options[:build_args] = ["compile"] before running the steps from the top level Rakefile. Thus it skips the extensions but installs the other files. But then requiring sass-embedded in irb keeps saying that it will be ignored as it's extensions are not built.

The possible fixes could be:

• Centralise the build procedure. It is easier to converge things into one Rakefile inside ext/sass and maybe run something like rake compile install so the tasks run from one place. Which I can then use in the args. I need some help to do that.

• Introduce additional steps in rbinstall.rb to run required tasks before proceeding with the normal installation.

  This is more hacky and might take time to work on as there are caveats. For example, the rake it should use should be the one from target OS's ruby build. It may be tricky to ensure because ruby's installation first requires a host build and then a target build.

I hope I was able to describe the issue in detail. Any help would be appreciated!

[gouravkhunger]

I know that sass-embedded ships native pre-compiled versions of the gem for android architectures. So I can always fallback to running gem install sass-embedded when I run my android app and it will work. But it will be nice to have it done during build step so as to have keep everything in the bootstrap ready to go. I have all other required gems compiled and working.

And also, I've tried fixating the versions I include in the bundled_gems file to like 1.77.4-aarch64-linux-android instead of just 1.77.4 but that ends up skipping bundling the gem altogether. Right now even though the extensions are skipped, I do have it unpacked in the bootstrap at least.

[ntkme]

You don't need the Rakefile at the root of the project. That is just a helper file for local development. During the gem installation only ext/sass/Rakefile is needed.

[gouravkhunger]

I had checked the release workflow to get the gist.

https://github.com/sass-contrib/sass-embedded-host-ruby/blob/266af6151c32dc9506af31397c65254339501ef1/.github/workflows/release.yml#L85-L86

Nice, but well there's no cli.rb or embedded_sass_pb.rb by default, I only see the cli.rb being set up by root level Rakefile.

[gouravkhunger]

Actually, the build steps act like it is building the gem from source. It's not like a regular gem install sass-embedded.

[ntkme]

I'm assuming you're cross compiling? That would likely be where your issue is coming from. ext/sass/Rakefile builds a few things:

Let's say build arch is the arch of the machine that builds, and target arch is android.

1. dart-sass + cli.rb - this need to be the target arch.
2. protoc.exe + embedded_sass_pb.rb - this depends on dart-sass + cli.rb to be the build arch, in order to run the cli and get the right version. - This is a conflict with step 2, and that's why the Rakefile at the root actually runs ext/sass/Rakefile twice for cross-compiling. You can use rake parameter overrides to force it to use a certain file without dealing with this architecture difference.

[ntkme]

I only see the cli.rb being set up by root level Rakefile.

The install (default) target in ext/sass/Rakefile will set it up if not found: https://github.com/sass-contrib/sass-embedded-host-ruby/blob/b33d2795e72c72bb40cf14d94c2e64ac5fb108b4/ext/sass/Rakefile#L7-L9

[gouravkhunger]

Yes it is a cross compilation using Android NDK in a termux-packages build system. I build ruby from a linux container for 4 target android arches.

Could you please describe the build steps for a normal system. Maybe I could try adapt rbinstall.rb or other steps for it it in my set up.

[ntkme]

Here is the general steps this gem builds:

1. Download ext/sass/dart-sass for current RbConfig::CONFIG['host_os'] and RbConfig::CONFIG['host_cpu'], and generate ext/sass/cli.rb.
2. (Optional) If embedded_sass_pb.rb does not exists:
   1. Run ./dart-sass/sass --embedded --version to get protocolVersion.
   2. Download embedded_sass.proto for current protocolVersion.
   3. Download protoc.exe for current RbConfig::CONFIG['host_os'] and RbConfig::CONFIG['host_cpu'].
   4. Run protoc.exe to generate embedded_sass_pb.rb.
3. (Optional) If cross compiling for a different architecture:
   1. Remove ext/sass/dart-sass and ext/sass/cli.rb.
   2. Redownload for the target architecture.

A few notes:

• Step 2 is only required when compiling from the git source. The published platform ruby gem has embedded_sass_pb.rb pre-bundled.
• The Rakefile at the project root is a mainly just a helper to make running step 3 easy.

So for your purpose, I think what you can do is:

1. Clone this git repo, pre-build embedded_sass_pb.rb, then remove ext/sass/dart-sass and ext/sass/cli.rb.
2. When compiling for the target arch, rerun rake target cli.rb with architecture override, which will download the dart-sass.

In other words, try prepare the git repo into the state at step 3.i, then when installing you only need step 3.ii

[ntkme]

Ok. I think it's going to be very difficult to build the extension with the rbinstall.rb's build-ext mechanism. My recommendation is to prepare the repo into a state that is ready for the architecture you're targeting, and then just use rbinstall.rb to bundle it.

For example, save the follow script as prepare.rb:

require 'bundler'

platform = ARGV[0]

system(*%w[git checkout HEAD -- sass-embedded.gemspec])

system(*%w[bundle])

system(*%w[bundle exec rake compile], "ext_platform=#{platform}")

ENV['gem_platform'] = platform

spec = Bundler.load_gemspec_uncached('sass-embedded.gemspec')

spec.platform = 'ruby'

File.write('sass-embedded.gemspec', spec.to_ruby)

Then, you can run ruby prepare.rb aarch64-linux-android, and this will prepare the repo to be in already compiled and ready to use state for aarch64-linux-android. Since you have a patching step, you can use the patch step to do this.

[ntkme]

Normally, cross-compile relies on the cross-compile toolchain to target the right architecture. However, this gem is an exception that it does not use any compiler toolchain, thus from Rakefile's point of view it has no way to properly know what cross-compile target you're running for. Therefore, I think the workaround above might be the best you can do.

[gouravkhunger]

Hey, this was really helpful! I understand the procedure a lot better now. I tried to adapt it to my set up. For example adding an adapted version of the root level Rakefile and applying it's steps using something like:

if spec.name == "sass-embedded"
    output = system("cd #{srcdir}/.bundle/gems/#{gem_name} && gem install google-protobuf && rake compile ext_platform=#{RbConfig::CONFIG['platform']}")
    puts output
end

I was able to see logs and confirm the required target arch files were being prepared as you mentioned. The build succeeded.

But I kept getting this error during runtime on the app that certain files were still still from the host not the target arch:

:/data/data/sh.gourav.jekyllex/files/home $ irb
Ignoring sass-embedded-1.77.4 because its extensions are not built. Try: gem pristine sass-embedded --version 1.77.4
irb(main):001> require 'sass-embedded'
=> true
irb(main):002> Sass.compile_string('')
/data/data/sh.gourav.jekyllex/files/usr/lib/ruby/gems/3.3.0/gems/sass-embedded-1.77.4-x86_64-linux-gnu/lib/sass/compiler/connection.rb:58: warning: error: "/data/app/~~403zZg0feGSMNUJRtmSEvA==/sh.gourav.jekyllex-kuFpzrgCodwdC3hznV-OUQ==/lib/arm64/lib1835.so" is for EM_X86_64 (62) instead of EM_AARCH64 (183)
/data/data/sh.gourav.jekyllex/files/usr/lib/ruby/gems/3.3.0/gems/sass-embedded-1.77.4-x86_64-linux-gnu/lib/sass/compiler/varint.rb:20:in `readbyte': end of file reached (EOFError)
        from /data/data/sh.gourav.jekyllex/files/usr/lib/ruby/gems/3.3.0/gems/sass-embedded-1.77.4-x86_64-linux-gnu/lib/sass/compiler/varint.rb:20:in `block in read'
        from <internal:kernel>:187:in `loop'
        from /data/data/sh.gourav.jekyllex/files/usr/lib/ruby/gems/3.3.0/gems/sass-embedded-1.77.4-x86_64-linux-gnu/lib/sass/compiler/varint.rb:19:in `read'
        from /data/data/sh.gourav.jekyllex/files/usr/lib/ruby/gems/3.3.0/gems/sass-embedded-1.77.4-x86_64-linux-gnu/lib/sass/compiler/connection.rb:43:in `block (2 levels) in listen'
        from <internal:kernel>:187:in `loop'
        from /data/data/sh.gourav.jekyllex/files/usr/lib/ruby/gems/3.3.0/gems/sass-embedded-1.77.4-x86_64-linux-gnu/lib/sass/compiler/connection.rb:42:in `block in listen'
irb(main):003>

It turns out that it was the best for me to somehow install the platform native gem directly from rubygems. So I ended up adding this to rbinstall.rb after the bundled-gems installation steps. And things just work fine now.

system("gem install sass-embedded -v 1.77.4 --platform #{RbConfig::CONFIG['platform']} --ignore-dependencies -i #{install_dir}")

Maybe I'll try to implement it in graceful way in the future but I guess this is the best to get things rolling for now.