Update Ruby gem with arm64 binaries (Apple M1 support)

[mloughran]

The Ruby gem is shipped with compiled extensions (via https://github.com/rake-compiler/rake-compiler), but it is missing arm64 binaries.

This results in errors like the following when installing the gem and requiring:

$ gem install google-protobuf

$ ruby -e 'require "google/protobuf"'
Traceback (most recent call last):
    2: from -e:1:in `<main>'
    1: from <snip>/rubygems/core_ext/kernel_require.rb:92:in `require'
<snip>/rubygems/core_ext/kernel_require.rb:92:in `require': cannot load such file -- google/protobuf (LoadError)
    6: from -e:1:in `<main>'
    5: from <snip>/rubygems/core_ext/kernel_require.rb:156:in `require'
    4: from <snip>/rubygems/core_ext/kernel_require.rb:168:in `rescue in require'
    3: from <snip>/rubygems/core_ext/kernel_require.rb:168:in `require'
    2: from <snip>/gems/google-protobuf-3.14.0-universal-darwin/lib/google/protobuf.rb:49:in `<top (required)>'
    1: from <snip>/rubygems/core_ext/kernel_require.rb:92:in `require'
<snip>/rubygems/core_ext/kernel_require.rb:92:in `require': dlopen(<snip>/gems/google-protobuf-3.14.0-universal-darwin/lib/google/2.7/protobuf_c.bundle, 9): no suitable image found.  Did find: (LoadError)
    <snip>/gems/google-protobuf-3.14.0-universal-darwin/lib/google/2.7/protobuf_c.bundle: mach-o, but wrong architecture
    <snip>/gems/google-protobuf-3.14.0-universal-darwin/lib/google/2.7/protobuf_c.bundle: mach-o, but wrong architecture - <snip>/gems/google-protobuf-3.14.0-universal-darwin/lib/google/2.7/protobuf_c.bundle
    7: from -e:1:in `<main>'
    6: from <snip>/rubygems/core_ext/kernel_require.rb:156:in `require'
    5: from <snip>/rubygems/core_ext/kernel_require.rb:168:in `rescue in require'
    4: from <snip>/rubygems/core_ext/kernel_require.rb:168:in `require'
    3: from <snip>/gems/google-protobuf-3.14.0-universal-darwin/lib/google/protobuf.rb:48:in `<top (required)>'
    2: from <snip>/gems/google-protobuf-3.14.0-universal-darwin/lib/google/protobuf.rb:51:in `rescue in <top (required)>'
    1: from <snip>/rubygems/core_ext/kernel_require.rb:92:in `require'
<snip>/rubygems/core_ext/kernel_require.rb:92:in `require': cannot load such file -- google/protobuf_c (LoadError)

A workaround is to force compilation of native extensions on install which works perfectly:

$ gem install google-protobuf --platform=ruby

or globally via this bundler setting:

$ bundle config set force_ruby_platform true

Related to #8062

[LucaProvencal]

Confirmed that

$ bundle config set force_ruby_platform true

in combination with grpc v1.35.0 upgrade solved this issue for me.

[NielsKSchjoedt]

This still does not work for me. I have v. 3.15.3 installed and it throws this:

Traceback (most recent call last):
       16: from bootsnap (1.7.0) lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
       15: from google-protobuf-3.15.3-universal (darwin) lib/google/protobuf/any_pb.rb:4:in `<main>'
       14: from zeitwerk (2.4.2) lib/zeitwerk/kernel.rb:34:in `require'
       13: from bootsnap (1.7.0) lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
       12: from bootsnap (1.7.0) lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
       11: from bootsnap (1.7.0) lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
       10: from bootsnap (1.7.0) lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
        9: from bootsnap (1.7.0) lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
        8: from google-protobuf-3.15.3-universal (darwin) lib/google/protobuf.rb:48:in `<main>'
        7: from google-protobuf-3.15.3-universal (darwin) lib/google/protobuf.rb:51:in `rescue in <main>'
        6: from zeitwerk (2.4.2) lib/zeitwerk/kernel.rb:34:in `require'
        5: from bootsnap (1.7.0) lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:44:in `require'
        4: from bootsnap (1.7.0) lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
        3: from bootsnap (1.7.0) lib/bootsnap/load_path_cache/loaded_features_index.rb:89:in `register'
        2: from bootsnap (1.7.0) lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
        1: from bootsnap (1.7.0) lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
LoadError (cannot load such file -- google/protobuf_c)

If I try to load the gem locally and alter lines inside lib/google/protobuf.rb like so:

  # begin
    require "google/#{RUBY_VERSION.sub(/\.\d+$/, '')}/protobuf_c"
  # rescue LoadError
  #   require 'google/protobuf_c'
  # end

I get:

LoadError (dlopen(/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/google-protobuf-3.15.3-universal-darwin/lib/google/2.7/protobuf_c.bundle, 9): no suitable image found.  Did find:)
    /Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/google-protobuf-3.15.3-universal-darwin/lib/google/2.7/protobuf_c.bundle: mach-o, but wrong architecture
    /Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/google-protobuf-3.15.3-universal-darwin/lib/google/2.7/protobuf_c.bundle: stat() failed with errno=25 - /Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/google-protobuf-3.15.3-universal-darwin/lib/google/2.7/protobuf_c.bundle

Any clue why this happens?

[NielsKSchjoedt]

Okay I tried updating to v 3.15.4 then it worked. 1 hour later it suddenly stopped working again 🤯 Something is really fishy...

LoadError (dlopen(/Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/google-protobuf-3.15.4-universal-darwin/lib/google/2.7/protobuf_c.bundle, 9): no suitable image found.  Did find:)
    /Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/google-protobuf-3.15.4-universal-darwin/lib/google/2.7/protobuf_c.bundle: mach-o, but wrong architecture
    /Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/google-protobuf-3.15.4-universal-darwin/lib/google/2.7/protobuf_c.bundle: mach-o, but wrong architecture - /Users/niels-kristian/.rvm/gems/ruby-2.7.2@rails6/gems/google-protobuf-3.15.4-universal-darwin/lib/google/2.7/protobuf_c.bundle

I just tried to checkout the ruby gem and see if it could build, it cannot: https://github.com/protocolbuffers/protobuf/issues/8372

[teddy1004]

solved on my new M1 Mac, try the following steps:

1. gem uninstall google-protobuf, and remove all google-protobuf
2. reinstall with gem install google-protobuf -v 3.15.8 --platform=ruby or any version you want

seems bundle config force ruby platform not working, we have to reinstall google-protobuf platform: ruby from scratch

[ericboehs]

None of these solutions worked for me on 3.17.3 on a 13" M1 MBP. I'm running the latest Monterey public beta (beta 5 I believe) so maybe that's the issue?

Installation went fine but when I go to require it I get a "No such file" error when looking for the arm64e bundle. Here's the full error:

.bundle/bundle/ruby/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/dependencies.rb:332:in `require': dlopen(.bundle/bundle/ruby/2.7.0/gems/google-protobuf-3.17.3-universal-darwin/lib/google/2.7/protobuf_c.bundle, 0x0009): tried: '.bundle/bundle/ruby/2.7.0/gems/google-protobuf-3.17.3-universal-darwin/lib/google/2.7/protobuf_c.bundle' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/local/lib/protobuf_c.bundle' (no such file), '/usr/lib/protobuf_c.bundle' (no such file) - .bundle/bundle/ruby/2.7.0/gems/google-protobuf-3.17.3-universal-darwin/lib/google/2.7/protobuf_c.bundle (LoadError)

I tried on Ruby 2.6.8p205 and 2.7.4p191. I tried recompiling Ruby several times using Xcode Command Line tools and the full version of Xcode (Mac App Store).

[Overload119]

Is there anyway to force bundler to use the Rubygems version installed with --platform ruby ?

Whenever I run bundle I see it's using the universal-darwin version. I've tried doing platform: :ruby to the line in the Gemfile but that doesn't work.

[bouk]

If you want to prevent bundler from installing these gems, add the following to you Gemfile:

# HACK(bouk): Overwrite Bundler's platform matcher to ignore universal CPU
# The protobuf and gRPC 'universal' macOS gems break on M1
module Bundler::MatchPlatform
  def match_platform(p)
    return false if ::Gem::Platform === platform && platform.cpu == "universal"
    Bundler::MatchPlatform.platforms_match?(platform, p)
  end
end

This will overwrite bundler's platform matcher to ignore any universal gems

[mloughran]

This was closed by #8232 (and released in 3.19.0).

[ngan]

I'm not sure if this is actually fixed. I'm still getting the same problem (protobuf_c.bundle: mach-o, but wrong architecture) after upgrading to 3.19.0.

I also don't see an arm64-darwin build for 3.19.0:

[acozzette]

I think some more work is required on the protobuf team's part before we can start publishing Mac arm64 binaries. The build is happening on an x86 machine and we probably need to tweak some things to get cross-compilation working.

[jnarowski]

+1 for this

[dangerrg]

I've tried all of the suggestions above, but still doesn't work for me:

I get this error:

/Users/dangerrg/.rvm/gems/ruby-3.0.3/gems/bootsnap-1.9.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `
require': dlopen(/Users/dangerrg/.rvm/gems/ruby-3.0.3/gems/digest-3.1.0/lib/digest.bundle, 0x0009): tried: '/Users/dange
rrg/.rvm/gems/ruby-3.0.3/gems/digest-3.1.0/lib/digest.bundle' (mach-o file, but is an incompatible architecture (have 'x
86_64', need 'arm64e')), '/usr/local/lib/digest.bundle' (no such file), '/usr/lib/digest.bundle' (no such file) - /Users
/dangerrg/.rvm/gems/ruby-3.0.3/gems/digest-3.1.0/lib/digest.bundle (LoadError)

I'm running on Mac M1:

About your application's environment
Rails version             7.0.0
Ruby version              ruby 3.0.3p157 (2021-11-24 revision 3fb7d2cadc) [arm64-darwin21]
RubyGems version          3.2.32
Rack version              2.2.3
Middleware                ActionDispatch::HostAuthorization, Rack::Sendfile, ActionDispatch::Static, ActionDispatch::Exe
cutor, ActionDispatch::ServerTiming, ActiveSupport::Cache::Strategy::LocalCache::Middleware, Rack::Runtime, Rack::Method
Override, ActionDispatch::RequestId, ActionDispatch::RemoteIp, Sprockets::Rails::QuietAssets, Rails::Rack::Logger, Actio
nDispatch::ShowExceptions, WebConsole::Middleware, ActionDispatch::DebugExceptions, ActionDispatch::ActionableExceptions
, ActionDispatch::Reloader, ActionDispatch::Callbacks, ActiveRecord::Migration::CheckPending, ActionDispatch::Cookies, A
ctionDispatch::Session::CookieStore, ActionDispatch::Flash, ActionDispatch::ContentSecurityPolicy::Middleware, ActionDis
patch::PermissionsPolicy::Middleware, Rack::Head, Rack::ConditionalGet, Rack::ETag, Rack::TempfileReaper
Application root          /Users/dangerrg/Documents/rails_7_projects/rails_7_blog
Environment               development
Database adapter          sqlite3
Database schema version   20220105220427

[nickmealey]

Updates on this issue? It appears to be ongoing and I'm only able to bundle with bundle config set force_ruby_platform true

[JasonLunn]

Similar to #8682, closing this because in version 3.22 protobuf is planning to experiment with releasing only a source gem.

If this experiment is unsuccessful and we revert to building binary gems for some platforms, we'll evaluate whether it makes sense to add arm64 darwin at that time.

CC: @deannagarcia @haberman

[njmulsqb]

I am running this on M2, facing the same issue

``require': dlopen(/Library/Ruby/Gems/2.6.0/gems/google-protobuf-3.21.12-x86_64-darwin/lib/google/2.6/protobuf_c.bundle, 0x0009): tried: '/Library/Ruby/Gems/2.6.0/gems/google-protobuf-3.21.12-x86_64-darwin/lib/google/2.6/protobuf_c.bundle' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Library/Ruby/Gems/2.6.0/gems/google-protobuf-3.21.12-x86_64-darwin/lib/google/2.6/protobuf_c.bundle' (no such file), '/Library/Ruby/Gems/2.6.0/gems/google-protobuf-3.21.12-x86_64-darwin/lib/google/2.6/protobuf_c.bundle' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')) - /Library/Ruby/Gems/2.6.0/gems/google-protobuf-3.21.12-x86_64-darwin/lib/google/2.6/protobuf_c.bundle (LoadError)` 

dont know what to do?

[njmulsqb]

I am running this on M2, facing the same issue

``require': dlopen(/Library/Ruby/Gems/2.6.0/gems/google-protobuf-3.21.12-x86_64-darwin/lib/google/2.6/protobuf_c.bundle, 0x0009): tried: '/Library/Ruby/Gems/2.6.0/gems/google-protobuf-3.21.12-x86_64-darwin/lib/google/2.6/protobuf_c.bundle' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Library/Ruby/Gems/2.6.0/gems/google-protobuf-3.21.12-x86_64-darwin/lib/google/2.6/protobuf_c.bundle' (no such file), '/Library/Ruby/Gems/2.6.0/gems/google-protobuf-3.21.12-x86_64-darwin/lib/google/2.6/protobuf_c.bundle' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')) - /Library/Ruby/Gems/2.6.0/gems/google-protobuf-3.21.12-x86_64-darwin/lib/google/2.6/protobuf_c.bundle (LoadError)` 

dont know what to do?

I resolved the issue by ditching homebrew and following the ruby installation docs provided on Jekyll site

[Hazel233]

solved on my new M1 Mac, try the following steps:

1. gem uninstall google-protobuf, and remove all google-protobuf
2. reinstall with gem install google-protobuf -v 3.15.8 --platform=ruby or any version you want

seems bundle config force ruby platform not working, we have to reinstall google-protobuf platform: ruby from scratch

This really helps after searching for a lot methods.

[haberman]

Protobuf is releasing with arm64-darwin binary gems. For example, here is the latest one: https://rubygems.org/gems/google-protobuf/versions/3.24.2-arm64-darwin

However, the current gems require Ruby >=2.7, and I believe macOS is still shipping with Ruby 2.6.

So to use the current version of the protobuf package, you would need to install a separate Ruby from Homebrew (or from source).