search

socketry / nio4r

Cross-platform asynchronous I/O primitives for scalable network clients and servers.
Other
970 stars 86 forks source link

Can't load C ext on Apple M1 #259

Closed jasl closed 3 years ago

jasl commented 3 years ago 
Traceback (most recent call last):
    23: from /usr/local/bin/bacon:23:in `<main>'
    22: from /usr/local/bin/bacon:23:in `load'
    21: from /Library/Ruby/Gems/2.6.0/gems/bacon-1.2.0/bin/bacon:115:in `<top (required)>'
    20: from /Library/Ruby/Gems/2.6.0/gems/bacon-1.2.0/bin/bacon:115:in `each'
    19: from /Library/Ruby/Gems/2.6.0/gems/bacon-1.2.0/bin/bacon:116:in `block in <top (required)>'
    18: from /Library/Ruby/Gems/2.6.0/gems/bacon-1.2.0/bin/bacon:116:in `load'
    17: from spec/specification_spec.rb:1:in `<top (required)>'
    16: from spec/specification_spec.rb:1:in `require'
    15: from /Users/jasl/Workspaces/Ruby/Core/spec/spec_helper.rb:60:in `<top (required)>'
    14: from /Library/Ruby/Gems/2.6.0/gems/activesupport-6.1.1/lib/active_support/core_ext/kernel/reporting.rb:15:in `silence_warnings'
    13: from /Library/Ruby/Gems/2.6.0/gems/activesupport-6.1.1/lib/active_support/core_ext/kernel/reporting.rb:28:in `with_warnings'
    12: from /Library/Ruby/Gems/2.6.0/gems/activesupport-6.1.1/lib/active_support/core_ext/kernel/reporting.rb:15:in `block in silence_warnings'
    11: from /Users/jasl/Workspaces/Ruby/Core/spec/spec_helper.rb:61:in `block in <top (required)>'
    10: from /Users/jasl/Workspaces/Ruby/Core/spec/spec_helper.rb:61:in `require'
     9: from /Users/jasl/Workspaces/Ruby/Core/lib/cocoapods-core/cdn_source.rb:6:in `<top (required)>'
     8: from /Users/jasl/Workspaces/Ruby/Core/lib/cocoapods-core/cdn_source.rb:6:in `require'
     7: from /Library/Ruby/Gems/2.6.0/gems/async-1.28.3/lib/async.rb:25:in `<top (required)>'
     6: from /Library/Ruby/Gems/2.6.0/gems/async-1.28.3/lib/async.rb:25:in `require_relative'
     5: from /Library/Ruby/Gems/2.6.0/gems/async-1.28.3/lib/async/reactor.rb:25:in `<top (required)>'
     4: from /Library/Ruby/Gems/2.6.0/gems/async-1.28.3/lib/async/reactor.rb:25:in `require_relative'
     3: from /Library/Ruby/Gems/2.6.0/gems/async-1.28.3/lib/async/wrapper.rb:23:in `<top (required)>'
     2: from /Library/Ruby/Gems/2.6.0/gems/async-1.28.3/lib/async/wrapper.rb:23:in `require'
     1: from /Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio.rb:23:in `<top (required)>'
/Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio.rb:23:in `require': dlopen(/Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio4r_ext.bundle, 0x0009): missing compatible arch in /Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio4r_ext.bundle - /Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio4r_ext.bundle (LoadError)

BTW, NIO4R_PURE=true works

jasl commented 3 years ago

In addition, I found custom build Ruby on M1 Mac can load c ext properly, but in some cases, it will crash, I don't know how to dig this, so I push a crash log here cocoapods.log

Maybe we have to force to use pure Ruby fallback on M1 Mac

ioquatix commented 3 years ago

missing compatible arch looks like it's not being compiled correctly.

ioquatix commented 3 years ago

@nobu told me:

A Rubygem's issue. Rubygems puts the last compiled bundle in the architecture neutral path (/Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio4r_ext.bundle). The correct bundle should be another architecture dependent path, if it has been compiled.

@hsbt said:

Do not use system ruby

@jasl Can you check this advice and let me know if it helps?

jasl commented 3 years ago

@nobu told me:

A Rubygem's issue. Rubygems puts the last compiled bundle in the architecture neutral path (/Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio4r_ext.bundle). The correct bundle should be another architecture dependent path, if it has been compiled.

@hsbt said:

Do not use system ruby

@jasl Can you check this advice and let me know if it helps?

I'm afraid these won't help...

  1. This issue was found by CocoaPods, this tool is aim for iOS developers, most of them don't have motivation to install Ruby...
  2. I know the M1 macOS bundled Ruby looks very strange, it can be treated as both x86 and arm64

For missing compatible arch I suggest CocoaPods set NIO4R_PURE=true, and it works well

BUT there is another problem,

In addition, I found custom build Ruby on M1 Mac can load c ext properly, but in some cases, it will crash, I don't know how to dig this, so I push a crash log here cocoapods.log

This case also sovled by set NIO4R_PURE=true.

But it made me concern: previously, I was only thought ffi has compatibility issue with M1 macOS, now nio4r too, maybe more c-ext gems have protential compatibility issue.

ioquatix commented 3 years ago

@jasl why can't we build fat binary?

MSP-Greg commented 3 years ago

Question from a 'macOS' challenged user - can one build 'ARM-based macOS' in 'the cloud'? More importantly, run CI with it?

jasl commented 3 years ago

@jasl why can't we build fat binary?

This would be better, but it also requires MRI fat binary too? now only macOS system Ruby is fat binary, is it easy to made MRI become fat binary?

jasl commented 3 years ago

Question from a 'macOS' challenged user - can one build 'ARM-based macOS' in 'the cloud'? More importantly, run CI with it?

I'm afraid not, first macOS isn't opensource... second the License issue, we have to waiting AWS/Azure/TravisCI update their Mac Mini to M1 Mac...

AFAIK, HomeBrew team got some M1 Macs they're running ARM-related jobs locally, Someone sponsored CocoaPods a M1 Mac Mini as CI via BuildKite

ioquatix commented 3 years ago

If someone wants to sponsor a Mac Mini M1 I will set up a build server for anyone in the Ruby community who wants to use it via GitHub actions (remote instance) within reason.

jasl commented 3 years ago

If someone wants to sponsor a Mac Mini M1 I will set up a build server for anyone in the Ruby community who wants to use it via GitHub actions (remote instance) within reason.

I just sponsor you from GitHub sponsor for $100 a month for 1 year, so it should be $1200 in total that should enough to buy a 16G memory 512G SSD M1 Mac Mini!

Please help to ensure Ruby can working well with M1 macOS.

BTW, I use GH sponsor becasue I don't have a easy way to transfer you the money.

ioquatix commented 3 years ago

Hey, that's amazing. Thanks for your contribution. I'll get the hardware ordered tomorrow!

ioquatix commented 3 years ago

Okay, I was excited, so I ordered it already. It will take 2-3 weeks to arrive so please be patient.

bryanp commented 3 years ago

FWIW it seems to work for me on an M1 Mini:

~ $ ruby -v
ruby 3.0.0p0 (2020-12-25 revision 95aff21468) [arm64-darwin20]
~ $ gem install nio4r
Fetching nio4r-2.5.4.gem
Building native extensions. This could take a while...
Successfully installed nio4r-2.5.4
1 gem installed
~ $ irb
irb(main):001:0> require "nio"
=> true

I built Ruby using ruby-install, installed via homebrew.

@jasl Are you doing anything other than require "nio" to cause the failure you're seeing? Any other tips for reproducing?

jasl commented 3 years ago

FWIW it seems to work for me on an M1 Mini:

~ $ ruby -v
ruby 3.0.0p0 (2020-12-25 revision 95aff21468) [arm64-darwin20]
~ $ gem install nio4r
Fetching nio4r-2.5.4.gem
Building native extensions. This could take a while...
Successfully installed nio4r-2.5.4
1 gem installed
~ $ irb
irb(main):001:0> require "nio"
=> true

I built Ruby using ruby-install, installed via homebrew.

@jasl Are you doing anything other than require "nio" to cause the failure you're seeing? Any other tips for reproducing?

this issue only occurred on System Ruby, not affect on self-build Ruby

tarcieri commented 3 years ago

@jasl you said there were crashes here: https://github.com/socketry/nio4r/issues/259#issuecomment-763549608

Can you reproduce any of those?

jasl commented 3 years ago

@jasl you said there were crashes here: #259 (comment)

Can you reproduce any of those?

Just re-confirm the crash still happens, I push a reproducible branch https://github.com/jasl/CocoaPods/tree/m1-crash-case

Step (on a M1 Mac)

tarcieri commented 3 years ago

@jasl thank you!

ioquatix commented 3 years ago

In nio4r repository, on M1 Mac, I tried to build native extension, and it did seem to correctly build fat binary:

otool -fah nio4r_ext.bundle 
Fat headers
fat_magic 0xcafebabe
nfat_arch 2
architecture 0
    cputype 16777223
    cpusubtype 3
    capabilities 0x0
    offset 16384
    size 16432
    align 2^14 (16384)
architecture 1
    cputype 16777228
    cpusubtype 2
    capabilities 0x0
    offset 49152
    size 16738
    align 2^14 (16384)
nio4r_ext.bundle (architecture x86_64):
Mach header
      magic  cputype cpusubtype  caps    filetype ncmds sizeofcmds      flags
 0xfeedfacf 16777223          3  0x00           8    12        648 0x00000085
nio4r_ext.bundle (architecture arm64e):
Mach header
      magic  cputype cpusubtype  caps    filetype ncmds sizeofcmds      flags
 0xfeedfacf 16777228          2  0x00           8    14        648 0x00000085

All tests passed (bundle exec rspec).

So, I'll try to reproduce CocoaPods issue using only native Ruby.

ioquatix commented 3 years ago

Okay, so I tried m1-crash-branch of CocoaPods as outlined above.

I installed gems by deleting Gemfile.lock (don't have bundler 2 on native Ruby).

Then, checkout all submodule using git submodule update --init --recursive

I then did bundle install.

Finally try to run tests with bundle exec rake:

bundler: failed to load command: bacon (/Users/samuel/Documents/Programming/tmp/CocoaPods/vendor/bundle/ruby/2.6.0/bin/bacon)
LoadError: dlopen(/Users/samuel/Documents/Programming/tmp/CocoaPods/vendor/bundle/ruby/2.6.0/gems/bigdecimal-1.3.5/lib/bigdecimal.bundle, 0x0009): missing compatible arch in /Users/samuel/Documents/Programming/tmp/CocoaPods/vendor/bundle/ruby/2.6.0/gems/bigdecimal-1.3.5/lib/bigdecimal.bundle - /Users/samuel/Documents/Programming/tmp/CocoaPods/vendor/bundle/ruby/2.6.0/gems/bigdecimal-1.3.5/lib/bigdecimal.bundle

Strangely, I checked this binary and it does appear to have the right architectures:

samuel@Motoko ~/D/P/t/CocoaPods (m1-crash-case) [1]> otool -fah /Users/samuel/Documents/Programming/tmp/CocoaPods/vendor/bundle/ruby/2.6.0/gems/bigdecimal-1.3.5/lib/bigdecimal.bundle
Fat headers
fat_magic 0xcafebabe
nfat_arch 2
architecture 0
    cputype 16777223
    cpusubtype 3
    capabilities 0x0
    offset 16384
    size 16432
    align 2^14 (16384)
architecture 1
    cputype 16777228
    cpusubtype 2
    capabilities 0x0
    offset 49152
    size 16738
    align 2^14 (16384)
/Users/samuel/Documents/Programming/tmp/CocoaPods/vendor/bundle/ruby/2.6.0/gems/bigdecimal-1.3.5/lib/bigdecimal.bundle (architecture x86_64):
Mach header
      magic  cputype cpusubtype  caps    filetype ncmds sizeofcmds      flags
 0xfeedfacf 16777223          3  0x00           8    12        648 0x00000085
/Users/samuel/Documents/Programming/tmp/CocoaPods/vendor/bundle/ruby/2.6.0/gems/bigdecimal-1.3.5/lib/bigdecimal.bundle (architecture arm64e):
Mach header
      magic  cputype cpusubtype  caps    filetype ncmds sizeofcmds      flags
 0xfeedfacf 16777228          2  0x00           8    14        648 0x00000085
ioquatix commented 3 years ago

I tried to look for system library but it also seems okay:

samuel@Motoko ~/D/P/t/CocoaPods (m1-crash-case) [1]> otool -fah /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/universal-darwin20/bigdecimal.bundle
Fat headers
fat_magic 0xcafebabe
nfat_arch 2
architecture 0
    cputype 16777223
    cpusubtype 3
    capabilities 0x0
    offset 16384
    size 94816
    align 2^14 (16384)
architecture 1
    cputype 16777228
    cpusubtype 2
    capabilities 0x80
    offset 114688
    size 110976
    align 2^14 (16384)
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/universal-darwin20/bigdecimal.bundle (architecture x86_64):
Mach header
      magic  cputype cpusubtype  caps    filetype ncmds sizeofcmds      flags
 0xfeedfacf 16777223          3  0x00           8    15       1688 0x00000085
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/universal-darwin20/bigdecimal.bundle (architecture arm64e):
Mach header
      magic  cputype cpusubtype  caps    filetype ncmds sizeofcmds      flags
 0xfeedfacf 16777228          2  0x80           8    16       1672 0x00000085
ioquatix commented 3 years ago

I found some useful discussion here: https://github.com/rubygems/rubygems/issues/4234

I can definitely reproduce part of the issue:

samuel@Motoko ~ [1]> sudo gem install nio4r --version 2.5.4
Fetching nio4r-2.5.4.gem
Building native extensions. This could take a while...
Successfully installed nio4r-2.5.4
Parsing documentation for nio4r-2.5.4
Installing ri documentation for nio4r-2.5.4
Done installing documentation for nio4r after 0 seconds
1 gem installed
samuel@Motoko ~> irb

WARNING: This version of ruby is included in macOS for compatibility with legacy software. 
In future versions of macOS the ruby runtime will not be available by 
default, and may require you to install an additional package.

irb(main):001:0> require 'nio'
Traceback (most recent call last):
       10: from /usr/bin/irb:23:in `<main>'
        9: from /usr/bin/irb:23:in `load'
        8: from /Library/Ruby/Gems/2.6.0/gems/irb-1.0.0/exe/irb:11:in `<top (required)>'
        7: from (irb):1
        6: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:34:in `require'
        5: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:130:in `rescue in require'
        4: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:130:in `require'
        3: from /Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio.rb:23:in `<top (required)>'
        2: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        1: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
LoadError (dlopen(/Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio4r_ext.bundle, 0x0009): missing compatible arch in /Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio4r_ext.bundle - /Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio4r_ext.bundle)
ioquatix commented 3 years ago

When I checked, the extension is compiled correctly:

> lipo -archs /Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio4r_ext.bundle
x86_64 arm64e

But what's even more odd:

> arch -x86_64 irb

WARNING: This version of ruby is included in macOS for compatibility with legacy software. 
In future versions of macOS the ruby runtime will not be available by 
default, and may require you to install an additional package.

irb(main):001:0> require 'nio'
Traceback (most recent call last):
       10: from /usr/bin/irb:23:in `<main>'
        9: from /usr/bin/irb:23:in `load'
        8: from /Library/Ruby/Gems/2.6.0/gems/irb-1.0.0/exe/irb:11:in `<top (required)>'
        7: from (irb):1
        6: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:34:in `require'
        5: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:130:in `rescue in require'
        4: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:130:in `require'
        3: from /Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio.rb:23:in `<top (required)>'
        2: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        1: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
LoadError (dlsym(0x7ffdaa474a80, Init_nio4r_ext): symbol not found - /Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio4r_ext.bundle)
> arch -arm64e irb

WARNING: This version of ruby is included in macOS for compatibility with legacy software. 
In future versions of macOS the ruby runtime will not be available by 
default, and may require you to install an additional package.

irb(main):001:0> require 'nio'
Traceback (most recent call last):
       10: from /usr/bin/irb:23:in `<main>'
        9: from /usr/bin/irb:23:in `load'
        8: from /Library/Ruby/Gems/2.6.0/gems/irb-1.0.0/exe/irb:11:in `<top (required)>'
        7: from (irb):1
        6: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:34:in `require'
        5: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:130:in `rescue in require'
        4: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:130:in `require'
        3: from /Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio.rb:23:in `<top (required)>'
        2: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        1: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
LoadError (dlopen(/Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio4r_ext.bundle, 0x0009): missing compatible arch in /Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio4r_ext.bundle - /Library/Ruby/Gems/2.6.0/gems/nio4r-2.5.4/lib/nio4r_ext.bundle)
ioquatix commented 3 years ago

Okay, I think I found the problem: 

> nm nio4r_ext.bundle

nio4r_ext.bundle (for architecture x86_64):
                 U dyld_stub_binder

nio4r_ext.bundle (for architecture arm64e):
no symbols

There are no symbols in the native extension :p

ioquatix commented 3 years ago 
linking shared-object nio4r_ext.bundle
ld: warning: directory not found for option '-L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.2.Internal.sdk/usr/local/lib'
ld: warning: ignoring file monitor.o, building for macOS-arm64e but attempting to link with file built for unknown-arm64
ld: warning: ignoring file bytebuffer.o, building for macOS-arm64e but attempting to link with file built for unknown-arm64
ld: warning: ignoring file nio4r_ext.o, building for macOS-arm64e but attempting to link with file built for unknown-arm64
ld: warning: ignoring file selector.o, building for macOS-arm64e but attempting to link with file built for unknown-arm64
ld: warning: directory not found for option '-L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.2.Internal.sdk/usr/local/lib'
ld: warning: ignoring file bytebuffer.o, building for macOS-x86_64 but attempting to link with file built for unknown-arm64
ld: warning: ignoring file monitor.o, building for macOS-x86_64 but attempting to link with file built for unknown-arm64
ld: warning: ignoring file nio4r_ext.o, building for macOS-x86_64 but attempting to link with file built for unknown-arm64
ld: warning: ignoring file selector.o, building for macOS-x86_64 but attempting to link with file built for unknown-arm64

Linking errors.

ioquatix commented 3 years ago

Okay, I found the DLDFLAGS are broken at least for M1.

Adding the following seems to make it build correctly:

# extconf.rb

# ... snip ...

$DLDFLAGS.gsub!(/\-arch\s+[^\s]+/, "")

dir_config "nio4r_ext"
create_makefile "nio4r_ext"

The problem seems to be that the linker gets confused what to do when you specify the arch flags.

ioquatix commented 3 years ago

This is now released and I tested it locally.