apache / arrow

Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing
https://arrow.apache.org/
Apache License 2.0
13.99k stars 3.41k forks source link

[Release][Ruby] Investigate why we require to install libffi manually on a new M2 to run verification #40195

Open raulcd opened 5 months ago

raulcd commented 5 months ago

Describe the bug, including details regarding any error messages, version, and platform.

As discussed here: https://github.com/apache/arrow/pull/39960#discussion_r1491520898

I did require to add:

   # libffi required for gobject-introspection gem
   brew install libffi

In order to run verification successfully on a new macOS ARM (M2). Otherwise we got the following error:

Resolving dependencies...
Fetching power_assert 2.0.3
Fetching native-package-installer 1.1.9
Fetching pkg-config 1.5.6
Installing power_assert 2.0.3
Installing native-package-installer 1.1.9
Installing pkg-config 1.5.6
Fetching test-unit 3.6.2
Fetching glib2 4.2.0
Installing test-unit 3.6.2
Installing glib2 4.2.0 with native extensions
Fetching gobject-introspection 4.2.0
Installing gobject-introspection 4.2.0 with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

current directory:
/private/var/folders/rr/t_15q7dx4_lf1nv0btypffcw0000gn/T/arrow-15.0.0.XXXXX.li0gqOMB3O/apache-arrow-15.0.0/c_glib/vendor/bundle/ruby/3.3.0/gems/gobject-introspection-4.2.0/ext/gobject-introspection
/opt/homebrew/opt/ruby/bin/ruby extconf.rb
checking for --enable-debug-build option... no
checking for -Wall option to compiler... yes
checking for -Wcast-align option to compiler... yes
checking for -Wextra option to compiler... yes
checking for -Wformat=2 option to compiler... yes
checking for -Winit-self option to compiler... yes
checking for -Wlarger-than-65500 option to compiler... yes
checking for -Wmissing-declarations option to compiler... yes
checking for -Wmissing-format-attribute option to compiler... yes
checking for -Wmissing-include-dirs option to compiler... yes
checking for -Wmissing-noreturn option to compiler... yes
checking for -Wmissing-prototypes option to compiler... yes
checking for -Wnested-externs option to compiler... yes
checking for -Wold-style-definition option to compiler... yes
checking for -Wpacked option to compiler... yes
checking for -Wp,-D_FORTIFY_SOURCE=2 option to compiler... yes
checking for -Wpointer-arith option to compiler... yes
checking for -Wundef option to compiler... yes
checking for -Wout-of-line-declaration option to compiler... yes
checking for -Wunsafe-loop-optimizations option to compiler... no
checking for -Wwrite-strings option to compiler... yes
checking for Homebrew... yes
checking for gobject-introspection-1.0... yes (1.78.1)
creating rbgiversion.h
creating gobject-introspection-enum-types.c
creating gobject-introspection-enum-types.h
creating ruby-gobject-introspection.pc
creating Makefile

current directory:
/private/var/folders/rr/t_15q7dx4_lf1nv0btypffcw0000gn/T/arrow-15.0.0.XXXXX.li0gqOMB3O/apache-arrow-15.0.0/c_glib/vendor/bundle/ruby/3.3.0/gems/gobject-introspection-4.2.0/ext/gobject-introspection
make DESTDIR\= sitearchdir\=./.gem.20240216-45788-684atu sitelibdir\=./.gem.20240216-45788-684atu clean

current directory:
/private/var/folders/rr/t_15q7dx4_lf1nv0btypffcw0000gn/T/arrow-15.0.0.XXXXX.li0gqOMB3O/apache-arrow-15.0.0/c_glib/vendor/bundle/ruby/3.3.0/gems/gobject-introspection-4.2.0/ext/gobject-introspection
make DESTDIR\= sitearchdir\=./.gem.20240216-45788-684atu sitelibdir\=./.gem.20240216-45788-684atu
compiling gobject-introspection-enum-types.c
compiling rb-gi-arg-info.c
In file included from rb-gi-arg-info.c:21:
In file included from ./rb-gi-private.h:24:
In file included from
/private/var/folders/rr/t_15q7dx4_lf1nv0btypffcw0000gn/T/arrow-15.0.0.XXXXX.li0gqOMB3O/apache-arrow-15.0.0/c_glib/vendor/bundle/ruby/3.3.0/gems/glib2-4.2.0/ext/glib2/../../lib/rbgobject.h:27:
In file included from
/private/var/folders/rr/t_15q7dx4_lf1nv0btypffcw0000gn/T/arrow-15.0.0.XXXXX.li0gqOMB3O/apache-arrow-15.0.0/c_glib/vendor/bundle/ruby/3.3.0/gems/glib2-4.2.0/ext/glib2/../../lib/rbgutil.h:26:
In file included from /opt/homebrew/Cellar/ruby/3.3.0/include/ruby-3.3.0/ruby/encoding.h:22:
/opt/homebrew/Cellar/ruby/3.3.0/include/ruby-3.3.0/ruby/internal/encoding/ctype.h:82:46: warning: unused parameter 'enc' [-Wunused-parameter]
rb_enc_isascii(OnigCodePoint c, rb_encoding *enc)
                                             ^
In file included from rb-gi-arg-info.c:21:
In file included from ./rb-gi-private.h:28:
/opt/homebrew/Cellar/gobject-introspection/1.78.1/include/gobject-introspection-1.0/girffi.h:25:10: fatal error: 'ffi.h' file not found
#include <ffi.h>
         ^~~~~~~
1 warning and 1 error generated.
make: *** [rb-gi-arg-info.o] Error 1

make failed, exit code 2

Gem files will remain installed in
/private/var/folders/rr/t_15q7dx4_lf1nv0btypffcw0000gn/T/arrow-15.0.0.XXXXX.li0gqOMB3O/apache-arrow-15.0.0/c_glib/vendor/bundle/ruby/3.3.0/gems/gobject-introspection-4.2.0 for inspection.
Results logged to
/private/var/folders/rr/t_15q7dx4_lf1nv0btypffcw0000gn/T/arrow-15.0.0.XXXXX.li0gqOMB3O/apache-arrow-15.0.0/c_glib/vendor/bundle/ruby/3.3.0/extensions/arm64-darwin-23/3.3.0/gobject-introspection-4.2.0/gem_make.out

  /opt/homebrew/Cellar/ruby/3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:125:in `run'
  /opt/homebrew/Cellar/ruby/3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:51:in `block in make'
  /opt/homebrew/Cellar/ruby/3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:43:in `each'
  /opt/homebrew/Cellar/ruby/3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:43:in `make'
  /opt/homebrew/Cellar/ruby/3.3.0/lib/ruby/3.3.0/rubygems/ext/ext_conf_builder.rb:42:in `build'
  /opt/homebrew/Cellar/ruby/3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:193:in `build_extension'
  /opt/homebrew/Cellar/ruby/3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:227:in `block in build_extensions'
  /opt/homebrew/Cellar/ruby/3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:224:in `each'
  /opt/homebrew/Cellar/ruby/3.3.0/lib/ruby/3.3.0/rubygems/ext/builder.rb:224:in `build_extensions'
  /opt/homebrew/Cellar/ruby/3.3.0/lib/ruby/3.3.0/rubygems/installer.rb:852:in `build_extensions'
  /opt/homebrew/Cellar/ruby/3.3.0/lib/ruby/3.3.0/bundler/rubygems_gem_installer.rb:76:in `build_extensions'
  /opt/homebrew/Cellar/ruby/3.3.0/lib/ruby/3.3.0/bundler/rubygems_gem_installer.rb:28:in `install'
  /opt/homebrew/Cellar/ruby/3.3.0/lib/ruby/3.3.0/bundler/source/rubygems.rb:205:in `install'
  /opt/homebrew/Cellar/ruby/3.3.0/lib/ruby/3.3.0/bundler/installer/gem_installer.rb:54:in `install'
  /opt/homebrew/Cellar/ruby/3.3.0/lib/ruby/3.3.0/bundler/installer/gem_installer.rb:16:in `install_from_spec'
  /opt/homebrew/Cellar/ruby/3.3.0/lib/ruby/3.3.0/bundler/installer/parallel_installer.rb:132:in `do_install'
  /opt/homebrew/Cellar/ruby/3.3.0/lib/ruby/3.3.0/bundler/installer/parallel_installer.rb:123:in `block in worker_pool'
  /opt/homebrew/Cellar/ruby/3.3.0/lib/ruby/3.3.0/bundler/worker.rb:62:in `apply_func'
  /opt/homebrew/Cellar/ruby/3.3.0/lib/ruby/3.3.0/bundler/worker.rb:57:in `block in process_queue'
  <internal:kernel>:187:in `loop'
  /opt/homebrew/Cellar/ruby/3.3.0/lib/ruby/3.3.0/bundler/worker.rb:54:in `process_queue'
  /opt/homebrew/Cellar/ruby/3.3.0/lib/ruby/3.3.0/bundler/worker.rb:90:in `block (2 levels) in create_threads'

An error occurred while installing gobject-introspection (4.2.0), and Bundler cannot continue.

In Gemfile:
  gobject-introspection
Failed to verify release candidate. See /var/folders/rr/t_15q7dx4_lf1nv0btypffcw0000gn/T/arrow-15.0.0.XXXXX.li0gqOMB3O for details.

Some info on the same comment thread:

% pkg-config --cflags gobject-introspection-1.0
-I/opt/homebrew/Cellar/gobject-introspection/1.78.1/include/gobject-introspection-1.0 -I/opt/homebrew/Cellar/glib/2.78.4/include -I/opt/homebrew/Cellar/glib/2.78.4/include/glib-2.0 -I/opt/homebrew/Cellar/glib/2.78.4/lib/glib-2.0/include -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/Cellar/pcre2/10.42/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/include/ffi

raul@raul-MacBook-Pro % ls -l /Library/Developer/CommandLineTools/SDKs/
total 0
drwxr-xr-x  7 root  wheel  224 Nov 19  2021 MacOSX12.1.sdk
drwxr-xr-x  3 root  wheel   96 Feb 10 14:11 MacOSX12.3.sdk
raul@raul-MacBook-Pro % sw_vers -productVersion
14.3.1

After manually adding the link sudo ln -s MacOSX12.3.sdk /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk it still failed.

Component(s)

Release, Ruby

kou commented 5 months ago

Could you show the output of the following command line after you install pkg-config gem by PATH=$(brew --prefix ruby)/bin:$PATH gem install pkg-config?

PATH=$(brew --prefix ruby)/bin:$PATH ruby -r pkg-config -e 'p PKGConfig.cflags("gobject-introspection-1.0")'
kou commented 5 months ago

Could you also show the output of the following command line?

pkg-config --debug --cflags libffi
raulcd commented 4 months ago

Sorry @kou , I forgot about this:

raul@raul-MacBook-Pro release % PATH=$(brew --prefix ruby)/bin:$PATH ruby -r pkg-config -e 'p PKGConfig.cflags("gobject-introspection-1.0")'
"-I/opt/homebrew/Cellar/gobject-introspection/1.78.1/include/gobject-introspection-1.0 -I/opt/homebrew/Cellar/glib/2.78.4/include/glib-2.0 -I/opt/homebrew/Cellar/glib/2.78.4/lib/glib-2.0/include -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/Cellar/pcre2/10.42/include -I/opt/homebrew/Cellar/glib/2.78.4/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/include/ffi"
raul@raul-MacBook-Pro release % pkg-config --debug --cflags libffi
Error printing enabled by default due to use of output options besides --exists, --atleast/exact/max-version or --list-all. Value of --silence-errors: 0
Error printing enabled
Adding virtual 'pkg-config' package to list of known packages
Looking for package 'libffi'
Looking for package 'libffi-uninstalled'
Reading 'libffi' from file '/opt/homebrew/Library/Homebrew/os/mac/pkgconfig/12/libffi.pc'
Parsing package file '/opt/homebrew/Library/Homebrew/os/mac/pkgconfig/12/libffi.pc'
  line>homebrew_sdkroot=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk
 Variable declaration, 'homebrew_sdkroot' has value '/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk'
  line>prefix=${homebrew_sdkroot}/usr
 Variable declaration, 'prefix' has value '/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr'
  line>exec_prefix=/usr
 Variable declaration, 'exec_prefix' has value '/usr'
  line>libdir=${exec_prefix}/lib
 Variable declaration, 'libdir' has value '/usr/lib'
  line>toolexeclibdir=${libdir}
 Variable declaration, 'toolexeclibdir' has value '/usr/lib'
  line>includedir=${prefix}/include/ffi
 Variable declaration, 'includedir' has value '/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/include/ffi'
  line>
  line>Name: libffi
  line>Description: Library supporting Foreign Function Interfaces
  line>Version: 3.4-rc1
  line>Libs: -L${toolexeclibdir} -lffi
  line>Cflags: -I${includedir}
Path position of 'libffi' is 5
Adding 'libffi' to list of known packages
Package libffi has -L /usr/lib in Libs
Removing -L /usr/lib from libs for libffi
 post-recurse: libffi
adding CFLAGS_OTHER string ""
 post-recurse: libffi
 original: libffi
   sorted: libffi
adding CFLAGS_I string "-I/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/include/ffi "
returning flags string "-I/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/include/ffi"
-I/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/include/ffi
raul@raul-MacBook-Pro release % 
kou commented 4 months ago

Thanks for remembering this!

It seems that detected -Is are expected. ffi.h may not exist in /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/include/ffi.

Could you show the output of the following command lines?

ls -lah /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/include
ls -lah /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/include/ffi