stanhu
commented
4 years ago Compiling with -fPIC did not seem to solve this problem.
Closed stanhu closed 4 years ago
stanhu
commented
4 years ago Compiling with -fPIC did not seem to solve this problem.
mudge
commented
4 years ago Hi @stanhu,
Thanks for reporting this.
I'm having trouble reproducing it. I tried the following on macOS 10.15.4 and Ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c):
url "https://github.com/google/re2/archive/2020-03-03.tar.gz"
version "20200303"
sha256 "04ee2aaebaa5038554683329afc494e684c30f82f2a1e47eb62450e59338f84d"
# ...
bottle do
cellar :any
sha256 "abb07cfb9def8d6f25a0d5694f0f23ba44b5b500f2d45d25d27515f62802c01b" => :catalina
sha256 "92c4603fad274003e71699376b20a40c6e57c13f878774a894e28a9ef73295c1" => :mojave
sha256 "58ac31c06c851bc5632c3a0703c0d41e55d8b015229cf03de09ee4a8701d92ef" => :high_sierra
endgem install re2pry:~> pry -rre2
[1] pry(main)> regexp = RE2::Regexp.new("{", log_errors: false)
=> #<RE2::Regexp /{/>
[2] pry(main)> puts regexp.error unless regexp.ok?
=> nilbrew upgrade re2: url "https://github.com/google/re2/archive/2020-04-01.tar.gz"
version "20200401"
sha256 "98794bc5416326817498384a9c43cbb5a406bab8da9f84f83c39ecad43ed5cea"
# ...
bottle do
cellar :any
sha256 "430e7efee518f5235ea75afc039168ce38c5407b525301199ccbc6c698300bb8" => :catalina
sha256 "67eed1f38907d1d7318b5d95f035ca18ff6186d85ee690560b8b1cd387f8afd4" => :mojave
sha256 "f111b7c906ab090d9db56e1f183f94d02e8cf5ce521e4ae4ff8e64d974cb5cbb" => :high_sierra
end~> pry -rre2
[1] pry(main)> regexp = RE2::Regexp.new("{", log_errors: false)
=> #<RE2::Regexp /{/>
[2] pry(main)> puts regexp.error unless regexp.ok?
=> nilLooking at re2.bundle's links, they seem consistent between versions:
~/.gem/ruby/2.6.5/gems/re2-1.2.0/ext/re2> otool -L re2.bundle
re2.bundle:
/usr/local/opt/re2/lib/libre2.0.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 902.1.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)I know there is a breaking change coming in re2 (removing some deprecated APIs) so the SONAME has changed from 6 to 7, even upgrading to HEAD with Homebrew, your commands still pass for me:
[1] pry(main)> regexp = RE2::Regexp.new("{", log_errors: false)
=> #<RE2::Regexp /{/>
[2] pry(main)> puts regexp.error unless regexp.ok?
=> nilThe bigger picture question here is what is reasonable behaviour for the gem when the underlying library is upgraded? Especially if some internal re2 API changes (as RE2::Match did several years ago), I'm not sure what the gem can do if the linked library has shifted under its feet. We could avoid depending on the external library altogether and bundle re2 inside the gem (as Nokogiri does with libxml2) but this seems like a major, breaking change to the behaviour of the library.
Hopefully there's some smaller issue we can fix here.
stanhu
commented
4 years ago You may need to rewind further to https://github.com/Homebrew/homebrew-core/blob/549a5c81c67e401fd3c75339157b3b936ebbbbed/Formula/re2.rb.
The bigger picture question here is what is reasonable behaviour for the gem when the underlying library is upgraded? Especially if some internal re2 API changes (as RE2::Match did several years ago), I'm not sure what the gem can do if the linked library has shifted under its feet. We could avoid depending on the external library altogether and bundle re2 inside the gem (as Nokogiri does with libxml2) but this seems like a major, breaking change to the behaviour of the library.
That's a good question. We have a check for ffi and eventmachine that fails if the require step fails. Could we make re2 just fail fast on require 're2' if the underlying library changes? That would at least make the problem easier to diagnose.
mudge
commented
4 years ago Ah, that did the trick: compiling the gem against re2 2020-01-01 and then upgrading to 2020-04-01 causes the segfault when I call RE2::Regexp#error.
Looking at the diff between the two versions I see that there was a breaking ABI change and Google changed the soname from 0 to 6. (I see the latest release has changed again from 6 to 7).
However, it looks like Homebrew explicitly changes the soname of the library to 0 regardless of ABI version. Removing the lines from the re2 Formula that change the dylib ID and add symlinks stops Homebrew masquerading libre2.6 and libre2.7 as libre2.0 and causes the gem to fail fast as soon as you require it:
# Against re 2020-01-01 without the dylib ID changing or symlinks
~> gem install re2
Fetching re2-1.2.0.gem
Building native extensions. This could take a while...
Successfully installed re2-1.2.0
1 gem installed
~/.gem/ruby/2.6.5/gems/re2-1.2.0/ext/re2> otool -L re2.bundle
re2.bundle:
/usr/local/opt/re2/lib/libre2.0.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 902.1.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)
# Upgrading to 2020-04-01 without the dylib ID changing or symlinks
~> brew upgrade re2
~/.gem/ruby/2.6.5/gems/re2-1.2.0/ext/re2> otool -L re2.bundle
re2.bundle:
/usr/local/opt/re2/lib/libre2.0.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 902.1.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)
# Now libre2.0.dylib is entirely missing
~> ls -la /usr/local/opt/re2/lib
total 2168
drwxr-xr-x 7 mudge admin 224 2 May 10:47 .
drwxr-xr-x 9 mudge admin 288 2 May 10:47 ..
-r--r--r-- 1 mudge admin 348556 2 May 10:47 libre2.6.0.0.dylib
lrwxr-xr-x 1 mudge admin 18 2 May 10:47 libre2.6.dylib -> libre2.6.0.0.dylib
-r--r--r-- 1 mudge admin 755048 2 May 10:47 libre2.a
lrwxr-xr-x 1 mudge admin 18 2 May 10:47 libre2.dylib -> libre2.6.0.0.dylib
drwxr-xr-x 3 mudge admin 96 2 May 10:47 pkgconfig
# Try requiring re2 linked against the now-missing re2 version
~/.gem/ruby/2.6.5/gems/re2-1.2.0/ext/re2> pry -rre2
Traceback (most recent call last):
9: from /Users/mudge/.gem/ruby/2.7.1/bin/pry:23:in `<main>'
8: from /Users/mudge/.gem/ruby/2.7.1/bin/pry:23:in `load'
7: from /Users/mudge/.gem/ruby/2.7.1/gems/pry-0.13.1/bin/pry:12:in `<top (required)>'
6: from /Users/mudge/.gem/ruby/2.7.1/gems/pry-0.13.1/lib/pry/cli.rb:89:in `parse_options'
5: from /Users/mudge/.gem/ruby/2.7.1/gems/pry-0.13.1/lib/pry/pry_class.rb:146:in `final_session_setup'
4: from /Users/mudge/.gem/ruby/2.7.1/gems/pry-0.13.1/lib/pry/pry_class.rb:104:in `load_requires'
3: from /Users/mudge/.gem/ruby/2.7.1/gems/pry-0.13.1/lib/pry/pry_class.rb:104:in `each'
2: from /Users/mudge/.gem/ruby/2.7.1/gems/pry-0.13.1/lib/pry/pry_class.rb:105:in `block in load_requires'
1: from /Users/mudge/.rubies/ruby-2.7.1/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'
/Users/mudge/.rubies/ruby-2.7.1/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require': cannot load such file -- re2 (LoadError)
13: from /Users/mudge/.gem/ruby/2.7.1/bin/pry:23:in `<main>'
12: from /Users/mudge/.gem/ruby/2.7.1/bin/pry:23:in `load'
11: from /Users/mudge/.gem/ruby/2.7.1/gems/pry-0.13.1/bin/pry:12:in `<top (required)>'
10: from /Users/mudge/.gem/ruby/2.7.1/gems/pry-0.13.1/lib/pry/cli.rb:89:in `parse_options'
9: from /Users/mudge/.gem/ruby/2.7.1/gems/pry-0.13.1/lib/pry/pry_class.rb:146:in `final_session_setup'
8: from /Users/mudge/.gem/ruby/2.7.1/gems/pry-0.13.1/lib/pry/pry_class.rb:104:in `load_requires'
7: from /Users/mudge/.gem/ruby/2.7.1/gems/pry-0.13.1/lib/pry/pry_class.rb:104:in `each'
6: from /Users/mudge/.gem/ruby/2.7.1/gems/pry-0.13.1/lib/pry/pry_class.rb:105:in `block in load_requires'
5: from /Users/mudge/.rubies/ruby-2.7.1/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:156:in `require'
4: from /Users/mudge/.rubies/ruby-2.7.1/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:168:in `rescue in require'
3: from /Users/mudge/.rubies/ruby-2.7.1/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:168:in `require'
2: from /Users/mudge/.gem/ruby/2.7.1/gems/re2-1.2.0/lib/re2.rb:6:in `<top (required)>'
1: from /Users/mudge/.rubies/ruby-2.7.1/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'
/Users/mudge/.rubies/ruby-2.7.1/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require': dlopen(/Users/mudge/.gem/ruby/2.7.1/gems/re2-1.2.0/lib/re2.bundle, 9): Library not loaded: /usr/local/opt/re2/lib/libre2.0.dylib (LoadError)
Referenced from: /Users/mudge/.gem/ruby/2.7.1/gems/re2-1.2.0/lib/re2.bundleLooking on Ubuntu, it seems the libre2-dev and libre2-5 package correctly suffix the shared libraries with the soname so I suspect this is a macOS and Homebrew issue only.
If I raise a pull request against Homebrew to stop overwriting the ABI version, this means the gem would fail fast if you try to require it when the underlying libre2 ABI version has changed and seems the most reasonable approach.
This wouldn't require any change to the gem itself.
What do you think?
mudge
commented
4 years ago This has now been merged into Homebrew core; could you please re-test with the latest formula (2020-05-01 revision 1) and let me know if it behaves as you expect?
Perhaps use the 2020-04-01 formula to install re2 with a soname of 0 (though it should be 6):
Then install and compile the gem against that version with gem install re2 and confirm it works as expected:
$ irb -rre2
> RE2('{').error
=> nilThen upgrade to the latest, fixed version of the re2 Formula (which has a soname of 7) and try to run the same command again:
~> irb -rre2
/Users/mudge/.rubies/ruby-2.7.1/lib/ruby/2.7.0/irb/init.rb:290: warning: LoadError: dlopen(/Users/mudge/.gem/ruby/2.7.1/gems/re2-1.2.0/lib/re2.bundle, 9): Library not loaded: /usr/local/opt/re2/lib/libre2.0.dylib
Referenced from: /Users/mudge/.gem/ruby/2.7.1/gems/re2-1.2.0/lib/re2.bundle
Reason: image not found - /Users/mudge/.gem/ruby/2.7.1/gems/re2-1.2.0/lib/re2.bundle@mudge Awesome, that worked for me as well. Thanks so much!
stanhu
commented
4 years ago Maybe I should create a separate issue, but does gem pristine re2 work for you now? I am seeing:
$ gem pristine re2
Restoring gems to pristine condition...
Building native extensions. This could take a while...
ERROR: While executing gem ... (Gem::Ext::BuildError)
ERROR: Failed to build gem native extension.
current directory: /Users/stanhu/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/re2-1.1.1/ext/re2
/Users/stanhu/.rbenv/versions/2.6.5/bin/ruby -I /Users/stanhu/.rbenv/versions/2.6.5/lib/ruby/2.6.0 -r ./siteconf20200502-7576-t86j84.rb extconf.rb
checking for -lstdc++... yes
checking for stdint.h... yes
checking for rb_str_sublen()... yes
checking for -lre2... yes
checking for re2 requires C++11 compiler... yes
checking for RE2::Match() with endpos argument... yes
creating Makefile
current directory: /Users/stanhu/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/re2-1.1.1/ext/re2
make "DESTDIR=" clean
current directory: /Users/stanhu/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/re2-1.1.1/ext/re2
make "DESTDIR="
compiling re2.cc
In file included from re2.cc:9:
In file included from /Users/stanhu/.rbenv/versions/2.6.5/include/ruby-2.6.0/ruby.h:33:
In file included from /Users/stanhu/.rbenv/versions/2.6.5/include/ruby-2.6.0/ruby/ruby.h:2111:
/Users/stanhu/.rbenv/versions/2.6.5/include/ruby-2.6.0/ruby/intern.h:56:19: warning: 'register' storage class specifier is deprecated and incompatible with C++17 [-Wdeprecated-register]
void rb_mem_clear(register VALUE*, register long);
^~~~~~~~~
/Users/stanhu/.rbenv/versions/2.6.5/include/ruby-2.6.0/ruby/intern.h:56:36: warning: 'register' storage class specifier is deprecated and incompatible with C++17 [-Wdeprecated-register]
void rb_mem_clear(register VALUE*, register long);
^~~~~~~~~
re2.cc:261:37: error: no member named 'utf8' in 're2::RE2::Options'
p->pattern->options().utf8() ? "UTF-8" : "ISO-8859-1"));
~~~~~~~~~~~~~~~~~~~~~ ^
re2.cc:37:36: note: expanded from macro 'ENCODED_STR_NEW'
int _enc = rb_enc_find_index(encoding); \
^~~~~~~~
re2.cc:459:35: error: no member named 'utf8' in 're2::RE2::Options'
p->pattern->options().utf8() ? "UTF-8" : "ISO-8859-1"));
~~~~~~~~~~~~~~~~~~~~~ ^
re2.cc:37:36: note: expanded from macro 'ENCODED_STR_NEW'
int _enc = rb_enc_find_index(encoding); \
^~~~~~~~
re2.cc:483:33: error: no member named 'utf8' in 're2::RE2::Options'
p->pattern->options().utf8() ? "UTF-8" : "ISO-8859-1");
~~~~~~~~~~~~~~~~~~~~~ ^
re2.cc:37:36: note: expanded from macro 'ENCODED_STR_NEW'
int _enc = rb_enc_find_index(encoding); \
^~~~~~~~
re2.cc:614:29: error: no member named 'utf8' in 're2::RE2::Options'
p->pattern->options().utf8() ? "UTF-8" : "ISO-8859-1");
~~~~~~~~~~~~~~~~~~~~~ ^
re2.cc:37:36: note: expanded from macro 'ENCODED_STR_NEW'
int _enc = rb_enc_find_index(encoding); \
^~~~~~~~
re2.cc:687:19: error: no member named 'set_utf8' in 're2::RE2::Options'
re2_options.set_utf8(RTEST(utf8));
~~~~~~~~~~~ ^
re2.cc:770:29: error: no member named 'utf8' in 're2::RE2::Options'
p->pattern->options().utf8() ? "UTF-8" : "ISO-8859-1");
~~~~~~~~~~~~~~~~~~~~~ ^
re2.cc:37:36: note: expanded from macro 'ENCODED_STR_NEW'
int _enc = rb_enc_find_index(encoding); \
^~~~~~~~
re2.cc:788:29: error: no member named 'utf8' in 're2::RE2::Options'
p->pattern->options().utf8() ? "UTF-8" : "ISO-8859-1");
~~~~~~~~~~~~~~~~~~~~~ ^
re2.cc:37:36: note: expanded from macro 'ENCODED_STR_NEW'
int _enc = rb_enc_find_index(encoding); \
^~~~~~~~
re2.cc:818:42: error: no member named 'utf8' in 're2::RE2::Options'
return BOOL2RUBY(p->pattern->options().utf8());
~~~~~~~~~~~~~~~~~~~~~ ^
re2.cc:21:23: note: expanded from macro 'BOOL2RUBY'
#define BOOL2RUBY(v) (v ? Qtrue : Qfalse)
^
re2.cc:1015:31: error: no member named 'utf8' in 're2::RE2::Options'
p->pattern->options().utf8() ? "UTF-8" : "ISO-8859-1");
~~~~~~~~~~~~~~~~~~~~~ ^
re2.cc:37:36: note: expanded from macro 'ENCODED_STR_NEW'
int _enc = rb_enc_find_index(encoding); \
^~~~~~~~
re2.cc:1046:39: error: no member named 'utf8' in 're2::RE2::Options'
BOOL2RUBY(p->pattern->options().utf8()));
~~~~~~~~~~~~~~~~~~~~~ ^
re2.cc:21:23: note: expanded from macro 'BOOL2RUBY'
#define BOOL2RUBY(v) (v ? Qtrue : Qfalse)
^
re2.cc:1116:33: error: no member named 'utf8' in 're2::RE2::Options'
p->pattern->options().utf8() ? "UTF-8" : "ISO-8859-1"),
~~~~~~~~~~~~~~~~~~~~~ ^
re2.cc:37:36: note: expanded from macro 'ENCODED_STR_NEW'
int _enc = rb_enc_find_index(encoding); \
^~~~~~~~
re2.cc:1287:31: error: no member named 'utf8' in 're2::RE2::Options'
p->pattern->options().utf8() ? "UTF-8" : "ISO-8859-1");
~~~~~~~~~~~~~~~~~~~~~ ^
re2.cc:37:36: note: expanded from macro 'ENCODED_STR_NEW'
int _enc = rb_enc_find_index(encoding); \
^~~~~~~~
re2.cc:1324:31: error: no member named 'utf8' in 're2::RE2::Options'
p->pattern->options().utf8() ? "UTF-8" : "ISO-8859-1");
~~~~~~~~~~~~~~~~~~~~~ ^
re2.cc:37:36: note: expanded from macro 'ENCODED_STR_NEW'
int _enc = rb_enc_find_index(encoding); \
^~~~~~~~
2 warnings and 13 errors generated.
make: *** [re2.o] Error 1
make failed, exit code 2
Gem files will remain installed in /Users/stanhu/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/re2-1.1.1 for inspection.
Results logged to /Users/stanhu/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/extensions/x86_64-darwin-19/2.6.0/re2-1.1.1/gem_make.outI think you need to update to re2 1.2.0 as 2020-05-01 removed the utf8 option (hence the soname bump): see https://github.com/mudge/re2/issues/40.
stanhu
commented
4 years ago @mudge Once again, thanks! That did the trick.
I'm not sure if this is an actual issue with the C extension or whether this just means
libre2needs to be compiled with-fPICfor this not to happen (will test this out). But whenever libre2 is updated (e.g. from v1.1 to v1.2 recently in https://github.com/Homebrew/homebrew-core/blob/40c227a5885561519827edecffd959f6316163d8/Formula/re2.rb#L4), it seems that re2 seg faults iferroris called. Runninggem pristine re2fixes the problem, but I'm wondering if this seg fault is something we can avoid. A simple script that reproduces the problem:Sometimes it fails:
An
objdump -dof thebundlefile shows: