Closed flavorjones closed 2 years ago
Hi @flavorjones,
Let me look into this! The rubysig.h
was to be deprecated for a while now, I suppose, it finally happened. Bear with me!
Update:
I need to check what else changed in Ruby 3.1, as where this branched into during the preprocessor pass should not really be possible on a modern Ruby.
Krzysztof
Hi @flavorjones,
Using your branch and my local Ruby 3.1.0 things seem to work perfectly as per:
I had to make sure nothing broke with this latest release of Ruby. Next, let me try the rake-compiler/rake-compiler-dock environment.
Also, we never had issues with this before Ruby 3.1, I believe.
Update:
No, rake-compiler-dock
version 1.1.0
which offers Ruby 3.0 works perfectly fine.
You should also check by running rake gem:x86_64-linux
and build in the rake-compiler-dock image. That ruby has different compile-time options than my ruby-build-installed ruby on my host (and so might yours?)
Hi @flavorjones,
You should also check by running
rake gem:x86_64-linux
and build in the rake-compiler-dock image. That ruby has different compile-time options than my ruby-build-installed ruby on my host (and so might yours?)
This is what I am doing at the moment. And there is a very curious thing about this Docker container. For example, the following works fine:
In this case, mkmf
had no issues locating the rb_thread_call_without_gvl()
function, which should be exported and available on any modern Ruby. However, the following that is using rake compile
fails and the aforementioned function cannot be found, as per:
I am not sure (yet) why the latter version does not work whereas former works fine. Perhaps a difference in paths? I can't imagine Rake and the rake-compiler
Ruby Gem settings something special, however, I need to confirm this.
Sorry for the delay with resolution!
Krzysztof
why the latter version does not work whereas former works fine
I believe the difference is between the rvm-installed rubies (just 2.5.9 and 3.1.0) used in the "working" scenario, and the rubies you're compiling against used in the "failing" scenario:
[flavorjones@454e95346e25 ruby]$ cat ~/.rake-compiler/config.yml
---
rbconfig-x86_64-linux-gnu-2.4.0: "/usr/local/rake-compiler/ruby/x86_64-redhat-linux/ruby-2.4.0/lib/ruby/2.4.0/x86_64-linux-gnu/rbconfig.rb"
rbconfig-x86_64-linux-2.4.0: "/usr/local/rake-compiler/ruby/x86_64-redhat-linux/ruby-2.4.0/lib/ruby/2.4.0/x86_64-linux-gnu/rbconfig.rb"
rbconfig-x86_64-linux-gnu-2.5.0: "/usr/local/rake-compiler/ruby/x86_64-redhat-linux/ruby-2.5.0/lib/ruby/2.5.0/x86_64-linux-gnu/rbconfig.rb"
rbconfig-x86_64-linux-2.5.0: "/usr/local/rake-compiler/ruby/x86_64-redhat-linux/ruby-2.5.0/lib/ruby/2.5.0/x86_64-linux-gnu/rbconfig.rb"
rbconfig-x86_64-linux-gnu-2.6.0: "/usr/local/rake-compiler/ruby/x86_64-redhat-linux/ruby-2.6.0/lib/ruby/2.6.0/x86_64-linux-gnu/rbconfig.rb"
rbconfig-x86_64-linux-2.6.0: "/usr/local/rake-compiler/ruby/x86_64-redhat-linux/ruby-2.6.0/lib/ruby/2.6.0/x86_64-linux-gnu/rbconfig.rb"
rbconfig-x86_64-linux-gnu-2.7.0: "/usr/local/rake-compiler/ruby/x86_64-redhat-linux/ruby-2.7.0/lib/ruby/2.7.0/x86_64-linux-gnu/rbconfig.rb"
rbconfig-x86_64-linux-2.7.0: "/usr/local/rake-compiler/ruby/x86_64-redhat-linux/ruby-2.7.0/lib/ruby/2.7.0/x86_64-linux-gnu/rbconfig.rb"
rbconfig-x86_64-linux-gnu-3.0.0: "/usr/local/rake-compiler/ruby/x86_64-redhat-linux/ruby-3.0.0/lib/ruby/3.0.0/x86_64-linux-gnu/rbconfig.rb"
rbconfig-x86_64-linux-3.0.0: "/usr/local/rake-compiler/ruby/x86_64-redhat-linux/ruby-3.0.0/lib/ruby/3.0.0/x86_64-linux-gnu/rbconfig.rb"
rbconfig-x86_64-linux-gnu-3.1.0: "/usr/local/rake-compiler/ruby/x86_64-redhat-linux/ruby-3.1.0/lib/ruby/3.1.0/x86_64-linux-gnu/rbconfig.rb"
rbconfig-x86_64-linux-3.1.0: "/usr/local/rake-compiler/ruby/x86_64-redhat-linux/ruby-3.1.0/lib/ruby/3.1.0/x86_64-linux-gnu/rbconfig.rb"
Instead of running ruby ext/magic/extconf.rb
inside the container, instead run bundle exec rake gem:x86_64-linux:builder
and you'll reproduce the issue.
@kwilczynski I spent a bit of time this morning trying to build a ruby on my development machine that would reproduce what we're seeing in the RCD image with the xruby 3.1.0, and I can't.
@larskanis Do you have any pointers on where to investigate what's going on here? If you want to get quick reproduction, check out this branch and run bundle exec rake gem:x86_64-linux
. This doesn't reproduce with any other Ruby 3.1.0 that I've built, even one built with the same configure options, and I'm struggling to understand what other variables might exist.
I can reproduce the issue locally. I'll check what's going on.
There's another pkg-config related issue in the x64-mingw-ucrt
image of rake-compiler-dock here, but it might be unrelated here.
xruby-3.0 links to libruby-static
, while for some reason xruby-3.1 doesn't. This probably is a bug in rake-compiler-dock. We do some magic there with the libruby
files.
A workaround is:
have_library('ruby-static')
have_func('rb_thread_call_without_gvl', 'ruby/thread.h')
Using 'ruby/thread.h' in have_func
isn't necessary, but is faster, since it detects the function in the first compiler run, while without it, two runs are necessary.
Hi @larskanis,
Thank you for looking into this!
[...]
A workaround is:
have_library('ruby-static') have_func('rb_thread_call_without_gvl', 'ruby/thread.h')
This helps!
A slightly different question as I am still, however, puzzled why do we need to add the following explicitly:
have_library('pthread')
have_library('rt')
have_library('dl')
have_library('crypt')
I don't know much about mkmf
internals, however, I would imagine that these are default libraries to link Ruby against? Would you happen to know? I wonder if we are doing something wrong here.
Using 'ruby/thread.h' in
have_func
isn't necessary, but is faster, since it detects the function in the first compiler run, while without it, two runs are necessary.
Interestingly, the extconf.rb
appears to be run multiple times. I am not sure why that is. Not the case with Ruby 3.0.x, however.
Krzysztof
Interestingly, the
extconf.rb
appears to be run multiple times. I am not sure why that is. Not the case with Ruby 3.0.x, however.
This is because two implementations of mkmf.rb
are active. Although ruby-core's mkmf.rb
isn't required in your extconf.rb
it is already active because of rake-compiler's .rake-compiler-siteconf.rb
. See here.
A slightly different question as I am still, however, puzzled why do we need to add the following explicitly:
have_library('pthread') have_library('rt') have_library('dl') have_library('crypt')
That's not intended. Looks like an issue of rake-compiler-dock. I'll investigate it.
OK, I've pushed a slimmer changeset, which slims down the patched mkmf.rb
to be just the pkg_config()
method, and it has two outcomes:
:shrug:
@kwilczynski Are you waiting on me for anything else related to this? I think it might be good to go.
Hi @flavorjones,
Sorry for the delay!
@kwilczynski Are you waiting on me for anything else related to this? I think it might be good to go.
No. Everything looks good! Thank you so much! Also, thank you for reporting this issue upstream so that everyone else can benefit from the fix/workaround, etc.
To add - everything builds fine, indeed, even though (at least in my case) the extconf.rb
seems to be running twice.
By the way, you are a project member with commit bit, thus similarly to @stanhu feel free to close Pull Requests, push commits, etc. I fully trust the two of you, of course. 😄
Krzysztof
@kwilczynski Looking into why Ruby 2.6 and 2.7 are failing here: https://gitlab.com/stanhu/ruby-magic/-/jobs/2123000713
I guess a retry fixed it.
Pull in the Ruby master mkmf.rb temporarily.
Note that ruby-magic.c doesn't compile in the rake-compiler-dock environment, failing with:
which comes from the GVL/blocking region section in common.h. I don't know how to resolve that, and I'm asking for help @stanhu @kwilczynski