search

sparklemotion / nokogiri

Nokogiri (é‹ø) makes it easy and painless to work with XML and HTML from Ruby.
https://nokogiri.org/
MIT License
6.14k stars 899 forks source link

[install] Why are the pre-compiled versions not used in `Ruby 3.4.0-preview2`? #3315

Closed mediafinger closed 2 days ago

mediafinger commented 3 days ago

What happened

We updated our project to Ruby 3.4.0-preview2 and everything works fine. But when I run bundle install (for the first time), bundler is not trying to install the pre-compiled versions, but instead wants to compile it on the system.

$> bundle install

...
Installing nokogiri 1.16.7 with native extensions

I know how to install the gem by compiling it locally. On my machine I had to run:

To make bundle install pass.

The issue

The RubyLSP extension on VSCode does not start anymore, as it can not run bundle (as it fails on nokogiri). Unfortunately I was not able to set the flags for bundle config (maybe you can tell me which should work?), which I hoped would be picked up by the RubyLSP process.

Can I somehow force it to keep using the precompile versions?

Btw: in the change from Ruby 3.3.5 to 3.4.0-preview2 our Gemfile.lock changed from:

    nokogiri (1.16.7-aarch64-linux)
      racc (~> 1.4)
    nokogiri (1.16.7-arm-linux)
      racc (~> 1.4)
    nokogiri (1.16.7-arm64-darwin)
      racc (~> 1.4)
    nokogiri (1.16.7-x86-linux)
      racc (~> 1.4)
    nokogiri (1.16.7-x86_64-darwin)
      racc (~> 1.4)
    nokogiri (1.16.7-x86_64-linux)
      racc (~> 1.4)

to

    nokogiri (1.16.7)
      mini_portile2 (~> 2.8.2)
      racc (~> 1.4)

So we used the precompiled versions before. And we still have the platforms in the Gemfile.lock:

PLATFORMS
  aarch64-linux
  arm-linux
  arm64-darwin
  x86-linux
  x86_64-darwin
  x86_64-linux

Have you read and followed the installation tutorial at http://www.nokogiri.org/tutorials/installing_nokogiri.html?

Tell us about your system!

What is the output from ruby -v? ruby 3.4.0preview2 (2024-10-07 master 32c733f57b) +PRISM [arm64-darwin23]

What is the output from gem -v? 3.6.0.dev

What is the output from gem env?

RubyGems Environment:
  - RUBYGEMS VERSION: 3.6.0.dev
  - RUBY VERSION: 3.4.0 (2024-10-07 patchlevel -1) [arm64-darwin23]
  - INSTALLATION DIRECTORY: /Users/andy/.gem/ruby/3.4.0
  - USER INSTALLATION DIRECTORY: /Users/andy/.gem/ruby/3.4.0+0
  - RUBY EXECUTABLE: /Users/andy/.rubies/ruby-3.4.0-preview2/bin/ruby
  - GIT EXECUTABLE: /opt/homebrew/bin/git
  - EXECUTABLE DIRECTORY: /Users/andy/.gem/ruby/3.4.0/bin
  - SPEC CACHE DIRECTORY: /Users/andy/.gem/specs
  - SYSTEM CONFIGURATION DIRECTORY: /Users/andy/.rubies/ruby-3.4.0-preview2/etc
  - RUBYGEMS PLATFORMS:
     - ruby
     - arm64-darwin-23
  - GEM PATHS:
     - /Users/andy/.gem/ruby/3.4.0
     - /Users/andy/.rubies/ruby-3.4.0-preview2/lib/ruby/gems/3.4.0+0
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => true
     - :bulk_threshold => 1000
  - REMOTE SOURCES:
     - https://rubygems.org/
  - SHELL PATH:
     - /Users/andy/.gem/ruby/3.4.0/bin
     - /Users/andy/.rubies/ruby-3.4.0-preview2/lib/ruby/gems/3.4.0+0/bin
     - /Users/andy/.rubies/ruby-3.4.0-preview2/bin
     - /Users/andy/.asdf/shims
     - /opt/homebrew/opt/asdf/libexec/bin
     - /opt/homebrew/sbin
     - /opt/homebrew/bin
     - /opt/homebrew/opt/libpq/bin
     - /Applications/Postgres.app/Contents/Versions/latest/bin
     - /usr/local/bin
     - /System/Cryptexes/App/usr/bin
     - /usr/bin
     - /bin
     - /usr/sbin
     - /sbin
     - /var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin
     - /var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin
     - /var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin
     - /opt/homebrew/sbin
     - /opt/homebrew/bin
     - /opt/homebrew/opt/libpq/bin
     - /Applications/Postgres.app/Contents/Versions/latest/bin
     - /Users/andy/.rubies/ruby-3.1.2/bin/gem
     - /Users/andy/.asdf/shims/node
     - /usr/local/bin/code
     - /Users/andy/.rubies/ruby-3.1.2/bin/gem
     - /Users/andy/.asdf/shims/node
     - /usr/local/bin/code

If you're using Bundler:

If you're on MacOS, please note:

Apple clang version 16.0.0 (clang-1600.0.26.3)
Target: arm64-apple-darwin23.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
flavorjones commented 2 days ago

@mediafinger I hear your frustration and I'm sorry you're having a problem here. But putting ?! in the title adds emotion to this issue that is unnecessary, and I encourage you to skip that bit in the future.

Ruby 3.4 is not officially supported yet by Nokogiri, and we will not ship precompiled versions of Nokogiri for that version of Ruby until the ABI is declared "frozen", which is unlikely to happen until very late in the year.

~What you're describing sounds like a bundler issue to me, and I recommend that you delete your Gemfile.lock and re-run bundle install from scratch. If that doesn't work, please open a new issue at https://github.com/rubygems/rubygems and tag me on it so I can help diagnose what's going on.~ I've deleted this paragraph because I misunderstood what you were describing.

flavorjones commented 2 days ago

Please let me edit my above comment: after re-reading I understand that you are asking for a precompiled version, and I thought you were saying it wouldn't install at all.

To re-state what I said above: Ruby 3.4 is not finalized yet, so we cannot safely ship precompiled versions. If you want to test Ruby 3.4 previews out, you will need to compile C extensions at installation time.

I'm not sure how to get around the ruby-lsp problems you're describing, but if you've installed Nokogiri properly compiled I don't understand why it wouldn't just pick that version of the gem up. You may want to file an issue with that project to ask for help.

flavorjones commented 2 days ago

I also want to note that your description of how you compiled Nokogiri doesn't quite sound right to me: you shouldn't need to brew-install libiconv, as Nokogiri should find and link against the system libiconv (we test this in our CI suite).

Specifically, needing to export those environment variables might only work in your current shell -- and possibly not in ruby-lsp processes. I'd be interested in seeing whatever error/failure messages you get from installing Nokogiri under Ruby 3.4.0-preview2 without setting those env vars or using homebrew, to understand the problem better. And I'd be interested in seeing what error messages are being emitted by ruby-lsp when it either bundles or boots.

I'll leave this closed, but I'm happy to keep this thread going to try to point you in the right direction.

mediafinger commented 2 days ago

Thank you for your detailed answers @flavorjones

I was not aware that pre-compiled versions are only available after the .0 version of Ruby has been released, thanks for the info.

I have been surprised about the libiconv dependency as well. So to be able to re-create the error logs, I:

bundle

gem install nokogiri

After I set the two ENV mentioned above - and actually without installing libiconv - I compiled nokogiri successfully again.

RubyLSP extension in VSCode

Looking at this, I assume RubyLSP does have to compile nokogiri again, as it is not using the directory $HOME/.gem/ruby/3.4.0, but instead $HOME/.gem/ruby/3.4.0+0. šŸ¤”

I don't know anything about how RubyLSP usually handles this. Do you think this is worth a bug report, or a feature request, on the RubyLSP repo?

Additional question

checking for whether -I is accepted as CPPFLAGS... yes
checking for whether /opt/homebrew/include is accepted as CPPFLAGS... no
checking for whether -L is accepted as LDFLAGS... yes
checking for whether /opt/homebrew/lib is accepted as LDFLAGS... no

Are those no entries controlled by nokogiri, or is this under control of RubyGems or bundler?

PS: I wasn't that emotional when writing the issue and didn't think about my use of "?!" šŸ˜‰

flavorjones commented 2 days ago

@mediafinger Re: not finding libiconv, I see this in your mkmf.log:

DYLD_LIBRARY_PATH=.:/Users/andy/.rubies/ruby-3.4.0-preview2/lib

Can you try again in an environment where you're not clobbering that environment variable? I suspect this may be preventing you from resolving libiconv.

I also see this error repeatedly:

clang: error: no such file or directory: 'arm64'

which I think is caused by the -arch arm64 parameter. I'm not sure what the value here is supposed to be, but this might also be preventing mkmf from compiling the executables it uses to test for the presence of libiconv. Worth looking into.

flavorjones commented 2 days ago

Re: -arch arm64, can you check what the value of RbConfig::CONFIG["ARCH_FLAG"] and RbConfig::CONFIG["target_cpu"] are (you can print this from an irb prompt if you like)?

mediafinger commented 1 day ago

Regarding DYLD_LIBRARY_PATH - I can't imagine I am setting this anywhere explicitly. Neither in the shell I used yesterday to create the output (at least I think it is still the same session) nor in a new session does it show up when I use printenv.

irb output:

app(dev) > RbConfig:: CONFIG["ARCH_FLAG"]
"-arch arm64"

app(dev) > RbConfig:: CONFIG["target_cpu"]
"arm64"
flavorjones commented 1 day ago

Looking at this, I assume RubyLSP does have to compile nokogiri again, as it is not using the directory $HOME/.gem/ruby/3.4.0, but instead $HOME/.gem/ruby/3.4.0+0. šŸ¤”

I don't know anything about how RubyLSP usually handles this. Do you think this is worth a bug report, or a feature request, on the RubyLSP repo?

I think this is probably the best next step.

mediafinger commented 1 day ago

Thanks a lot for your support!

Just to close this issue here, for now I solved it by symlinking the gem folder used by RubyLSP to the existing gem folder in which the compiled nokogiri lies:

~/.gem/ruby > ln -s 3.4.0 3.4.0+0