search

inscapist / ruby-nix

Generates reproducible ruby/bundler app environment with Nix
MIT License
114 stars 5 forks source link

Unable to use platform gems #45

Closed seroperson closed 1 month ago

seroperson commented 1 month ago

Hello! Thank you for your work on ruby-nix!

I build jekyll website with it and it works smoothly, but until the only platform is ruby. If I add some additional platforms, like x86_64-linux, it doesn't work at all. I have encountered several "variants" of this issue while trying to make it work. Note: I work in WSL and I haven't tested it in other environments, but I don't think WSL is the problem (but still maybe it is).

So, the first one:

Consider my Gemfile looks like:

source "https://rubygems.org"
ruby "3.2.4"
gem "rspec", "~> 3.0"
gem "nokogiri"

My flake.nix:

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs";
    ruby-nix = {
      url = "github:inscapist/ruby-nix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs = {
    self,
    nixpkgs,
    ruby-nix
  }: let
    system = "x86_64-linux";

    pkgs = import nixpkgs {
      inherit system;
    };
    rubyNix = ruby-nix.lib pkgs;

    rubyDep = pkgs.ruby_3_2;

    deps = [ env ruby pkgs.bundix ];

    inherit (rubyNix {
      ruby = rubyDep;
      name = "test-gem";
      gemset = ./gemset.nix;
      gemConfig = pkgs.defaultGemConfig;
    })
      env ruby;
  in {
    packages.${system} = let
      bundlecli = pkgs.writeShellApplication {
        name = "bundle";
        runtimeInputs = deps;
        text = ''
          export BUNDLE_PATH=vendor/bundle
          bundle "$@"
        '';
      };
    in {
      bundle = bundlecli;
      bundix = pkgs.bundix;
      default = bundlecli;
    };

    devShells.${system}.default = pkgs.mkShell {
      shellHook = ''
        export BUNDLE_PATH=vendor/bundle
      '';
      buildInputs = deps;
    };
  };
}

Now I do (I'm manually adding ruby platform here):

rm -rf vendor Gemfile.lock gemset.nix && nix run .#bundle -- lock --update --add-platform ruby && nix run .#bundix -- --ruby=ruby_3_2 -m

Output:

Fetching gem metadata from https://rubygems.org/.......
Resolving dependencies...
Writing lockfile to /home/seroperson/Development/other/test-gem/Gemfile.lock
Writing lockfile to /home/seroperson/Development/other/test-gem/Gemfile.lock
Fetching gem metadata from https://rubygems.org/.......
Fetching diff-lcs 1.5.1
Fetching rspec-support 3.13.1
Fetching racc 1.8.1
Installing rspec-support 3.13.1
Fetching rspec-core 3.13.0
Installing diff-lcs 1.5.1
Installing racc 1.8.1 with native extensions
Fetching rspec-expectations 3.13.2
Fetching rspec-mocks 3.13.1
Installing rspec-core 3.13.0
Installing rspec-mocks 3.13.1
Installing rspec-expectations 3.13.2
Fetching rspec 3.13.0
Fetching nokogiri 1.16.7 (x86_64-linux)
Installing rspec 3.13.0
Installing nokogiri 1.16.7 (x86_64-linux)
Bundle complete! 2 Gemfile dependencies, 9 gems now installed.
Bundled gems are installed into `./vendor/bundle`
Updating files in vendor/cache
  * diff-lcs-1.5.1.gem
  * racc-1.8.1.gem
  * nokogiri-1.16.7-x86_64-linux.gem
  * rspec-support-3.13.1.gem
  * rspec-core-3.13.0.gem
  * rspec-expectations-3.13.2.gem
  * rspec-mocks-3.13.1.gem
  * rspec-3.13.0.gem
path is '/nix/store/gz7h6fkfsb7m9lcj74gmc3rmvg6x2y2y-rspec-support-3.13.1.gem'
03z7gpqz5xkw9rf53835pa8a9vgj4lic54rnix9vfwmp2m7pv1s8 => rspec-support-3.13.1.gem
path is '/nix/store/xbnhv40y73pp4jv3hibhspn5bk99z5sl-rspec-mocks-3.13.1.gem'
0f3vgp43hajw716vmgjv6f4ar6f97zf50snny6y3fy9kkj4qjw88 => rspec-mocks-3.13.1.gem
path is '/nix/store/hpcbsb4dlsmnzi0l78zh99ci543b6la0-rspec-expectations-3.13.2.gem'
0nm4qx9bgfzwfc1q0l3sj50vf88q1mbwkkqndbzc08wrnd5bjpsn => rspec-expectations-3.13.2.gem
path is '/nix/store/xa8k81gdz7abzlzfiyphrgsw6bb41gf5-rspec-core-3.13.0.gem'
0k252n7s80bvjvpskgfm285a3djjjqyjcarlh3aq7a4dx2s94xsm => rspec-core-3.13.0.gem
path is '/nix/store/rw8jq4a2kb6vdlrx3ny55118cl327xm2-rspec-3.13.0.gem'
14xrp8vq6i9zx37vh0yp4h9m0anx9paw200l1r5ad9fmq559346l => rspec-3.13.0.gem
path is '/nix/store/il5dwg6pwr3a3099ydsrcq4yxxxk3jgf-racc-1.8.1.gem'
0byn0c9nkahsl93y9ln5bysq4j31q8xkf2ws42swighxd4lnjzsa => racc-1.8.1.gem
path is '/nix/store/9k29xwwgjif9q5fh37byx03q5ybvf7pf-nokogiri-1.16.7-x86_64-linux.gem'
1xj0dawahc1jbw0w99dlzvln0d8nf66422y6fzw2m56m863447ly => nokogiri-1.16.7-x86_64-linux.gem
path is '/nix/store/61vjc90xd0i0iwfm9gzy0zgq99ylw48h-nokogiri-1.16.7.gem'
15gysw8rassqgdq3kwgl4mhqmrgh7nk2qvrcqp4ijyqazgywn6gq => nokogiri-1.16.7.gem
path is '/nix/store/ywprvv6iw24qy4ppsvy12cfjgzb01nns-mini_portile2-2.8.7.gem'
1q1f2sdw3y3y9mnym9dhjgsjr72sq975cfg5c4yx7gwv8nmzbvhk => mini_portile2-2.8.7.gem
path is '/nix/store/gsqfzb0js8nn5prqp65841yy5krvsbjg-diff-lcs-1.5.1.gem'
1znxccz83m4xgpd239nyqxlifdb7m8rlfayk6s259186nkgj6ci7 => diff-lcs-1.5.1.gem

Generated Gemfile.lock:

GEM
  remote: https://rubygems.org/
  specs:
    diff-lcs (1.5.1)
    mini_portile2 (2.8.7)
    nokogiri (1.16.7)
      mini_portile2 (~> 2.8.2)
      racc (~> 1.4)
    nokogiri (1.16.7-x86_64-linux)
      racc (~> 1.4)
    racc (1.8.1)
    rspec (3.13.0)
...

PLATFORMS
  ruby
  x86_64-linux

DEPENDENCIES
  nokogiri
  rspec (~> 3.0)

RUBY VERSION
   ruby 3.2.4p170

BUNDLED WITH
   2.4.19

And gemset.nix:

{
  diff-lcs = {
    groups = ["default"];
    platforms = [];
    source = {
      remotes = ["https://rubygems.org"];
      sha256 = "1znxccz83m4xgpd239nyqxlifdb7m8rlfayk6s259186nkgj6ci7";
      type = "gem";
    };
    version = "1.5.1";
  };
  mini_portile2 = {
    groups = ["default"];
    platforms = [];
    source = {
      remotes = ["https://rubygems.org"];
      sha256 = "1q1f2sdw3y3y9mnym9dhjgsjr72sq975cfg5c4yx7gwv8nmzbvhk";
      type = "gem";
    };
    version = "2.8.7";
  };
  nokogiri = {
    dependencies = ["mini_portile2" "racc"];
    groups = ["default"];
    platforms = [];
    source = {
      remotes = ["https://rubygems.org"];
      sha256 = "15gysw8rassqgdq3kwgl4mhqmrgh7nk2qvrcqp4ijyqazgywn6gq";
      type = "gem";
    };
    version = "1.16.7";
  };
  racc = {
    groups = ["default"];
    platforms = [];
    source = {
      remotes = ["https://rubygems.org"];
      sha256 = "0byn0c9nkahsl93y9ln5bysq4j31q8xkf2ws42swighxd4lnjzsa";
      type = "gem";
    };
    version = "1.8.1";
  };
  rspec = {
    dependencies = ["rspec-core" "rspec-expectations" "rspec-mocks"];
    groups = ["default"];
    platforms = [];
    source = {
      remotes = ["https://rubygems.org"];
      sha256 = "14xrp8vq6i9zx37vh0yp4h9m0anx9paw200l1r5ad9fmq559346l";
      type = "gem";
    };
    version = "3.13.0";
  };
  ...
}

Now I enter nix develop and do rspec (simple test with just require "nokogiri" and some dummy prints). Output:

/nix/store/kdizyn2pyl1s4qp8qi1dsp2yi8lpb128-ruby-3.2.4/lib/ruby/3.2.0/bundler/definition.rb:540:in `materialize': Could not find nokogiri-1.16.7-x86_64-linux in locally installed gems (Bundler::GemNotFound)
        from /nix/store/kdizyn2pyl1s4qp8qi1dsp2yi8lpb128-ruby-3.2.4/lib/ruby/3.2.0/bundler/definition.rb:200:in `specs'
        from /nix/store/kdizyn2pyl1s4qp8qi1dsp2yi8lpb128-ruby-3.2.4/lib/ruby/3.2.0/bundler/definition.rb:266:in `specs_for'
        from /nix/store/kdizyn2pyl1s4qp8qi1dsp2yi8lpb128-ruby-3.2.4/lib/ruby/3.2.0/bundler/runtime.rb:18:in `setup'
        from /nix/store/kdizyn2pyl1s4qp8qi1dsp2yi8lpb128-ruby-3.2.4/lib/ruby/3.2.0/bundler.rb:162:in `setup'
        from /nix/store/3xngsyhkvrvqwwpqnjh1ig1s78dp8jl5-test-gem/bin/rspec:32:in `bundler_setup!'
        from /nix/store/3xngsyhkvrvqwwpqnjh1ig1s78dp8jl5-test-gem/bin/rspec:36:in `<main>'
seroperson commented 1 month ago

The second one:

Now my Gemfile and flake.nix remain the same, I removed all locks and all data and did everything again, but without ruby platform:

rm -rf vendor Gemfile.lock gemset.nix && nix run .#bundle -- lock --update && nix run .#bundix -- --ruby=ruby_3_2 -m

Output: 

Resolving dependencies...
Writing lockfile to /home/seroperson/Development/other/test-gem/Gemfile.lock
Writing lockfile to /home/seroperson/Development/other/test-gem/Gemfile.lock
Fetching gem metadata from https://rubygems.org/.......
Fetching rspec-support 3.13.1
Fetching diff-lcs 1.5.1
Fetching racc 1.8.1
Installing rspec-support 3.13.1
Installing diff-lcs 1.5.1
Installing racc 1.8.1 with native extensions
Fetching rspec-core 3.13.0
Fetching rspec-expectations 3.13.2
Fetching rspec-mocks 3.13.1
Installing rspec-mocks 3.13.1
Installing rspec-expectations 3.13.2
Installing rspec-core 3.13.0
Fetching rspec 3.13.0
Fetching nokogiri 1.16.7 (x86_64-linux)
Installing rspec 3.13.0
Installing nokogiri 1.16.7 (x86_64-linux)
Bundle complete! 2 Gemfile dependencies, 9 gems now installed.
Bundled gems are installed into `./vendor/bundle`
Updating files in vendor/cache
  * diff-lcs-1.5.1.gem
  * racc-1.8.1.gem
  * nokogiri-1.16.7-x86_64-linux.gem
  * rspec-support-3.13.1.gem
  * rspec-core-3.13.0.gem
  * rspec-expectations-3.13.2.gem
  * rspec-mocks-3.13.1.gem
  * rspec-3.13.0.gem
path is '/nix/store/gz7h6fkfsb7m9lcj74gmc3rmvg6x2y2y-rspec-support-3.13.1.gem'
03z7gpqz5xkw9rf53835pa8a9vgj4lic54rnix9vfwmp2m7pv1s8 => rspec-support-3.13.1.gem
path is '/nix/store/xbnhv40y73pp4jv3hibhspn5bk99z5sl-rspec-mocks-3.13.1.gem'
0f3vgp43hajw716vmgjv6f4ar6f97zf50snny6y3fy9kkj4qjw88 => rspec-mocks-3.13.1.gem
path is '/nix/store/hpcbsb4dlsmnzi0l78zh99ci543b6la0-rspec-expectations-3.13.2.gem'
0nm4qx9bgfzwfc1q0l3sj50vf88q1mbwkkqndbzc08wrnd5bjpsn => rspec-expectations-3.13.2.gem
path is '/nix/store/xa8k81gdz7abzlzfiyphrgsw6bb41gf5-rspec-core-3.13.0.gem'
0k252n7s80bvjvpskgfm285a3djjjqyjcarlh3aq7a4dx2s94xsm => rspec-core-3.13.0.gem
path is '/nix/store/rw8jq4a2kb6vdlrx3ny55118cl327xm2-rspec-3.13.0.gem'
14xrp8vq6i9zx37vh0yp4h9m0anx9paw200l1r5ad9fmq559346l => rspec-3.13.0.gem
path is '/nix/store/il5dwg6pwr3a3099ydsrcq4yxxxk3jgf-racc-1.8.1.gem'
0byn0c9nkahsl93y9ln5bysq4j31q8xkf2ws42swighxd4lnjzsa => racc-1.8.1.gem
path is '/nix/store/9k29xwwgjif9q5fh37byx03q5ybvf7pf-nokogiri-1.16.7-x86_64-linux.gem'
1xj0dawahc1jbw0w99dlzvln0d8nf66422y6fzw2m56m863447ly => nokogiri-1.16.7-x86_64-linux.gem
path is '/nix/store/gsqfzb0js8nn5prqp65841yy5krvsbjg-diff-lcs-1.5.1.gem'
1znxccz83m4xgpd239nyqxlifdb7m8rlfayk6s259186nkgj6ci7 => diff-lcs-1.5.1.gem

Generated Gemfile.lock:

GEM
  remote: https://rubygems.org/
  specs:
    diff-lcs (1.5.1)
    nokogiri (1.16.7-x86_64-linux)
      racc (~> 1.4)
    racc (1.8.1)
...

PLATFORMS
  x86_64-linux

DEPENDENCIES
  nokogiri
  rspec (~> 3.0)

RUBY VERSION
   ruby 3.2.4p170

BUNDLED WITH
   2.4.19

gemset.nix:

{
  diff-lcs = {
    groups = ["default"];
    platforms = [];
    source = {
      remotes = ["https://rubygems.org"];
      sha256 = "1znxccz83m4xgpd239nyqxlifdb7m8rlfayk6s259186nkgj6ci7";
      type = "gem";
    };
    version = "1.5.1";
  };
  nokogiri = {
    dependencies = ["racc"];
    groups = ["default"];
    platforms = [];
    source = {
      remotes = ["https://rubygems.org"];
      sha256 = "1xj0dawahc1jbw0w99dlzvln0d8nf66422y6fzw2m56m863447ly";
      type = "gem";
    };
    version = "1.16.7";
  };
  racc = {
    groups = ["default"];
    platforms = [];
    source = {
      remotes = ["https://rubygems.org"];
      sha256 = "0byn0c9nkahsl93y9ln5bysq4j31q8xkf2ws42swighxd4lnjzsa";
      type = "gem";
    };
    version = "1.8.1";
  };
...
}

Now it fails at nix develop:

error: hash mismatch in fixed-output derivation '/nix/store/6m4c69zqndpgpx5ycv8qbqplqx6xjwb2-nokogiri-1.16.7.gem.drv':
         specified: sha256-nh5ChkHVlCr4d8YLQYxxFjVg6f60pcQBXzIwqLhqQPY=
            got:    sha256-+BnL/fsKexnJxSxvLKY98OWKYSX08Tlwe1hrlRHX/pU=
error: 1 dependencies of derivation '/nix/store/cya0jd838w1qsq8qd8aw5x62hqhg3adw-ruby3.2-nokogiri-1.16.7.drv' failed to build
error: 1 dependencies of derivation '/nix/store/k2wkvidaq0kbz0y1rlp4q16glnwnpw5w-test-gem-ruby-env.drv' failed to build
error: 1 dependencies of derivation '/nix/store/vz2f98pnwazsbhlv2wcr5dd5y1ghqzjb-test-gem-ruby.drv' failed to build
error: 1 dependencies of derivation '/nix/store/35adhqk4q55infdvbrkaq9iz0x376k82-nix-shell-env.drv' failed to build

What I found is that one of these hashes belongs to platformed gem and the other one belongs to non-platformed.

seroperson commented 1 month ago

The only working workaround for me:

I manually removed x86_64-linux platform and its' dependencies from Gemfile.lock:

GEM
  remote: https://rubygems.org/
  specs:
    diff-lcs (1.5.1)
    mini_portile2 (2.8.7)
    nokogiri (1.16.7)
      mini_portile2 (~> 2.8.2)
      racc (~> 1.4)
    racc (1.8.1)
...

PLATFORMS
  ruby

DEPENDENCIES
  nokogiri
  rspec (~> 3.0)

RUBY VERSION
   ruby 3.2.4p170

BUNDLED WITH
   2.4.19

Run rm -rf vendor gemset.nix && nix run .#bundle -- lock --update && nix run .#bundix -- --ruby=ruby_3_2 -m:

Fetching gem metadata from https://rubygems.org/.......
Resolving dependencies...
Writing lockfile to /home/seroperson/Development/other/test-gem/Gemfile.lock
Writing lockfile to /home/seroperson/Development/other/test-gem/Gemfile.lock
Fetching gem metadata from https://rubygems.org/.......
Fetching rspec-support 3.13.1
Fetching racc 1.8.1
Fetching diff-lcs 1.5.1
Installing rspec-support 3.13.1
Fetching rspec-core 3.13.0
Installing diff-lcs 1.5.1
Installing racc 1.8.1 with native extensions
Fetching rspec-expectations 3.13.2
Fetching rspec-mocks 3.13.1
Installing rspec-core 3.13.0
Installing rspec-expectations 3.13.2
Installing rspec-mocks 3.13.1
Fetching nokogiri 1.16.7 (x86_64-linux)
Fetching rspec 3.13.0
Installing rspec 3.13.0
Installing nokogiri 1.16.7 (x86_64-linux)
Bundle complete! 2 Gemfile dependencies, 9 gems now installed.
Bundled gems are installed into `./vendor/bundle`
Updating files in vendor/cache
  * diff-lcs-1.5.1.gem
  * racc-1.8.1.gem
  * nokogiri-1.16.7-x86_64-linux.gem
  * rspec-support-3.13.1.gem
  * rspec-core-3.13.0.gem
  * rspec-expectations-3.13.2.gem
  * rspec-mocks-3.13.1.gem
  * rspec-3.13.0.gem
path is '/nix/store/gz7h6fkfsb7m9lcj74gmc3rmvg6x2y2y-rspec-support-3.13.1.gem'
03z7gpqz5xkw9rf53835pa8a9vgj4lic54rnix9vfwmp2m7pv1s8 => rspec-support-3.13.1.gem
path is '/nix/store/xbnhv40y73pp4jv3hibhspn5bk99z5sl-rspec-mocks-3.13.1.gem'
0f3vgp43hajw716vmgjv6f4ar6f97zf50snny6y3fy9kkj4qjw88 => rspec-mocks-3.13.1.gem
path is '/nix/store/hpcbsb4dlsmnzi0l78zh99ci543b6la0-rspec-expectations-3.13.2.gem'
0nm4qx9bgfzwfc1q0l3sj50vf88q1mbwkkqndbzc08wrnd5bjpsn => rspec-expectations-3.13.2.gem
path is '/nix/store/xa8k81gdz7abzlzfiyphrgsw6bb41gf5-rspec-core-3.13.0.gem'
0k252n7s80bvjvpskgfm285a3djjjqyjcarlh3aq7a4dx2s94xsm => rspec-core-3.13.0.gem
path is '/nix/store/rw8jq4a2kb6vdlrx3ny55118cl327xm2-rspec-3.13.0.gem'
14xrp8vq6i9zx37vh0yp4h9m0anx9paw200l1r5ad9fmq559346l => rspec-3.13.0.gem
path is '/nix/store/il5dwg6pwr3a3099ydsrcq4yxxxk3jgf-racc-1.8.1.gem'
0byn0c9nkahsl93y9ln5bysq4j31q8xkf2ws42swighxd4lnjzsa => racc-1.8.1.gem
path is '/nix/store/61vjc90xd0i0iwfm9gzy0zgq99ylw48h-nokogiri-1.16.7.gem'
15gysw8rassqgdq3kwgl4mhqmrgh7nk2qvrcqp4ijyqazgywn6gq => nokogiri-1.16.7.gem
path is '/nix/store/ywprvv6iw24qy4ppsvy12cfjgzb01nns-mini_portile2-2.8.7.gem'
1q1f2sdw3y3y9mnym9dhjgsjr72sq975cfg5c4yx7gwv8nmzbvhk => mini_portile2-2.8.7.gem
path is '/nix/store/gsqfzb0js8nn5prqp65841yy5krvsbjg-diff-lcs-1.5.1.gem'
1znxccz83m4xgpd239nyqxlifdb7m8rlfayk6s259186nkgj6ci7 => diff-lcs-1.5.1.gem

Now nix develop and rspec, everything works fine:

Finished in 0.00148 seconds (files took 0.08135 seconds to load)
1 example, 0 failures
seroperson commented 1 month ago

Also I'm unable to get minify_html (version 0.15.0) gem working. It always attempts to download ruby-platformed gem and fails with 403. To reproduce, just add gem "minify_html" and make sure you have x86_64-linux in platforms. I think it is related to what I wrote above.

inscapist commented 1 month ago

Hi @seroperson , you are using pkgs.bundix from nix-community. It does not support native compiled gems like what you are doing here. You should use this fork instead.

To verify, you could rm gemset.nix and run

nix run github:inscapist/bundix

You can look at the diff of the regenerated gemset.nix to see the difference as well.

inscapist commented 1 month ago

And since you use flake, you could simply include bundix in the inputs

...
  inputs = {
    nixpkgs.url = "nixpkgs";
    ruby-nix.url = "github:inscapist/ruby-nix";
    # a fork that supports platform dependant gem
    bundix = {
      url = "github:inscapist/bundix/main";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    fu.url = "github:numtide/flake-utils";
    bob-ruby.url = "github:bobvanderlinden/nixpkgs-ruby";
    bob-ruby.inputs.nixpkgs.follows = "nixpkgs";
  };
...

Or you can refer examples/simple-app/flake.nix

seroperson commented 1 month ago

Oh, I accidentally missed this thing. Yea, with this bundix it works flawlessly. Sorry for beign dumb and thank you for your work again!