Add support for platform-dependant pre-compiled gems

lavoiesl commented 4 years ago

For example:

# Gemfile
source 'https://rubygems.org' do
  gem 'sorbet', '= 0.4.4821'
end

# Gemfile.lock
GEM
  remote: https://rubygems.org/
  specs:
    sorbet (0.4.4821)
      sorbet-static (= 0.4.4821)
    sorbet-static (0.4.4821-universal-darwin-14)
# ...

# gemset.nix
{
  sorbet = {
    dependencies = ["sorbet-static"];
    groups = ["default"];
    platforms = [];
    source = {
      remotes = ["https://rubygems.org"];
      sha256 = "0nci20x5vqd2q9pvykmam05xv95j7zsxxcj5favjb2knfk6z5jq4";
      type = "gem";
    };
    version = "0.4.4821";
  };
  sorbet-static = {
    groups = ["default"];
    platforms = [];
    source = {
      remotes = ["https://rubygems.org"];
      sha256 = "0lxw6il82h5m7syy0sswgd7k0hjr5kxvdm59s2kkbjsh8wdn1rnp";
      type = "gem";
    };
    version = "0.4.4821";
  };
}

$ bundix -l
[4.3 MiB DL]
path is '/nix/store/hlxbn8cnvpiaf8szz59cn93y9d6lz8a6-sorbet-static-0.4.4821-universal-darwin-14.gem'
0lxw6il82h5m7syy0sswgd7k0hjr5kxvdm59s2kkbjsh8wdn1rnp => sorbet-static-0.4.4821-universal-darwin-14.gem
# ...

Problem 1 - download url

Trying to start a shell with that would result in the wrong gem url trying to be downloaded:

trying https://rubygems.org/gems/sorbet-static-0.4.4821.gem
curl: (22) The requested URL returned error: 403 Forbidden
error: cannot download sorbet-static-0.4.4821.gem from any mirror

The URL should include the platform; namely https://rubygems.org/gems/sorbet-static-0.4.4821-universal-darwin-14.gem

This is what https://github.com/nix-community/bundix/pull/67 was attempting to fix.

Problem 2 - sha256

The version recorded in the gemset.nix ends up being for whatever platform recorded was in the Gemfile.lock. In practice, we need to download a gem that is compatible with our current platform.

Knowing the exact platform of a compiled gem is important because we need the correct URL and different binaries have different hashes.

Changes in this PR

Add sorbet to bundler-audit because it depends on a platform-dependant gem
Change the mock fetcher in tests to extend a real fetcher
Refactor Bundix::Source to return a hash one-level above
When we detect that a gem has a platform, check available versions (local or remote) for a matching platform.
Add the platform in the version recorded in the gemset

PR is broken down in smaller commits for easier review

Superseeds https://github.com/nix-community/bundix/pull/67; this PR doesn't assume that the gem exists with a version for the exact local platform. Instead, it finds the first matching platform in available gems.

Testing the PR

(With Gemfile above)

$ nix run nixpkgs.ruby -c /path/to/bundix -l

# default.nix
with (import <nixpkgs> {});
let
  gems = bundlerEnv {
    name = "test";
    inherit ruby;
    gemdir = ./.;
  };
in stdenv.mkDerivation {
  name = "test";
  buildInputs = [gems ruby];
}

$ nix-shell
[nix-shell:~]$ bundler list --paths
/nix/store/njz4hkf3cx2rzapdwds0rbzvzpcdwv97-test/lib/ruby/gems/2.6.0/gems/sorbet-0.4.4821
/nix/store/njz4hkf3cx2rzapdwds0rbzvzpcdwv97-test/lib/ruby/gems/2.6.0/gems/sorbet-static-0.4.4821-universal-darwin-17

Known caveats

This PR makes the gemset platform dependant.
The platform is detected based on the ruby that executes bundix

/cc @burke @morriar

lavoiesl commented 4 years ago

Is there anything else I can do to help support this PR?

This is blocking our use of bundix

alyssais commented 4 years ago

I don't think we can have platform-dependent gemsets. What do you envision a workflow looking like in that case? You have to run Bundix seperately on each platform you might want to build on?

lavoiesl commented 4 years ago

Correct. It's worth nothing that the gem's platform is tied to the ruby platform, not the host platform.

So, if two people are on darwin and are working on a project that mandates using a particular ruby version, that ruby would come from Nix, compiled for a particular platform, so it will be consistent across devices.

e.g. depending on ruby 2.6.5, which is compiled on x86_64-darwin-17 in Nixpkgs, would result in needing darwin-17 gems, regardless of the exact platform version (macOS version) of the user.

Because of that, as long as devs are on the same platform family and are using the same ruby version, they can share a gemset.nix

It is indeed unfortunate that this PR makes the gemset platform dependant if using platform dependant gems, but I feel it's a step in the right direction since currently, platform dependant gems are flat out not supported.

alyssais commented 4 years ago

It is indeed unfortunate that this PR makes the gemset platform dependant if using platform dependant gems, but I feel it's a step in the right direction since currently, platform dependant gems are flat out not supported.

Oh, I see. What does this look like in Bundler if you have some users on GNU/Linux and some on Darwin (or Musl/Linux, etc.)? I assume such a thing is supported?

lavoiesl commented 4 years ago

Yeah, it wouldn't be supported for now. Users could either generate the gemset themselves or maintain a per-platform gemset.

In the future, we could change it such that a single gemset could hold multiple platforms, but this has the pitfall of having to detect all platforms and identify which ones are relevant to the user. There is also the added complexity of figuring out the correct platform to install within Nix.

All that feels like a can of worms that I’d rather open on a later day.

jdelStrother commented 4 years ago

@lavoiesl do you know why given a gemfile like this:

source 'https://rubygems.org' do
  gem 'libv8'
end

bundix -l fetches from https://rubygems.org/gems/libv8-8.4.255.0.gem and not https://rubygems.org/gems/libv8-8.4.255.0-universal-darwin-17.gem ?

I notice Gemfile.lock also doesn't mention darwin, which is guess is the real source of the problem:

GEM
  remote: https://rubygems.org/
  specs:
    libv8 (8.4.255.0)

PLATFORMS
  ruby

DEPENDENCIES
  libv8!

BUNDLED WITH
   2.1.4

How is libv8 packaged differently compared to sorbet-static?

jdelStrother commented 3 years ago

Anyone still interested in this? Building on this PR, I've hacked together a proof-of-concept which allows for multi-platform gemsets here: https://github.com/nix-community/bundix/commit/6a921d00ec686fb63e20e7248c0cdf8a02f107cd

It generates gemsets like:

 {
   libv8-node = {
     groups = ["default"];
     platforms = [];
     source = {
+      platform = "ruby";
       remotes = ["https://rubygems.org"];
       sha256 = "0k2cqxnbm7lm284fa65bf4b18w7k6977mh6anyyai34r43g3vm34";
       type = "gem";
     };
+    nativeSources = [{
+      platform = "arm64-darwin-20";
+      remotes = ["https://rubygems.org"];
+      sha256 = "04qnpw0bm8j82km2whx5zhw59lm8isd3i540g1xfyc0mqw0vynhp";
+      type = "gem";
+    } {
+      platform = "x86_64-darwin-20";
+      remotes = ["https://rubygems.org"];
+      sha256 = "0r65dwxjrk6s9q4lf1jf7kg344qg3fn8d9r6nad91swmabpzd2bg";
+      type = "gem";
+    } {
+      platform = "x86_64-linux";
+      remotes = ["https://rubygems.org"];
+      sha256 = "04avzccvrjk4nbi7926cvbdkpkgc6a2m0gyz1s1k4x7q06lkdnnk";
+      type = "gem";
+    }];
     version = "15.14.0.0";
   };
 }

so source still references the regular gem, and an additional nativeSources list is added with any available natively-compiled gems.

Then I've tweaked nixpkgs' composeGemAttrs to look up the relevant source for the hostPlatform here: https://github.com/jdelStrother/nixpkgs/commit/e7316af13f357261cdf3788683c6ab18d6ee85cd

Still unresolved:

The gemsets that this generates are not currently compatible with the released version of nixpkgs. I was hoping that the nativeSources attribute would just be ignored there, but somehow it results in a hard-to-find "cannot coerce a set to a string" error.
My way of looking up native ruby platform from the nix hostPlatform is probably not good enough, not sure of a better way of doing it. I'm not totally clear on how nix handles the difference between, eg, x86_64-darwin-19 and x86_64-darwin-20.
nixpkg's gem-config seems mostly irrelevant for gems that offer precompiled versions. Currently I remove those using something like this in bundlerEnv, not sure if a smarter approach is desirable:
```
pkgs.bundlerEnv {
gemdir = ../.;
gemConfig = (builtins.removeAttrs pkgs.defaultGemConfig ["nokogiri" "libv8" "mini_racer"]);
...
}
```
Gem dependencies can differ between platforms, which we currently kind of ignore. In practice, I'm not found any real problems with this - eg nokogiri depends on mini_portile2 if installing the non-native versions, but it's not a big deal to include mini_portile2 even if you're installing the native versions. If it does cause problems, not sure of the best way of fixing. Possibly it would be better to completely restructure the gemset so that the top-level is keyed by platform, something along the lines of:

{
  ruby = {
    nokogiri = {
       dependencies = ["mini_portile2"];
       source = { ... source for the ruby version of the gem ... };
    };
    mini_portile2 = { ... };
  };

  x86_64-linux = {
    nokogiri = {
       dependencies = [];
       source = { ... source for the x86-linux version of the gem ... };
    };
  };

  x86_64-darwin19 = {
    nokogiri = {
       dependencies = [];
       source = { ... source for the x86-darwin version of the gem ... };
    };
  };
}

It would lead to a gemsets being much bigger though. If we took that approach, it seems like you might as well just have multiple gemsets.nix - forget about my nativeSources attribute and have bundix generate gemset.x86_64-linux.nix, gemset.x86_64-darwin.nix, etc.

omnibs commented 3 years ago

I was naively thinking about the gemset per arch approach too.

~We are currently avoiding solving this problem at work by using bundle install for dev (faster, and we can trigger from our shake build chain), using bundix so nix can generate container images for CI/prod without needing kvm to runAsRoot bundle install in VMs in macOS, and manually patching sorbet in the gemset whenever we have to re-generate it. Not great.~

Edit: We switched to using this PR and generating a gitignored gemset.nix.

maksar commented 3 years ago

So, it looks like currently there are two approaches – 1) generate a platform-dependant gemset.nix (this PR) 2) generate universal gemset.nix (from @jdelStrother)

But there is another possibility – generate different gemsets for different platforms. That way @omnibs will be able exclude from gitignore. bundlerEnv will choose appropriate gemset.nix file depending on a platform. If I'm not mistaken, it is possible to override the file file gemset attribute for now. In future, nixpkgs can accept a PR for doing this automatically (with fallback to a single gemset.nix for compatibility).

Any thoughts?

jdelStrother commented 3 years ago

IMO, we should generate a separate gemset file for each platform. The mess of files it'll generate is a little inelegant, but it's conceptually simpler than my universal gemset approach, and doesn't require changes to nixpkgs.

shepting commented 2 years ago

Any chance you've got time to have a look at this @manveru or @alyssais ? This would really help us trying to use a newer version of bundler in our Ruby code.

quinn-dougherty commented 2 years ago

Still interested in progress here.

peterhoeg commented 2 years ago

There really is nothing preventing you from just using different gemset.nix files today:

bundix --magic --gemset=foo.nix

So generate that per platform (admittedly not particularly elegant) and pass different gemsets to the bundlerEnv builder.

nixos-discourse commented 2 years ago

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/issues-with-nix-reproducibility-on-macos-trying-to-build-nokogiri-ruby-error-unknown-warning-option/22019/6

inscapist commented 1 year ago

A bit late to the scene (week 2~3 learning Nix). I had just published a flake that is built on top of @jdelStrother and @lavoiesl changes here. It supports native sources, among other things.

https://github.com/sagittaros/ruby-nix

manveru commented 1 year ago

@sagittaros would you be interested in becoming maintainer of bundix as well? I'm hardly doing anything with Ruby these days and feel we need someone with actual stake here.

inscapist commented 1 year ago

Thanks for the offer @manveru , I am unable to take this on now due to my lack of bandwidth. I am happy to help close a couple issues though.

manveru commented 1 year ago

No worries, any help is better than the current state of affairs :)

nix-community / bundix