Empty set for "gemset" gem

[nyarly]

Trying to pick up development of https://github.com/ruby-rdf/shex, I found that bundix produced a set in the gemset.nix like shex = {}; which led to tears when I tried to nix-shell. Should it be prevented in that case? There was a warning from bundix like "couldn't find a source for 'shex', skipping."

[nyarly]

I think this comes down to: the gem I'm working on is being treated as a "path" source, which bundix doesn't support (yet?). I'm not sure how to approach this. What I'd like is to be able to use bundix to set up a development shell for the gem until it's ready to be deployed. In the meantime, even editing the empty source set out of gemset.nix, nix-shell doesn't like the shell.nix I'm using, because /nix/store/...-gemfile-and-lockfile/Gemfile blows up in Bundler: there isn't a gemspec in that directory. It appears to the the local Gemfile and Gemfile.lock from the project copied into /nix/store.

It looks like the solution here would be to construct that ... derivation(?) to include the shex.gemspec, but I'm still a relative novice with Nix, so I'm not sure what I need to do to accomplish that.

[zimbatm]

Yes, unfortunately both bundix and the nix counter-part bundlerEnv don't support the "path" source. The reason is that the Gemfile and Gemfile.lock get copied into their own derivation and so the path doesn't make sense there.

As a hack you could load the gemspec using something like spec = eval File.read('shex.gemspec') and then inject the dependencies by listing the spec.runtime_dependencies.

[nyarly]

It looks like the dependencies get collected into the gemset.nix just fine. I assume it consumes the Gemfile.lock, which would explain that.

The problem then is that Bundler when is loading the Gemfile once it's copied into the gemfile-and-lockfile derivation, the gemspec is missing. Would it be possible for bundix(?) to copy *.gemspec into that derivation? Is that an easy enough change that I could PR it for you? (I roll deep with Ruby, and am keen to improve my Nix game.)

[manveru]

The problem with that is that gemspec often requires files in the project. Not sure if we can resolve this easily.

On Mon, Jan 23, 2017, 20:18 Judson Lester notifications@github.com wrote:

It looks like the dependencies get collected into the gemset.nix just fine. I assume it consumes the Gemfile.lock, which would explain that.

The problem then is that Bundler when is loading the Gemfile once it's copied into the gemfile-and-lockfile derivation, the gemspec is missing. Would it be possible for bundix(?) to copy *.gemspec into that derivation? Is that an easy enough change that I could PR it for you? (I roll deep with Ruby, and am keen to improve my Nix game.)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/manveru/bundix/issues/10#issuecomment-274588562, or mute the thread https://github.com/notifications/unsubscribe-auth/AAANs8aJnWjGH9UctmydpHqM0sSyha78ks5rVP0bgaJpZM4LqjW8 .

[nyarly]

Could bundleEnv look for gemfileExtraFiles, and add those to the gemfile-and-lock derivation? The writer of the expression seems to be in the right position to determine which files would need to be added...

[nyarly]

Is ^ a reasonable approach, @manveru, @zimbatm ?

[manveru]

Sounds like the best option we have at the moment m

On Wed, Jan 25, 2017, 22:22 Judson Lester notifications@github.com wrote:

Is ^ a reasonable approach, @manveru https://github.com/manveru, @zimbatm https://github.com/zimbatm ?

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/manveru/bundix/issues/10#issuecomment-275236623, or mute the thread https://github.com/notifications/unsubscribe-auth/AAANs0bQwIRxc6Q_8StQoWikIGgVfpthks5rV7zZgaJpZM4LqjW8 .

[zimbatm]

That would be a solution but I don't think it would be very satisfying. It's another flag that needs to be learned and tweaked as the project evolves.

I think bundix could be patched to emit a special "has path" flag in the gemset.nix when path dependencies are detected (and not add those dependencies to the gemset otherwise). There might not be an obvious extension point in the current gemset.nix format.

On the bundlerEnv side, if the flag is detected then the gemdir attribute would be required and be used as the source to copy all of the files into the derivation where the Gemfile and Gemfile.lock usually live.

[nyarly]

As we're discussing this, I'm finding it useful to have https://github.com/NixOS/nixpkgs/blob/6351c4d24b4cf0c75d51aeb8cd3c3720488a111d/pkgs/development/ruby-modules/bundler-env/default.nix for reference.

First, that's not... overreach? I mean, I guess disk is cheap and this is really a dev-only kind of feature - although, it begs the question: is there a control on either solution getting added to nixpkgs? While bundlerEnv gets used in actual expressions, a gemdir attribute would sure be impossible to satisfy?

Secondly, I think I see how that would handle "the gem I'm working on uses gemspec" but could it be expanded to handle explicit :path => in the Gemfile?

Third, would the gemset "has path" not be source.type = path?

[alyssais]

Since Bundix is able to generate its own dependency list (for the most part), could it just generate a new, normalised Gemfile to put into gemfile-and-lockfile?

[alyssais]

I hacked around for a while, and was eventually able to get a gem development environment going by writing a normalized Gemfile, (only including dependencies, not my gem itself) and then running Bundix on that:

#! /usr/bin/env nix-shell
#! nix-shell -i ruby -p ruby -p bundix

require "shellwords"

def sh(*args)
  $stderr.puts args.shelljoin
  system *args or fail
end

sh "bundle", "lock"

require "fileutils"
require "bundler"

lockfile = Bundler::LockfileParser.new(File.read("Gemfile.lock"))
gems = lockfile.specs.select { |spec| spec.source.is_a?(Bundler::Source::Rubygems) }
sources = [URI("https://rubygems.org/")] | gems.map(&:source).flat_map(&:remotes)

FileUtils.mkdir_p "nix/gem"
Dir.chdir "nix/gem" do
  File.open("Gemfile", "w") do |gemfile|
    sources.each { |source| gemfile.puts "source #{source.to_s.inspect}" }
    gemfile.puts

    gems.each do |gem|
      gemfile.puts "gem #{gem.name.inspect}, #{gem.version.to_s.inspect}"
    end
  end

  sh "bundix", "-l"
end

with import <nixpkgs> {};

(bundlerEnv {
  name = "dev";
  gemdir = ./nix/gem;
}).env

(I'm happy for anybody to use this code)

To the extent possible under law, I waive all copyright and related or neighboring rights to this work. See http://creativecommons.org/publicdomain/zero/1.0/.

[manveru]

I think we could at least make this an option, this seems like a really good step to ensure we won't run into trouble with "interesting" Gemfiles that have a lot of logic in them as well.

[alyssais]

That would be really cool — as it stands I think it’s impossible to get an environment for working on a gem (which means there’ll be a Gemfile with gemspec) without doing what I did above.

[alyssais]

Hmm… would bundle install --standalone help at all here?

[nyarly]

The bundlerEnv expressions have some support for copying gems into the store (i.e. if they're :path sources) (c.f. https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/ruby-modules/bundled-common/default.nix#L64). It should be straightforward to add a local source kind to https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/ruby-modules/bundled-common/default.nix#L64).

I think the next step would be to "poison" expressions that include path sources so that they can't be installed, or won't make it into nixpkgs, or something.