BundlerEnv UTF-8 encoding

raymondhoagland commented 7 years ago

Issue description

When trying to install certain gems through bundlerEnv (namely xcode-install) the nix-shell crashes while building packages, specifically when trying to execute gen-bin-stubs.rb for the gem.

Trying to track down the issue, it looks like the problem is that one of the executables uses the 🎉 symbol which isn't ASCII-safe and the script ultimately breaks here.

Modifying the gen-bin-stubs script and printing out the LC_ALL and LANG ENV values leads me to believe bundlerEnv is wiping the environment out as they both end up nil. Checking the default external_encoding in the script thereby shows Ruby is using US-ASCII, which causes the encoding snafu.

Is there any way to properly inject these variables into the bundlerEnv to account for this? Manually changing the script to read the file as UTF-8 gets around the problem but I'm pretty sure that's not a good fix-all. Workaround looks like

executables = File.read("#{path}/nix-support/gem-meta/executables", :encoding => 'utf-8').split

Another alternative (again not a great solution) is to set the default external encoding at the beginning of gen-bin-stubs through

Encoding.default_external="UTF-8"

The actual environment settings (outside of nix) uses

LC_ALL=en_US.UTF-8
LANG=en_US.UTF-8

Relevant piece of the default.nix script

with import <nixpkgs>{};

let
    gems = bundlerEnv {
        name = "test";
        inherit ruby_2_3;
        gemdir = ./.;
        # gemfile = ./Gemfile;
        # lockFile = ./Gemfile.lock;
        # gemset = ./gemset.nix;
    };
in

stdenv.mkDerivation rec {
    name = "test";
    version = "0.1.0";
    preInstall = ''
    unset GEM_PATH
    '';
    buildInputs = [
        # Ruby
        ruby_2_3
        bundler bundix
        libxslt libxml2 pkgconfig libffi libiconv
        gems

        wget
    ];
}

Steps to reproduce

Create a gemfile with xcode-install, generate the Gemfile.lock, and define the gemset.nix using bundix. Running nix-shell should display the error. (e.g.)

/nix/store/wz11skksfy6ps3kprd4g2w8x4c50xas4-gen-bin-stubs.rb:21:in `split': invalid byte sequence in US-ASCII (ArgumentError)
        from /nix/store/wz11skksfy6ps3kprd4g2w8x4c50xas4-gen-bin-stubs.rb:21:in `block in <main>'
        from /nix/store/wz11skksfy6ps3kprd4g2w8x4c50xas4-gen-bin-stubs.rb:17:in `each'
        from /nix/store/wz11skksfy6ps3kprd4g2w8x4c50xas4-gen-bin-stubs.rb:17:in `<main>'
builder for ‘/nix/store/fsnrwx4wkngg8jxgyshyfdfhdj5368z1-test.drv’ failed with exit code 1
error: build of ‘/nix/store/fsnrwx4wkngg8jxgyshyfdfhdj5368z1-test.drv’ failed

Technical details

System: Darwin Nix version: 1.11.7 Nixpkgs version: 17.09pre102933.26625c9 Sandboxing enabled: no

raymondhoagland commented 7 years ago

Installing glibcLocales seems to allow me to properly set the encoding with

export LC_ALL="en_US.utf-8"
export LANG="en_US.utf-8"

at various points, but none of which have any real impact. Adding a custom gemConfig doesn't help because it's not the actual build of the gem that is having issues, and the postBuild action is applied after stubbing so it's already failed.

The best workaround I've been able to come up with so far is to circumvent bundlerEnv by using buildRubyGem and maintaining a secondary gemset.nix file that only has xcode-install and its dependencies. If there's a better way to do this I'd still like to know because this seems like bad practice.

twhitehead commented 5 years ago

I believe the issue you are running into is in building the dependencies for your shell environment, and in the environment those dependencies are build, only the C and POSIX locales are available so it really doesn't matter what you set LANG or LC_* to.

You can see this by looking at the output locale -a gives in the standard builder environment

nix build '(with import <nixpkgs> { }; runCommandCC "locales" { } "locale -a")'
nix-store -l <path of derivation>

C
POSIX

stale[bot] commented 4 years ago

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

Search for maintainers and people that previously touched the related code and @ mention them in a comment.
Ask on the NixOS Discourse.
Ask on the #nixos channel on irc.freenode.net.

NixOS / nixpkgs