yarn2nix doesn't use .npmrc settings

loafofpiecrust commented 2 years ago

Describe the bug

The mkYarnPackage function doesn't apply settings from .npmrc. I use font awesome pro, which requires setting an authentication token in .npmrc. When I try to build a package depending on it, I get a 401 because the auth token isn't included in the request.

Edit: Looks like the real issue is actually getting Nix to include the auth token header when downloading packages to the nix store.

Steps To Reproduce

Steps to reproduce the behavior:

Add .npmrc to your repo with similar contents:

@fortawesome:registry=https://npm.fontawesome.com/
//npm.fontawesome.com/:_authToken=TOKEN

Add font awesome pro as dependency. (instructions)
Try to build the package with mkYarnPackage

I know reproducing this exact situation may be unreasonable since it's a paid package, so there's probably a similar test you could run.

Expected behavior

All settings in .npmrc applied when building a yarn package.

Additional context

In the code, looks like yarn2nix may just need to copy over .npmrc during the buildPhase.

Notify maintainers

@Stunkymonkey @happysalada

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

 - system: `"x86_64-darwin"`
 - host os: `Darwin 20.6.0, macOS 10.16`
 - multi-user?: `no`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.4pre20210802_47e96bb`
 - channels(taylor@outschool.com): `"darwin, nixpkgs-21.11pre312229.08ef0f28e3a"`
warning: warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
 - nixpkgs: `/Users/taylor@outschool.com/.nix-defexpr/channels/nixpkgs`

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute: [ mkYarnPackage ]

happysalada commented 2 years ago

Hey, thanks for reporting this!

Are you using mkYarnPackage with a yarn.nix file ? Or are you trying to use the yarn.lock directly ?

I think yarn.nix will try to just fetch that dependendency without any secrets. This seems to be a case where you need to override the definition for that dependency with your own. Could you use an approach similar to the one described in https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/javascript.section.md#overriding-dependency-behavior Unfortunately I don't see a way where you don't have to include your token in the override. Perhaps you don't commit that file to git and just show a template for others who want to do the same ?

loafofpiecrust commented 2 years ago

@happysalada Thanks for the quick response. I'm using yarn.lock directly, no intermediate yarn.nix. I read the doc you sent, and I've tried adding preBuild to the pkgConfig for the fontawesome packages where I copy in the .npmrc. I also tried adding yarnPreBuild to do the same. Here's the gist of it:

yarnPreBuild = ''
  cp ${./.npmrc} ./.npmrc
'';

Neither worked. Maybe I'm putting it in the wrong place, or maybe I need to tweak a different setting to get the build to use the auth token header rather than .npmrc here?

How would you configure one package to use an auth token header when fetching it? It's fine if it's in the derivation for now, I can figure out how to manage the secret after I just get something to work.

happysalada commented 2 years ago

I realise now my explanation wasn't super clear, sorry about that! :-)

yarn2nix without a yarn.nix will basically try to create a yarn.nix from the yarn.lock. It won't however respect the .npmrc, even if it is added.

What I had in mind was using the pkgConfig to override the source attribute so that it fetches the source with the token. Checking at what yarn2nix code does however I now realise that won't be possible.

Here is what I think would be the easiest way (even if it's not exactly ideal).

use yarn2nix to generate a yarn.nix. That is a file that contains all the dependency, how and where to fetch them. In that file you can replace your dependency with the url including the token.
This is not ideal in that it adds a secret to a file that shouldn't have it. So the counterpart is that you should probably not commit that file to your repo and just keep it on local.

I'm not the one who wrote the original yarn2nix and we are currently trying to make the next version of this tool. @DavHau we have to potentially consider this use case of dependencies that require secrets (I have no good idea yet on how to handle that, just saying it here so we keep it in mind somewhere).

@loafofpiecrust let me know if any of this doesn't make sense or if you struggle with anything.

loafofpiecrust commented 2 years ago

Thanks for the elaboration! I'll try generating a yarn.nix file and adding the auth token there. It's definitely not ideal, but it sounds like it'll work. In terms of handling this in future versions of yarn2nix, I would be fine with passing in a path to a file with my token or even using something like agenix if necessary. The big thing is leaking a secret into the nix store. If I have a private repo, I might be able to commit my secret to git through yarn.nix or otherwise, but it'll put the secret in the store, which I don't want because then it might stick around on CI servers, cachix, or my local machine.

happysalada commented 2 years ago

You're definitely right here. The idea would be to enable an override of a particular dependency with a path to read the secret. I think that use case has never been envisioned yet. Hopefully designing the next version of the tool, we can try to make this possible.

DavHau commented 2 years ago

I think authentication for fetching is nothing that can be implemented nicely in any nix library. Authentication must be supported by builtin fetchers directly, as it is already by builtins.fetchGit (using ssh authentication for example). I think it is best to avoid any fetching that requires authentication. @iosmanthus in your case it seems pretty straight forward to just download the file manually and self host it somewhere. As an alternative work-around you could set up a local proxy that adds the authentication for you, so that nix can fetch the file via that proxy without having to deal with authentication. But I think this is not necessary in your case.

loafofpiecrust commented 2 years ago

So I read a little bit more about this issue in general. @DavHau I don't think the answer is "avoid fetching that requires authentication" because that basically rules out any business use, and we can't expect all dependencies to be free all the time. @happysalada It looks like builtins.fetchurl actually supports a .netrc file where you can specify secrets (link)! This seems like an easy solution for me, the only catch right now is that yarn2nix uses pkgs.fetchurl instead of builtins.fetchurl. The only practical difference for this tool seems to be that it generates sha1 hashes (which are no longer supported) instead of sha256. It appears to me if yarn2nix generated yarn.nix files with sha256 or sha512 (which yarn generates!) hashes that use builtins.fetchurl, there's at least a system-level solution in netrc.

DavHau commented 2 years ago

I'd assume that builtins.fetchurl supports verifying sha1 hashes as well. Does it not? Maybe you can just make an overlay that replaces pkgs.fetchurl with builtins.fetchurl for your derivation.

It is interesting so see that different fetchers are preferred in different situations. This will help the development of dream2nix. There is already an allowBuiltinFetchers option. We should probably also add a preferBuiltinFetchers option. Also automatically re-calculating hashes to update outdated hash formats is already part of the concept.

loafofpiecrust commented 2 years ago

I thought builtins.fetchurl only supports sha256 hashes because sha1 is not considered as secure anymore (wikipedia has a good summary). I tried to just use builtins.fetchurl in my flake.nix and got errors because it didn't recognize the sha1 attribute. This means yarn2nix needs modification to support it, not just an overlay.

I like the idea of dream2nix, thanks for linking the project.

loafofpiecrust commented 2 years ago

I actually found an alternate solution to this issue than using builtins.fetchurl and a global netrc file. pkgs.fetchurl actually supports curlOpts, which allows me to pass headers to the request. At the moment, this means generating yarn.nix manually with yarn2nix, then my fontawesome dependencies look like this:

    {
      name = "https___npm.fontawesome.com__fortawesome_free_brands_svg_icons___5.15.4_free_brands_svg_icons_5.15.4.tgz";
      path = fetchurl {
        name = "https___npm.fontawesome.com__fortawesome_free_brands_svg_icons___5.15.4_free_brands_svg_icons_5.15.4.tgz";
        url  = "https://npm.fontawesome.com/@fortawesome/free-brands-svg-icons/-/5.15.4/free-brands-svg-icons-5.15.4.tgz";
        sha1 = "ec8a44dd383bcdd58aa7d1c96f38251e6fec9733";
        curlOpts = ''-K ${./.fontawesome}'';
      };
    }

And my .fontawesome file looks like this:

-H "Authorization: Bearer XXXXX"

With this method, I can either encrypt the file with agenix or live with it being in the store, but at least I can do it. Now, to make this particular solution work directly from yarn.lock, I just need a way to add extra options to specific dependency fetchurl calls during yarn2nix. Alternatively, to pass in an alternate fetchurl field.

happysalada commented 2 years ago

nice find! The problem with this it seems is that your secret will end up in the nix store. In this case, you don't like using a path and having the file directly available on disc ? You could potentially use nix to make sure that file ends up in etc for example and instead of fetchurl, just use the reference to that path.

loafofpiecrust commented 2 years ago

Yeah so I am passing a path to curl, ultimately. I just started here with putting it directly in the store for a test. Like you said, in production I might use a secret solution like agenix to put the encrypted file at a fixed path that I can reference instead. I think my PR should allow this use case.

stale[bot] commented 2 years ago

I marked this as stale due to inactivity. → More info

NixOS / nixpkgs