Expected AWS S3 Credential Locations are Inconsistent Across `nix` Commands for Daemon Users

[nrdxp]

Describe the bug

I have found some issues with how Nix reads AWS Credentials while setting up a post-build-hook that uploads to an S3 cache in a CI action. This particular action uses the same cache to pull already built resources, and the cache is private and so requires credentials which is how I found the problem:

• For commands that interact with the nix-daemon, such as nix build, the aws credentials file must exist at /root/.aws/credentials. Since there is no way to modify the daemon's environment after it starts, there is no way to pass in any AWS_* vars that would usually change this bahavior.

• For commands that don't interact with the daemon, such as nix copy the credentials are able to be read from the standard AWS credentials chain provided by the AWS SDK. Importantly, these come from the callers environment. So one can set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the environment, callnix copy --to s3://<some-cache> and have it work out.

Steps To Reproduce

1. setup a private s3 bucket that requires credentials to push and pull
2. setup your AWS credentials for your calling user
3. run nix copy --to s3://<your-cache>, if credentials are set properly it should work
4. Setup this same cache as a substituter in your nix.conf
5. run nix build on the same path you just uploaded (you may have to manually delete it to rebuild it). You should get a AWS warning, and the substitution will not take place.
6. now setup a proper credentials file at /root/.aws/credentials and try again, this time it should work

Expected behavior

We should only have to set AWS vars in one place for all commands. Since nix copy runs as calling user, while nix build runs as daemon user, this is currently impossible without setting credentials in at least two places. Perhaps a new nix.conf setting is needed to point to a single credentials file? Although personally, I would prefer it if I could dynamically pass in vars like AWS_ACCESS_KEY_ID to nix build, but that may be much trickier.

nix-env --version output

nix (Nix) 2.5pre20211203_2e606e8

Additional context

If you don't know how to setup an S3 cache to test this behavior, feel free to review the docs on it.

[nrdxp]

I'm going to close for now, as I seem to have mistaken something here. Some of my assumptions here appear to be incorrect and reproduction doesn't work as outline. I've got to do further testing to provide a more useful analysis.

[pinpox]

• Importantly, these come from the callers environment. So one can set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the environment, callnix copy --to s3://<some-cache> and have it work out.

This doesn't work for me. I've set AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID but nix copy ignores those variables. Configuring them in ~/.aws/credentials works, but is not really an option when trying to copy from a CI job

[nrdxp]

I'm reopening for now as this is still an issue. Not sure exactly which commands talk to daemon and which don't but I have shown that nix store ping does work with only local user credentials but nix build does not unless a credentials file exists for the root user, so that's at least a starting point for reproducing. Perhaps the nix.conf needs a setting to point to a credential file location?

[nrdxp]

@toonn, one critical issue here for Mac users that may be pertinent...

There doesn't seem to be a way for Mac users to supply s3 credentials at all for the post-build-hook. This is by virtue of the fact that the hook runs in a very minimal environment, so it cannot pickup credentials that may be set in the calling environment, and also due to the fact that single user mode is no longer supported on Mac, meaning the daemon looks at the root user for AWS credentials, and since Mac does not allow us to write (at least not that I could figure out) to the root user's directory, Mac users are basically SOL if they wanna upload to a protected s3 cache from a mac in a post-build-hook.

[kgtkr]

I solved it by doing sudo cp -R .aws /var/root/.aws. But I want a simple solution

[nrdxp]

I solved it by doing sudo cp -R .aws /var/root/.aws.

Yes, copying credentials to the root user does work, but it's more of a work around than a real solution. I was thinking maybe we need a new nix.conf setting for AWS creds. One can also set AWS_* env vars in the systemd service running NixOS as another (potentially cleaner) workaround.

[kgtkr]

If there is an item in nix.conf that specifies aws credentials like netrc-file, will it be resolved? aws authentication is not compatible with netrc. so netrc is not available. I worked around it by disabling minio authentication, deploying a basic auth proxy and configuring netrc, but it's difficult to use this method in s3.

[nrdxp]

If there is an item in nix.conf that specifies aws credentials like netrc-file, will it be resolved?

I'm saying one solution to the problem might be to add a whole new setting, maybe something like aws-id and aws-key to "nix.conf` as one potential solution. Such options do not currently exist though.

[codygman]

This bug is making trying out nix for my company very difficult since we use a private s3 as our binary cache and have multi-factor auth. Sadly it's currently not feasible to use a public cache.

There might be ways around it, but I haven't discovered them in my finite time I have to put towards this.

That means I have to manually generate a session token every time it expires and update /var/root/.aws/credentials.

I don't think nix will get very much adoption in my company if that's a required step to get cached builds.

[codygman]

Oh... quick idea:

The docs should probably mention this issue as well as give a synopsis of the nix build uses daemon (root), nix copy uses client, and that it can lead to confusing issues like this.

[blaggacao]

@codygman see: https://github.com/cachix/install-nix-action/pull/150

[endgame]

@codygman AIUI the official AWS solution for this problem is IAM Roles Anywhere, but if you don't have PKI set up then the AWS-flavoured version (Certificate Manager) looks like it gets expensive.

Could you stand up a small EC2 instance or something that has access to a role (so you can use an instance profile instead of long-lived keys and SSO) and pushes the AK/SK/Token to your Mac whenever things change?

[bouk]

If you want to configure nix-daemon access to the AWS credentials in your nix config you can do something like the following:

systemd.services.nix-daemon = {
  serviceConfig = {
    Environment = [ "AWS_SHARED_CREDENTIALS_FILE=%d/AWS_SHARED_CREDENTIALS_FILE" ];
    LoadCredential = [ "AWS_SHARED_CREDENTIALS_FILE:/etc/secrets/aws/credentials" ];
  };
};