S3 binary caches don't work with network proxies

[Fuuzetsu]

Describe the bug

We can't use S3 binary caches with network proxies.

This was hinted at in #3529 . But the symptom is worse: if you try to do nix-build or something, it'll hang somewhere in AWS SDK code and you have to open second terminal and kill -9 the nix process.

Steps To Reproduce

• set HTTP_PROXY and such values for your nix-daemon
• set s3:// binary cache
• run some command like nix-build '<nixpkgs>' -A hello --no-out-link for a package you don't have.

Expected behavior

Binary cache is queried properly.

nix-env --version output

nix-env (Nix) 2.3.3

Looking at source, I have no reason to believe this is not the case still in HEAD.

Additional context

As per https://github.com/aws/aws-sdk-cpp/issues/1049 , the values for any proxy have to be explicitly set by the SDK user. If we look at s3-binary-cache-store.cc, you'll see that nowhere does it do anything like this.

It is pretty annoying that they unset it because now nix presumably has to figure out what the proxy values are set, parse them, pass them to AWS SDK and have that just spit it back out.

[Fuuzetsu]

A possible quick workaround that's easier than trying to read HTTP_PROXY and such would be to make new variables like NIX_AWS_SDK_PROXY_HOST and NIX_AWS_SDK_PROXY_PORT and pass those to the SDK. It is up to the user to then set these properly rather than having nix try to figure it out from HTTP_PROXY and others.

[Fuuzetsu]

For anyone stumbling here, we have a following work-around. We have a location that has no Internet access and needs to use a proxy.

• Have a machine somewhere that can access the bucket without a proxy itself but that has connectivity to the original subnet. One such machine may be the machine hosting your HTTP proxy to start with.
• On this machine, listen on the virtual hostname <bucket>.s3.<region>.amazonaws.com.
• Generate self-signed SSL certificates for this hostname as the SDK wants to use HTTPS. You probably don't own amazonaws.com so we need to use self-signed certs.
• Make sure your proxy has the certs loaded. Distribute the certificate to the machines that want to use the cache.
• Add <bucket>.s3.<region>.amazonaws.com to point at your proxy IP or do whatever else to resolve the hostname to your IP.

With nginix, the setup (ruby templated) may look like

server {
    listen 443 ssl;

    server_name <%= @s3_bucket %>.s3.<%= @s3_endpoint_region %>.amazonaws.com;

    resolver <%= @s3_endpoint_resolver %>;

    location / {
        proxy_pass https://<%= @s3_bucket %>.s3.<%= @s3_endpoint_region %>.amazonaws.com;
    }

    ssl_certificate <%= @ssl_certificate %>;
    ssl_certificate_key <%= @ssl_certificate_key %>;

    access_log /var/log/nginx/site-<%= @s3_bucket %>.s3.<%= @s3_endpoint_region %>.amazonaws.com-access.log;
    error_log /var/log/nginx/site-<%= @s3_bucket %>.s3.<%= @s3_endpoint_region %>.amazonaws.com-error.log;
}

You can see it pass the request to the real AWS.

This is necessary as the AWS SDK will sign the requests, including the hostname it uses. So we can't use a different hostname.

[stale[bot]]

I marked this as stale due to inactivity. → More info

[Fuuzetsu]

Still very much an issue that we can't use HTTP proxy with S3 caches

[Fuuzetsu]

I need to add some info to https://github.com/NixOS/nix/issues/4883#issuecomment-855515862

As libcurl uses libresolve, to actually end up using your nginx proxy you need to make sure the AWS address resolves to your proxy address: /etc/hosts doesn't work as libresolve doesn't read it. For us we happen to have a DNS that all these machines use so I followed https://www.redpill-linpro.com/sysadvent/2015/12/08/dns-rpz.html and added the S3 endpoint to bind config.

[stale[bot]]

I marked this as stale due to inactivity. → More info

[Fuuzetsu]

Not stale unless we explicitly change how we use the AWS SDK.

[abathur]

Probably still important :)

[itzwam]

Patched https://github.com/NixOS/nix/blob/master/src/libstore/s3-binary-cache-store.cc#L146 to enable https://sdk.amazonaws.com/cpp/api/LATEST/aws-cpp-sdk-core/html/struct_aws_1_1_client_1_1_client_configuration.html#a0197eb33dffeb845f98d14e5058921c1

seems like I can query packages through the proxy

will PR soon™️

[romain-neil]

Hello, I open a pull request to try to keep track of resolution of this issue.