signed URLs broken when using default ports

[davepacheco]

Here's a working example of using msign to create a signature for a HEAD request:

$ echo $MANTA_URL
https://us-east.manta.joyent.com
$ signature=$(msign -m HEAD /dap/stor)
$ curl -I $signature
HTTP/1.1 200 OK
Connection: close
Content-Type: application/x-json-stream; type=directory
Result-Set-Size: 48
Date: Fri, 12 May 2017 23:27:55 GMT
Server: Manta
x-request-id: a0221a20-376a-11e7-abf0-cffc7df5d15a
x-response-time: 55
x-server-name: 56564894-a7c2-470e-a218-3d859e7e1687

Now if I tack on port 443 explicitly, it doesn't work:

$ export MANTA_URL=https://us-east.manta.joyent.com:443
$ signature=$(msign -m HEAD /dap/stor)
$ curl -I $signature
HTTP/1.1 403 Forbidden
Connection: close
Content-Type: application/json
Content-Length: 99
Content-MD5: o6kfsMz8ATeTfySea8Vqpg==
Date: Fri, 12 May 2017 23:28:12 GMT
Server: Manta
x-request-id: aa59c380-376a-11e7-9d64-49809d3aede9
x-response-time: 6
x-server-name: 3d2b5d91-5cd9-4123-89a5-794f44eab9fd

With a GET, we can see the actual error message:

$ signature=$(msign /dap/stor)
$ curl -i $signature
HTTP/1.1 403 Forbidden
Connection: close
Content-Type: application/json
Content-Length: 99
Content-MD5: o6kfsMz8ATeTfySea8Vqpg==
Date: Fri, 12 May 2017 23:31:08 GMT
Server: Manta
x-request-id: 1355ade0-376b-11e7-be27-4f086f970e06
x-response-time: 5
x-server-name: 1a1ff4e5-4e04-4e60-b993-689f95b67e89

{"code":"InvalidSignature","message":"The signature we calculated does not match the one you sent"}

This problem applies to all signed URLs, whether created with msign or the client's JavaScript interface.

I believe what's going on here is that the signature includes the "Host" header, and when we create the signature, we assume the "host" header will contain the port, but when Node goes to actually construct the Host header, it sees that the port is the default one and leaves it out. As a result, we sign a different Host header than we send, and the signature doesn't match.

The header we sign

To construct the signed url, msign invokes client.signURL. That function assembles a host and passes that to node-smartdc-auth's signUrl, which uses whatever it's given as the host header for the signature. So the header we ultimately sign is the one assembled in client.signURL, which is:

host: require('url').parse(self._url).host,

Now, self._url generally comes from MANTA_URL. We can easily evaluate this and see that we get a different result depending on whether the port is explicitly specified in the URL, even though the URLs are semantically equivalent:

$ node
> process.version
'v0.10.45'

> u = 'https://us-east.manta.joyent.com'
'https://us-east.manta.joyent.com'
> require('url').parse(u).host
'us-east.manta.joyent.com'

> u = 'https://us-east.manta.joyent.com:443'
'https://us-east.manta.joyent.com:443'
> require('url').parse(u).host
'us-east.manta.joyent.com:443'

The header we send

This took me a long time to root out because of a bunyan CLI issue (to be filed shortly). By modifying the code in restify-clients, I worked out that by the time we call Node's http.request() to make the request, we have not set a Host header. Node's default behavior is to add a Host header if it doesn't exist, based on the host to which you're sending the request. And it includes the port number if and only if it's not the default one:

      var hostHeader = host;
      if (port && +port !== defaultPort) {
        hostHeader += ':' + port;
      }
      this.setHeader('Host', hostHeader);

I confirmed that requests sent using the default port have a "Host" header with no port included, but requests sent with a different port have the port number in the "Host" header. (I confirmed this by using nc -l to set up a little server and running "mls" against it with different "http" URLs.)

Fix?

Edit: this is wrong -- see comment below.

It seems we could either:

1. Change the host header that we sign to leave out the port number if it's the default port for this protocol, or
2. We could explicitly pass our desired "host" header down from the Manta client, through the restify client, all the way to Node.

If we do (1), we're still subject to implicit behavior: Node could change its behavior here and that would break us again. If we do (2), then we're changing the over-the-wire behavior of the client. I don't know if this matters, but it's normally unusual to have a port in the Host header, and we'd now be sending that for every request.

We could do both (1) and (2): leave the port out of the header if it's the default port and pass the resulting host header down to the restify client.

[davepacheco]

The bunyan issue is node-bunyan#504.

[davepacheco]

The fix I suggested above doesn't make any sense. We don't control the Host header sent on the subsequent request -- the user in this case could have given this signed URL to curl or some other user agent that may or may not decide to provide the port number in the Host header.

One option may be to fix this on the server by saying if the signature fails to match, and the Host header has the default port, then we try the signature again assume that the default port had been passed in the Host header.