search

mhart / aws4

Signs and prepares Node.js requests using AWS Signature Version 4
MIT License
699 stars 175 forks source link

can this work with selling-partner api? possibly related to ignored headers #121

Closed ericblade closed 3 years ago

ericblade commented 3 years ago

I haven't quite yet figured out how to get a request signed by this through to work with SP-API, although I suspect it might be pretty close to it. If anyone knows a way to do it already, that'd be awesome, otherwise I think I'm going to do some poking around with this..

I do see this recent commit https://github.com/mhart/aws4/commit/1c5a4b67318b1d21b6e0309658b5c78b5162b791 that strips certain headers from the signing, but in the sp-api documentation, i see that it expects things like User-Agent to be included in the sign.

At the very least, I would think it would be useful to be able to configure any headers to be ignored, in the call to sign. I'm not sure if just making sure the headers signed in the example are the ones signed, will get us there. Amazon isn't exactly the best at telling us what we're doing wrong when we make an error. :-D

mhart commented 3 years ago

Doubtful related to ignored headers – the Selling Partner API Models uses the official Java SDK client, which is where I got the ignored headers from in the first place (v2 of the API)

It's a bit hard to debug without code – can you please show what you're trying?

ericblade commented 3 years ago

sp-api documentation shows that it is expecting user-agent to be included. https://github.com/amzn/selling-partner-api-docs/blob/main/guides/developer-guide/SellingPartnerApiDeveloperGuide.md#authorization-header

(if that link doesn't go right to it, search for "Authorization header")

My apologies, i meant to include that link in my original post and forgot.

It's entirely possible that i'm completely mucking something up, I'm having a bit of a hard time with following the docs on it, they're a bit clunky, and jump all over the place. . . . and i'm also attempting to do something that's a bit .. perhaps unorthodox.

I'm attempting to put together a module that uses swagger-client to dynamically generate an API, and then inject the proper headers and signing into it. I feel like I'm almost there, because last night when I was working on it, I had managed to work through several other errors, and finally landed on a "signature does not match" problem

Here's my code

    requestInterceptor = async (req: any) => { // req is a Request, but what the hell kind of Request? headers.append isn't there, and default Request headers is readonly.
        // https://gist.github.com/davidkelley/c1274cffdc0d9d782d7e
        function amzLongDate(date: Date) {
            return date.toISOString().replace(/[:\-]|\.\d{3}/g, '').substr(0, 17);
        }
        const requestDate = new Date();
        req.headers = {
            host: RegionServers[this.region].endpoint,
            'x-amz-access-token': (await this.lwa!.getAccessToken()) as string,
            'x-amz-date': amzLongDate(requestDate),
            'user-agent': 'sp-api-simple/0.1 (Language=JavaScript; Platform=Node)',
        }
        console.warn('**** requestInterceptor', req);
        const signedReq = aws4.sign({ ...req, service: 'execute-api' }, { secretAccessKey: this.clientSecret, accessKeyId: 'AKIAQJSLQX2LEXYO3CZC' });
        console.warn('**** sign result', signedReq);
        return signedReq;
        // return req;
    }

What I'm attempting to sign (and i'm not 100% positive this is all correct, but it seems to match with the sp-api documentation) looks like

{
  url: 'https://sellingpartnerapi-na.amazon.com/authorization/v1/authorizationCode?sellingPartnerId=1&developerId=1&mwsAuthToken=1',
  credentials: 'same-origin',
  headers: {
    host: 'sellingpartnerapi-na.amazon.com',
    'x-amz-access-token': 'Atza|IwEBIH...CGRHZ',
    'x-amz-date': '20201117T012143Z',
    'user-agent': 'sp-api-simple/0.1 (Language=JavaScript; Platform=Node)'
  },
  requestInterceptor: [AsyncFunction (anonymous)],
  method: 'GET'
}

this is what i get out of aws4:

{
  url: 'https://sellingpartnerapi-na.amazon.com/authorization/v1/authorizationCode?sellingPartnerId=1&developerId=1&mwsAuthToken=1',
  credentials: 'same-origin',
  headers: {
    host: 'sellingpartnerapi-na.amazon.com',
    'x-amz-access-token': 'Atza|...GRHZ',
    'x-amz-date': '20201117T012143Z',
    'user-agent': 'sp-api-simple/0.1 (Language=JavaScript; Platform=Node)',
    Authorization: 'AWS4-HMAC-SHA256 Credential=AKIAQJSLQX2LEXYO3CZC/20201117/us-east-1/execute-api/aws4_request, SignedHeaders=host;x-amz-access-token;x-amz-date, Signature=a92fc01aa175a1f1b26f3a4b2c56fff612f6c74dc5fc8d9a2cf3b59a80a640b9'
  },
  requestInterceptor: [AsyncFunction (anonymous)],
  method: 'GET',
  service: 'execute-api',
  hostname: 'sellingpartnerapi-na.amazon.com',
  path: '/'
}

... and lastly, this is what i get back from Amazon, with an error Forbidden 403:

{
  status: 403,
  statusCode: 403,
  response: {
    ok: false,
    url: 'https://sellingpartnerapi-na.amazon.com/authorization/v1/authorizationCode?sellingPartnerId=1&developerId=1&mwsAuthToken=1',
    status: 403,
    statusText: 'Forbidden',
    headers: {
      connection: 'close',
      'content-length': '1132',
      'content-type': 'application/json',
      date: [ 'Tue', '17 Nov 2020 01:21:43 GMT' ],
      'x-amz-apigw-id': 'WILSPGlvIAMFlMw=',
      'x-amzn-errortype': 'InvalidSignatureException',
      'x-amzn-requestid': 'a04b2e04-656c-430a-b80b-eb280958c5e9'
    },
    text: '{\n' +
      '  "errors": [\n' +
      '    {\n' +
      '      "message": "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n' +
      '\n' +
      'The Canonical String for this request should have been\n' +
      "'GET\n" +
      '/authorization/v1/authorizationCode\n' +
      'developerId=1&mwsAuthToken=1&sellingPartnerId=1\n' +
      'host:sellingpartnerapi-na.amazon.com\n' +
      'x-amz-access-token:Atza|...RHZ\n' +
      'x-amz-date:20201117T012143Z\n' +
      '\n' +
      'host;x-amz-access-token;x-amz-date\n' +
      "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'\n" +
      '\n' +
      'The String-to-Sign should have been\n' +
      "'AWS4-HMAC-SHA256\n" +
      '20201117T012143Z\n' +
      '20201117/us-east-1/execute-api/aws4_request\n' +
      "35df85cd77465f7dd987d464d46272d6673b8c654f1bfbb1aa74292b9c6c1d2a'\n" +
      '",\n' +
      '     "code": "InvalidSignature"\n' +
      '    }\n' +
      '  ]\n' +
      '}',
    data: '{\n' +
      '  "errors": [\n' +
      '    {\n' +
      '      "message": "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n' +
      '\n' +
      'The Canonical String for this request should have been\n' +
      "'GET\n" +
      '/authorization/v1/authorizationCode\n' +
      'developerId=1&mwsAuthToken=1&sellingPartnerId=1\n' +
      'host:sellingpartnerapi-na.amazon.com\n' +
      'x-amz-access-token:Atza...GRHZ\n' +
      'x-amz-date:20201117T012143Z\n' +
      '\n' +
      'host;x-amz-access-token;x-amz-date\n' +
      "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'\n" +
      '\n' +
      'The String-to-Sign should have been\n' +
      "'AWS4-HMAC-SHA256\n" +
      '20201117T012143Z\n' +
      '20201117/us-east-1/execute-api/aws4_request\n' +
      "35df85cd77465f7dd987d464d46272d6673b8c654f1bfbb1aa74292b9c6c1d2a'\n" +
      '",\n' +
      '     "code": "InvalidSignature"\n' +
      '    }\n' +
      '  ]\n' +
      '}',

Note that I'm not necessarily trying to make a request with meaningful results into the sp-api, right now i'm just trying to get a request successfully through TO any of the sp-api functions

mhart commented 3 years ago

The error's not saying anything about a missing header – it's saying the signature isn't matching.

I can't see where you're giving aws4 the path, for example. Check out the docs for usage: https://github.com/mhart/aws4#example

ericblade commented 3 years ago

right, and i have no idea what it's trying to sign versus what amazon's expecting it to sign.

I guess I need to parse that out of the url? I'll give it a spin, thanks for the advice. Between the sp-api docs, and the docs for this, I have absolutely no idea what i'm trying to get to converge. :-S

mhart commented 3 years ago

aws4 uses standard Node.js http options as shown in the docs: https://github.com/mhart/aws4#aws4signrequestoptions-credentials and it adds x-amz-date for you.

Try this:

const https = require('https')
const aws4  = require('aws4')

let opts = {
  service: 'execute-api',
  host: 'sellingpartnerapi-na.amazon.com',
  path: '/authorization/v1/authorizationCode?sellingPartnerId=1&developerId=1&mwsAuthToken=1',
  headers: {
    'x-amz-access-token': myLwaAccessToken,
    'user-agent': 'sp-api-simple/0.1 (Language=JavaScript; Platform=Node)',
  },
}

aws4.sign(opts)

https.request(opts, res => res.pipe(process.stdout)).end()
ericblade commented 3 years ago

yeah, i'm having a feeling that the Request object that swagger-client is wanting is .. different .. from the standard node request object. so just passing the incoming request through aws4 isn't going to work.. i think i need to do a bit of translation between them .. working that out. Thanks for your help!! Super appreciated. I also just realized looking at this thread that i'm not using Sandbox like I thought I was, so.. also helpful.

mhart commented 3 years ago

It looks like it uses fetch: https://github.com/swagger-api/swagger-js/blob/master/docs/usage/http-client.md

Which came a lot later than Node.js. Documented here: https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API

They're probably using https://www.npmjs.com/package/node-fetch to get it to work in Node.js – it's not supported natively in Node.js yet, but will be soon.

I have a fetch-compatible aws4 signer here: https://github.com/mhart/aws4fetch

But it's more intended for environments that have fetch and subtle crypto (like browsers or Cloudflare workers). You could probably get it working with Node.js, but you'd need to pull in a bunch of modules.

ericblade commented 3 years ago

ok, so, i'm trying this, just converting the request from swagger-client into options palatable to aws4, then passing it to https, and then sleeping forever so that swagger doesn't end up triggering an error

        const u = new URL(req.url);
        const opts = {
            service: 'execute-api',
            host: u.hostname,
            path: `${u.pathname}${u.searchParams?'?':''}${u.searchParams}`,
            headers: {
                'x-amz-access-token': (await this.lwa!.getAccessToken()) as string,
                'user-agent': 'sp-api-simple/0.1 (Language=JavaScript; Platform=Node)',
            },
        };

        const signedOpts = aws4.sign(opts, { secretAccessKey: this.clientSecret, accessKeyId: 'AKIAQJSLQX2LEXYO3CZC' });
        console.warn('* signedOpts=', signedOpts);
        https.request(signedOpts, res => res.pipe(process.stdout)).end();

output

* signedOpts= {
  service: 'execute-api',
  host: 'sandbox.sellingpartnerapi-na.amazon.com',
  path: '/authorization/v1/authorizationCode?sellingPartnerId=1&developerId=1&mwsAuthToken=1',
  headers: {
    'x-amz-access-token': 'Atza|IwE..............Ko23Bdf6vB',
    'user-agent': 'sp-api-simple/0.1 (Language=JavaScript; Platform=Node)',
    Host: 'sandbox.sellingpartnerapi-na.amazon.com',
    'X-Amz-Date': '20201117T042511Z',
    Authorization: 'AWS4-HMAC-SHA256 Credential=AKIAQJSLQX2LEXYO3CZC/20201117/us-east-1/execute-api/aws4_request, SignedHeaders=host;x-amz-access-token;x-amz-date, Signature=5ffc532955357f201ae6e62a0412b2e1006ea454a4d07062a5780e379615399a'
  }
}
{
  "errors": [
    {
      "message": "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

The Canonical String for this request should have been
'GET
/authorization/v1/authorizationCode
developerId=1&mwsAuthToken=1&sellingPartnerId=1
host:sandbox.sellingpartnerapi-na.amazon.com
x-amz-access-token:Atza|Iw....................hKo23Bdf6vB
x-amz-date:20201117T042511Z

host;x-amz-access-token;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'

The String-to-Sign should have been
'AWS4-HMAC-SHA256
20201117T042511Z
20201117/us-east-1/execute-api/aws4_request
6173e1e93cdd573c329a17e527b11d7c93e8fb4d7bae0dca6a21dba925153405'
",
     "code": "InvalidSignature"
    }
  ]
}

in comparing this to the ps-api documentation, that documentation has SignedHeaders=host;user-agent;x-amz-access-token;x-amz-date versus SignedHeaders=host;x-amz-access-token;x-amz-date here, and i just simply do not have anywhere near enough knowledge about what i'm working with to know if that's the source of the problem, or if i'm chasing the wrong squirrel

mhart commented 3 years ago

The fact that you got the signing to work (only to have a MissingAuthenticationToken error) means that it's not an issue with this library. The signature error you see there doesn't have user-agent in the expected canonical string either. It's almost certain that you're just not passing the params correctly to the http client you're using.

My advice is to get it to work with plain old Node.js code first and then figure out what you need to translate your params to work with swagger-client.

ericblade commented 3 years ago

I do appreciate the help. That's what I'm working with right now, is just plain https module, and still slamming into InvalidSignature. I'll keep plugging at it, and if I come up with anything that might relate to this, i'll let you know. Thanks!

mhart commented 3 years ago

Before you got an MissingAuthenticationToken error with the plain https module though – not a signing error.

What's the result of this?

const https = require('https')
const aws4  = require('aws4')

let opts = {
  service: 'execute-api',
  host: 'sellingpartnerapi-na.amazon.com',
  path: '/authorization/v1/authorizationCode?sellingPartnerId=1&developerId=1&mwsAuthToken=1',
  headers: {
    'x-amz-access-token': 'Atza|Iw....................hKo23Bdf6vB',
    'user-agent': 'sp-api-simple/0.1 (Language=JavaScript; Platform=Node)',
  },
}

aws4.sign(opts)

https.request(opts, res => res.pipe(process.stdout)).end()
mhart commented 3 years ago

Here's how I know the signing is working correctly:

If I run the code above (with my credentials set in AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env vars), I get the following error:

{
  "errors": [
    {
      "message": "Access to requested resource is denied.",
      "code": "Unauthorized",
      "details": "The access token you provided is revoked, malformed or invalid."
    }
  ]
}

However, if I mess with the Authorization header, which is where the signing is happening, like this:

aws4.sign(opts)
opts.headers.Authorization = opts.headers.Authorization.replace(/..$/, '00')

Then I get an InvalidSignature error, not an access token error

So the signature is validated before the access token is validated. And I can successfully get past the signature validation before I run into access token validation (because I'm not actually passing a valid access token)

ericblade commented 3 years ago

I get

{
  "errors": [
    {
      "message": "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

The Canonical String for this request should have been
'GET
/authorization/v1/authorizationCode
developerId=1&mwsAuthToken=1&sellingPartnerId=1
host:sellingpartnerapi-na.amazon.com
x-amz-access-token:Atza|Iw....................hKo23Bdf6vB
x-amz-date:20201117T211906Z

host;x-amz-access-token;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'

The String-to-Sign should have been
'AWS4-HMAC-SHA256
20201117T211906Z
20201117/us-east-1/execute-api/aws4_request
761d3f3c6333813cb1178f4e5b9965250e8a70644b3337d173ed7fb02b27f917'
",
     "code": "InvalidSignature"
    }
  ]
}
mhart commented 3 years ago

You're literally copying and pasting the code from https://github.com/mhart/aws4/issues/121#issuecomment-729207266 into a file (say index.js) and running node index.js and you're getting that error?

If so, I mean... I don't really know what to say. What Node.js version are you using?

ericblade commented 3 years ago

only other things i can come up with off the top, are maybe i'm using the entirely wrong thing for the secret and access key? or i've completely messed up my configuration in AWS somehow

normally using node 15 in windows, but here's node 13 in wsl/linux

:/mnt/c/src/sp-api-simple/src$ node test.js
(node:418) ExperimentalWarning: The ESM module loader is experimental.
{
  "errors": [
    {
      "message": "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

The Canonical String for this request should have been
'GET
/authorization/v1/authorizationCode
developerId=1&mwsAuthToken=1&sellingPartnerId=1
host:sellingpartnerapi-na.amazon.com
x-amz-access-token:Atza|Iw....................hKo23Bdf6vB
x-amz-date:20201118T033312Z

host;x-amz-access-token;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'

The String-to-Sign should have been
'AWS4-HMAC-SHA256
20201118T033312Z
20201118/us-east-1/execute-api/aws4_request
47f343effe66e5a4b0a7f24b8e525e4646f3399288d04e9eb8cb4aac9c5d7473'
",
     "code": "InvalidSignature"
    }
  ]
mhart commented 3 years ago

The ESM module loader is experimental – why is it saying that?

ericblade commented 3 years ago

the source directory is under a package.json that has type: "module" set

i can back it out, just a few moments to reset it to use require instead of import

.. edit: results are same in same environments, regardless of using modules or not

mhart commented 3 years ago

Yeah, only thing I can think would be your credentials then – most APIs are a little nicer with their errors if there's something wrong with the credentials, but this might be different.

Try just with any old random (legitimate) AWS credentials – it won't matter if the credentials have access to the API or not (eg, I just used my credentials and I don't have access) – at least then you can get past the signature error.

ericblade commented 3 years ago

I think I discovered what the problem was, overall -- I had confused my LWA secret with my AWS secret. Too many secrets and access codes and things going around in this stuff. :-D

I really, really appreciate your help with this. I am now down to

{
  "errors": [
    {
      "message": "Access to requested resource is denied.",
     "code": "Unauthorized",
     "details": ""
    }
  ]
}

which means it at least accepts everything i sent it. :-D thank you, thank you

ericblade commented 3 years ago

fwiw, on top of all that, it turns out that the selling-partner-api documentation is basically completely broken -- it has you configure the access permissions in a way that requires additional methods to be called that it never once touches on. Seems the "how to setup" and "how to use" portions were quite possibly written independently of each other, and not in a compatible fashion. :-D