shelfio / dynamodb-parallel-scan

Scan large DynamoDB tables faster with parallelism
MIT License
71 stars 10 forks source link

Getting UnrecognizedClientException error #233

Open camilalbmaia opened 1 year ago

camilalbmaia commented 1 year ago

I'm following the steps to use dynamodb-parallel-scan, but I'm getting the following error with the two latest versions (3.4.0 and 3.5.0):

UnrecognizedClientException: The security token included in the request is invalid.
      at throwDefaultError ([/var/task/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:8:22](mailto:/var/task/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:8:22))
      at [/var/task/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:18:39](mailto:/var/task/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:18:39)
      at de_DescribeTableCommandError ([/var/task/node_modules/@aws-sdk/client-dynamodb/dist-cjs/protocols/Aws_json1_0.js:1191:20](mailto:/var/task/node_modules/@aws-sdk/client-dynamodb/dist-cjs/protocols/Aws_json1_0.js:1191:20))
      at processTicksAndRejections (internal/process/task_queues.js:95:5)
      at async [/var/task/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24](mailto:/var/task/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24)
     at async [/var/task/node_modules/@aws-sdk/middleware-signing/dist-cjs/awsAuthMiddleware.js:14:20](mailto:/var/task/node_modules/@aws-sdk/middleware-signing/dist-cjs/awsAuthMiddleware.js:14:20)
      at async [/var/task/node_modules/@aws-sdk/middleware-retry/dist-cjs/retryMiddleware.js:27:46](mailto:/var/task/node_modules/@aws-sdk/middleware-retry/dist-cjs/retryMiddleware.js:27:46)
      at async [/var/task/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26](mailto:/var/task/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26)
      at async getTableItemsCount ([/var/task/node_modules/@shelf/dynamodb-parallel-scan/lib/ddb.js:42:16](mailto:/var/task/node_modules/@shelf/dynamodb-parallel-scan/lib/ddb.js:42:16))
      at async Object.parallelScanAsStream ([/var/task/node_modules/@shelf/dynamodb-parallel-scan/lib/parallel-scan-stream.js:25:26](mailto:/var/task/node_modules/@shelf/dynamodb-parallel-scan/lib/parallel-scan-stream.js:25:26)) { 
    '$fault': 'client',
    '$metadata': {
      httpStatusCode: 400,
      requestId: '9N2RIU65LR9UBHDFLOADD656OVVV4KQNSO5AEMVJF66Q9ASUAAJG',
      extendedRequestId: undefined,
      cfId: undefined,
      attempts: 1,
      totalRetryDelay: 0
    },
    __type: 'com.amazon.coral.service#UnrecognizedClientException'
  }

It works fine with version 3.3.0.

Do I need to use it in a different way for the latest versions? I'm using 'parallelScanAsStream'.

vladholubiev commented 1 year ago

@harazdovskiy do you think it's because of recent changes?

dan-doran-fp commented 1 year ago

This has begun happening for me, since upgrading, as well. I've set AWS_REGION, AWS_ACCESS_KEY_ID, and AWS_SECRET_ACCESS_KEY in the usual way. Thank you for any help!

UnrecognizedClientException: The security token included in the request is invalid.
    at throwDefaultError (/Users/dan/FilmpacProjects/music-api/node_modules/@smithy/smithy-client/dist-cjs/default-error-handler.js:8:22)
    at /Users/dan/FilmpacProjects/music-api/node_modules/@smithy/smithy-client/dist-cjs/default-error-handler.js:18:39
    at de_DescribeTableCommandError (/Users/dan/FilmpacProjects/music-api/node_modules/@aws-sdk/client-dynamodb/dist-cjs/protocols/Aws_json1_0.js:1191:20)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /Users/dan/FilmpacProjects/music-api/node_modules/@smithy/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
    at async /Users/dan/FilmpacProjects/music-api/node_modules/@aws-sdk/middleware-signing/dist-cjs/awsAuthMiddleware.js:14:20
    at async /Users/dan/FilmpacProjects/music-api/node_modules/@smithy/middleware-retry/dist-cjs/retryMiddleware.js:27:46
    at async /Users/dan/FilmpacProjects/music-api/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26
    at async getTableItemsCount (/Users/dan/FilmpacProjects/music-api/node_modules/@shelf/dynamodb-parallel-scan/lib/ddb.js:42:16)
    at async parallelScan (/Users/dan/FilmpacProjects/music-api/node_modules/@shelf/dynamodb-parallel-scan/lib/parallel-scan.js:20:26) {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 400,
    requestId: '8ARFRTBNBBM74VR4B6PUTEDU13VV4KQNSO5AEMVJF66Q9ASUAAJG',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  __type: 'com.amazon.coral.service#UnrecognizedClientException'
}
harazdovskiy commented 1 year ago

I guess this should fix the issue https://github.com/shelfio/dynamodb-parallel-scan/pull/235

harazdovskiy commented 1 year ago

@dan-doran-fp, please check whether your issue is resolved with the new version 3.5.2

jakedaleweb commented 1 year ago

Hi @harazdovskiy I hit this issue and tried 3.5.2 there is a new error now.

CredentialsProviderError: Could not load credentials from any providers

image
harazdovskiy commented 1 year ago

@jakedaleweb please try v3.5.3 it should be fixed now. Big thanks to @mpushkin!

peterabbott commented 1 year ago

@harazdovskiy have given 3.5.3 a try and the problem still exists.

It would be helpful if the credentials here could be linked to AwsCredentialIdentityProvider to be able to retreive credentials from the metadata service as required.

For example, something like this below. Then the getCredentials function could fetch the details when required.

export type Credentials = AwsCredentialIdentityProvider | {
  accessKeyId: string;
  secretAccessKey: string;
  sessionToken: string;
};
dan-doran-fp commented 1 year ago

@dan-doran-fp, please check whether your issue is resolved with the new version 3.5.2

The latest version is working for me. Fwiw, I set the AWS_PROFILE environment variable.

Thank you very much!

bernier commented 1 year ago

I still have to go back to 3.3.0. Can you please explain (and update the readme) how to setup the connexion for the latest version? Thanks