iot-data-plane client - Forbidden Error from lambda function invocation

wsmartinpreddio commented 9 months ago

Checkboxes for prior research

[X] I've gone through Developer Guide and API reference
[X] I've checked AWS Forums and StackOverflow.
[X] I've searched for previous similar issues and didn't find any solution.

Describe the bug

I have a lambda function which publishes to a number of topics from a lambda function. We have encountered, on some rare occasions, a ForbiddenException being thrown when attempting to publish. 99 times out of 100 the publish command succeeds, the other 1% we encounter the below error:

{
    "errorType": "ForbiddenException",
    "errorMessage": "Forbidden",
    "name": "ForbiddenException",
    "$fault": "client",
    "$metadata": {
        "httpStatusCode": 403,
        "requestId": "3a3ed9d1-ab7e-7871-df31-9353dbe3ae6e",
        "attempts": 1,
        "totalRetryDelay": 0
    },
    "message": "Forbidden",
    "traceId": "3a3ed9d1-ab7e-7871-df31-9353dbe3ae6e",
    "stack": [
        "ForbiddenException: Forbidden",
        "    at throwDefaultError (/var/runtime/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:8:22)",
        "    at /var/runtime/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:18:39",
        "    at de_PublishCommandError (/var/runtime/node_modules/@aws-sdk/client-iot-data-plane/dist-cjs/protocols/Aws_restJson1.js:472:20)",
        "    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)",
        "    at async /var/runtime/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24",
        "    at async /var/runtime/node_modules/@aws-sdk/middleware-signing/dist-cjs/awsAuthMiddleware.js:14:20",
        "    at async /var/runtime/node_modules/@aws-sdk/middleware-retry/dist-cjs/retryMiddleware.js:27:46",
        "    at async /var/runtime/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26",
        "    at async Activity.edit (file:///var/task/edit_activity.mjs:5624:5)",
        "    at async Runtime.handler (file:///var/task/edit_activity.mjs:6414:7)"
    ]
}

We invoke this lambda via an http api gateway integration and provide permissions via an IAM role.

{
      "document": {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Action": [
              "iot:Connect",
              "iot:Publish",
              "iot:Receive",
              "iot:RetainPublish"
            ],
            "Resource": "arn:aws:iot:us-east-1:{account}:*",
            "Effect": "Allow"
          }
        ]
      },
      "name": "IoTPublishPolicy",
      "type": "inline"
    },

The iot-data plane client is instantiated outside the handler.

import { IoTDataPlaneClient } from "@aws-sdk/client-iot-data-plane";

const iotDataClient = new IoTDataPlaneClient({
  region: "us-east-1",
});

SDK version number

@aws-sdk/package-name@version, ...

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

node 18

Reproduction Steps

Lambda function running node 18, invoked from aws http api gateway v2 endpoint with lambda integration, instantiate iot data plane client outside handler function and use to send publish command(s) within handler.

Observed Behavior

occasional ForbiddenException thrown by iot data plane client

Expected Behavior

publish command succeeds without error

Possible Solution

No response

Additional Information/Context

No response

aBurmeseDev commented 9 months ago

Hi @wsmartinpreddio - thanks for reaching out and sorry for the delay as issue reproduction took longer than expected.

I attempted to reproduce it multiple times but was unable to. To narrow down the root causes, can you verify the followings:

Which SDK version are you running?
Does the exception occurs intermittently?
What does your IAM permission of your lambda look like?
For your IoTPublishPolicy, I would suggest using policy that grants permission to connect to IoT with a client ID. Refers to policy example here

Also here's similar issue with different SDK reported on AWS community repost which you might find it useful.

If the issue persists, please share those info above and I'd be happy to investigate further. Best, John

wsmartinpreddio commented 9 months ago

Hi John,

Our lambdas are using the node docker image public.ecr.aws/lambda/nodejs:18-x86_64 and I'm using --external in my Dockerfile esbuild commands for the aws sdk to utilize the pre-bundled aws sdk.

That being said my package-lock.json has the @aws-sdk/client-iot-data-plane version at 3.451.0

Yes the exception does occur intermittently, we've only encountered it a handful of times and also had limited success being able to reproduce the error.

The iam policy is using 'iot' not 'iot-data' (see above for policy already provided) like in the re:post link you provided.

I provide permission for my lambda to connect using any client id, would there be any reason why making the iam policy more restrictive to a specific client id would be better than the already very broad permissions I grant the lambda?

Best,

Wade

aws / aws-sdk-js-v3