AWS Lambda tests running continuously - Doesn't stop even after 15 mins

[vicks07]

After downloading the blitz.yml and making some minor modifications to the file. (File Attached at the end)

Version info:

2.0.0-23

Running this command:

artillery run --platform aws:lambda --platform-opt region=ap-south-1 --count 2 blitz.yml

With the configuration on AWS for the user as

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "CreateOrGetLambdaRole",
            "Effect": "Allow",
            "Action": [
                "iam:CreateRole",
                "iam:GetRole",
                "iam:PassRole"
            ],
            "Resource": "arn:aws:iam::xxxxxxx:role/artilleryio-default-lambda-role"
        },
        {
            "Sid": "CreateLambdaPolicy",
            "Effect": "Allow",
            "Action": [
                "iam:CreatePolicy",
                "iam:AttachRolePolicy"
            ],
            "Resource": "arn:aws:iam::xxxxxxx:policy/artilleryio-lambda-policy"
        },
        {
            "Sid": "SQSPermissions",
            "Effect": "Allow",
            "Action": [
                "sqs:*"
            ],
            "Resource": "*"
        },
        {
            "Sid": "SQSListQueues",
            "Effect": "Allow",
            "Action": [
                "sqs:ListQueues"
            ],
            "Resource": "*"
        },
        {
            "Sid": "LambdaPermissions",
            "Effect": "Allow",
            "Action": [
                "lambda:InvokeAsync",
                "lambda:CreateFunction",
                "lambda:DeleteFunction",
                "lambda:InvokeFunction"
            ],
            "Resource": "arn:aws:lambda:*:xxxxxxx:function:artilleryio-*"
        },
        {
            "Sid": "S3Permissions",
            "Effect": "Allow",
            "Action": [
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:PutObject",
                "s3:ListBucket",
                "s3:GetLifecycleConfiguration",
                "s3:PutLifecycleConfiguration"
            ],
            "Resource": [
                "arn:aws:s3:::artilleryio-test-data-*",
                "arn:aws:s3:::artilleryio-test-data-*/*"
            ]
        }
    ]
}

I expected to see this happen: Which works as expected when I run the command on my local machine.

artillery run blitz.yml

Instead, this happened:

The command keeps running even after 15 mins. Until it is manually stopped. However, the lambda function and the s3 buckets are created as expected. But there is no report that is generated at the end as the script never stops.

One of the things that I noticed was that there was no SQS queue created, not sure if that is the issue. (Shouldn't be as full access has been given)

Files being used: File format changed to .txt just to upload it here. blitz.txt

PS: Also, some of the permissions to the IAM user had to be added as they were throwing an error initially.

[hassy]

Thank you for the bug report @vicks07!

Does this happen sporadically or every time you try to run a test?

If possible, when it happens again, could you take a look at the CloudWatch logs for the lambda functions that Artillery created to confirm that they run as expected.

Since the test run got to the "Running scenarios" part I would assume the SQS queue was created successfully (or Artillery would've thrown an error before then - code)

And lastly, could you share what extra permissions needed to be added to the IAM rules?

[vicks07]

I have tried running it several times and it happened every time I ran the tests. Sure, I'll have a look at the CloudWatch logs and update you.

Regarding the additional permissions, this is what I had to change in order to get the "running scenarios" which are as mentioned below within the (double asterisks).

{
            "Sid": "CreateOrGetLambdaRole",
            "Effect": "Allow",
            "Action": [
                "iam:CreateRole",
                "iam:GetRole",
                **"iam:PassRole"**
            ],
            "Resource": "arn:aws:iam::xxxxxxx:role/artilleryio-default-lambda-role"
},
{
            "Sid": "LambdaPermissions",
            "Effect": "Allow",
            "Action": [
                "lambda:InvokeAsync",
                "lambda:CreateFunction",
                "lambda:DeleteFunction",
                **"lambda:InvokeFunction"**
            ],
            "Resource": "arn:aws:lambda:*:xxxxxxx:function:artilleryio-*"
        },

Thank you for the quick response @hassy

[vicks07]

@hassy I checked the logs on Cloudwatch and they throw an error

ERROR AccessDenied: Access to the resource https://sqs.ap-south-1.amazonaws.com/ is denied

Attaching the logs from CloudWatch. cloudwatchlog.txt

[vicks07]

@hassy On further debugging, I found that the role artilleryio-default-lambda-role did not have the permission to access the SQS queue. I added the policy manually and it seems to be working now.

But these permissions were supposed to be added on their own as I did not create the role as it was mentioned as optional in the documentation.