SQS queue : Connection setup error

[Aikoze]

Hi ! :hand:

I'm trying to use SQS queue and I keep getting the same error :

I'm using API platform based on symfony then i'm using bref configuration for symfony.

The Amazon SQS queue "heroad-sqs-dev-Queue-10KLY4X240HKO" does not exists (or you don't have permissions on it), and can't be created when an account is provided.'

To explain what i want to do :

I have my main app ' web ' wich is an API i want to call a SQS queue to do a long task because the 29 second timeout is not suffisent for my task.

What i'm expecting is when i call my API from this special endpoint my worker will do the job in background without that 29 second timeout and sending me a notification when the task is finished.

here is my simple configuration in my serverless.yml :

provider:
    name: aws
    region: eu-west-3
    runtime: provided.al2
    environment:
        APP_ENV: stagging

resources:
    Resources:
        Queue:
            Type: AWS::SQS::Queue
            Properties:
                # This needs to be at least 6 times the lambda function's timeout
                # See https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html
                VisibilityTimeout: '960'
                RedrivePolicy:
                    deadLetterTargetArn: !GetAtt DeadLetterQueue.Arn
                    # Jobs will be retried 5 times
                    # The number needs to be at least 5 per https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html
                    maxReceiveCount: 5
        # The dead letter queue is a SQS queue that receives messages that failed to be processed
        DeadLetterQueue:
            Type: AWS::SQS::Queue
            Properties:
                # Messages are stored up to 14 days (the max)
                MessageRetentionPeriod: 1209600

plugins:
    - ./vendor/bref/bref

package:
    exclude:
        - node_modules/**
        - tests/**
        - var/**
        - public/build/**'

functions:
    # This function runs the Symfony website/API
    web:
        handler: public/index.php
        timeout: 28 # in seconds (API Gateway has a timeout of 29 seconds)
        layers:
            - ${bref:layer.php-74-fpm}
        events:
            -   httpApi: '*'

    worker:
        handler: bin/consumer.php
        timeout: 20 # in seconds
        reservedConcurrency: 5 # max. 5 messages processed in parallel
        layers:
            - ${bref:layer.php-74}
        events:
            # Read more at https://www.serverless.com/framework/docs/providers/aws/events/sqs/
            - sqs:
                  arn: arn:aws:sqs:eu-west-3:508623790554:heroad-sqs-dev-Queue-10KLY4X240HKO
                  # Only 1 item at a time to simplify error handling
                  batchSize: 1

I'm using the same worker for symfony 5.1 because I'm on 5.3

My messenger.yaml :

framework:
    messenger:
        transports:
            async: '%env(MESSENGER_TRANSPORT_DSN)%'
        routing:
            # async is whatever name you gave your transport above
            'App\Message\GeneratePlanning': async

in my .env.stagging I've

MESSENGER_TRANSPORT_DSN=https://sqs.eu-west-3.amazonaws.com/508623790554/heroad-sqs-dev-Queue-10KLY4X240HKO

And finaly my bus dispatch in an API endpoint controller

    /**
     * @Route("/generate", methods={"GET"})
     * @return JsonResponse|Response
     * @throws \Exception
     */
    public function generate(MessageBusInterface $bus)
    {
        $bus->dispatch(new GeneratePlanning('hello world'));

        return new JsonResponse(['success' => 'success'], 201);

    }

First break at : vendor/symfony/amazon-sqs-messenger/Transport/Connection.php (line 276)

[t-richard]

Hi :wave:

I think your Lambda is missing permissions to retrieve details about your queue and put a message in it.

You need to specifiy iam.role.statements with the appropriate IAM policy to allow your Lambda to push to the Queue.

See https://www.serverless.com/framework/docs/providers/aws/guide/iam/

You can also use the queue construct from Lift that would handle all of this for you : creating the queue, DLQ, redrive policy, setting permissions and referencing it to your env variable.

For more details, see https://github.com/getlift/lift/blob/master/docs/queue.md

[t-richard]

With Lift your serverless.yml would look something like (may need adjustments but this gives you an idea)

provider:
    name: aws
    region: eu-west-3
    runtime: provided.al2
    environment:
        APP_ENV: stagging
        MESSENGER_TRANSPORT_DSN: ${construct:jobs.queueUrl}

plugins:
    - ./vendor/bref/bref
    - serverless-lift

package:
    exclude:
        - node_modules/**
        - tests/**
        - var/**
        - public/build/**'

functions:
    # This function runs the Symfony website/API
    web:
        handler: public/index.php
        timeout: 28 # in seconds (API Gateway has a timeout of 29 seconds)
        layers:
            - ${bref:layer.php-74-fpm}
        events:
            -   httpApi: '*'

constructs:
    jobs:
        type: queue
        worker:
            handler: bin/consumer.php
            timeout: 20 # in seconds
            reservedConcurrency: 5 # max. 5 messages processed in parallel
            layers:
                - ${bref:layer.php-74}

[Aikoze]

With Lift your serverless.yml would look something like (may need adjustments but this gives you an idea)

provider:
    name: aws
    region: eu-west-3
    runtime: provided.al2
    environment:
        APP_ENV: stagging
        MESSENGER_TRANSPORT_DSN: ${construct:jobs.queueUrl}

plugins:
    - ./vendor/bref/bref
    - serverless-lift

package:
    exclude:
        - node_modules/**
        - tests/**
        - var/**
        - public/build/**'

functions:
    # This function runs the Symfony website/API
    web:
        handler: public/index.php
        timeout: 28 # in seconds (API Gateway has a timeout of 29 seconds)
        layers:
            - ${bref:layer.php-74-fpm}
        events:
            -   httpApi: '*'

constructs:
    jobs:
        type: queue
        worker:
            handler: bin/consumer.php
            timeout: 20 # in seconds
            reservedConcurrency: 5 # max. 5 messages processed in parallel
            layers:
                - ${bref:layer.php-74}

Thank you for trying to help, but it didn't work even with the lift configuration.. keep getting the same error.

For information i'm deploying this with AdministratorAccess IAM role.

[t-richard]

Thank you for trying to help, but it didn't work even with the lift configuration.. keep getting the same error.

@Aikoze Are you getting the same error with the same queue name heroad-sqs-dev-Queue-10KLY4X240HKO or is it a new one ?

I have a similar config and it works as expected.

[Aikoze]

Thank you for trying to help, but it didn't work even with the lift configuration.. keep getting the same error.

@Aikoze Are you getting the same error with the same queue name heroad-sqs-dev-Queue-10KLY4X240HKO or is it a new one ?

I have a similar config and it works as expected.

A new one built with lift named : heroad-sqs-dev-jobs

Also getting those lines :

"aws_code": "AWS.SimpleQueueService.NonExistentQueue",
"aws_message": "The specified queue does not exist or you do not have access to it.",
"aws_type": "Sender",
"aws_detail": ""

I'm also calling the endpoint with CURL or postman maybe I'm wrong, your configuration also use API endpoint to send the mesage ?

[t-richard]

A new one built with lift named : heroad-sqs-dev-jobs

That seems to be the intended value, I just wanted to make sure the old queue wasn't still being used.

I'm also calling the endpoint with CURL or postman maybe I'm wrong, your configuration also use API endpoint to call the event ?

Not sure what your mean here. i see you have a @Route("/generate", methods={"GET"}) so curl/using postman on

GET <API gateway endpoint>/generate should send the message to the Queue (and I guess this is where you get the error).

Or did you meant something else ?

Also, the bug may be related to auto_setup being enabled on your symfony messenger config, try with

framework:
    messenger:
        transports:
            async: 
                dsn: '%env(MESSENGER_TRANSPORT_DSN)%'
                options:
                    auto_setup: false
        routing:
            # async is whatever name you gave your transport above
            'App\Message\GeneratePlanning': async

[Aikoze]

@t-richard Thanks for all, this was the missing configuration

options:
    auto_setup: false