Message handler does not seem to be working

[pesseyjulien]

Hi,

Thanks for this package and the brefphp bundle in general, it's magical !

I do have an issue to make the consumer work on my lambda though.

My setup works fine when using the doctrine queue, but when I switch in order to use the SQS queue, the message is sent to the SQS and then the lambda seems to 'consume' the message, based on the logs, but my process (sending a message) is not happening.

I tried debugging my MessageHandler using var_dump and the Psr\Logger but nothing is being logged.

All I know is that the lambda run for about ~6s each time but nothing more :/

I checked the lambda timeout (15min), vpc, env varibales and the sqs config but I can't seem to find why it's not working.

An interesting fact is that one of my other message type is working when using the same sqs + lambda.

Any idea what I could be missing ?

Thanks in advance for your help, Julien Pessey

P.S : I did not know what config file to include (I basically followed the doc) so please feel free to ask me for any !

[t-richard]

My setup works fine when using the doctrine queue

I'm not sure what you did for that. How did you manage to do that and does that differ from what you're trying to do with SQS?

Could you maybe share your serverless.yml?

[pesseyjulien]

Hi,

What I was trying to say is that my messageHandler worked fine before migrating to the sqs/lambda systeme. Meaning it works if I use the 'default::doctrine' DSN and the messenger:consumer command. But If I setup the SQS queue instead, the lambda gets the message but nothing happens. Each time it runs for about 6s and thats it.

Here is my serverless.yml

# Read the documentation at https://www.serverless.com/framework/docs/providers/aws/guide/serverless.yml/
service: symfony

custom:
    secrets: ${file(secrets.js):secrets}
    db: ${file(secrets.js):db}

provider:
    name: aws
    region: eu-west-3
    stage: prod
    runtime: provided.al2
    environment:
        # Symfony environment variables
        APP_ENV: prod
        APP_SECRET: xxx
        IS_PROD: true
        IS_ADMIN: true
        SENTRY_DSN: xx
        elastic_transport: https
        elastic_port: 443
        elastic_host: xxx
        REDIS_URL: xx
        DATABASE_DRIVER: pdo_pgsql
        DATABASE_URL: ${self:custom.db}
        website_url: ${self:custom.secrets.website_url}
        email_contact: ${self:custom.secrets.email_contact}
        CLOUDINARY_URL: ${self:custom.secrets.CLOUDINARY_URL}
        CLOUDINARY_SECRET: ${self:custom.secrets.CLOUDINARY_SECRET}
        SENDINBLUE_KEY: ${self:custom.secrets.SENDINBLUE_KEY}
        SENDINBLUE_WORKER_LIST: ${self:custom.secrets.SENDINBLUE_WORKER_LIST}
        SENDINBLUE_PRO_LIST: ${self:custom.secrets.SENDINBLUE_PRO_LIST}
        stripe_secret: ${self:custom.secrets.stripe_secret}
        ONESIGNAL_APP_ID: ${self:custom.secrets.ONESIGNAL_APP_ID}
        ONESIGNAL_API_KEY: ${self:custom.secrets.ONESIGNAL_API_KEY}
        HUBSPOT_API_KEY: ${self:custom.secrets.HUBSPOT_API_KEY}
        GETSTREAM_KEY: ${self:custom.secrets.GETSTREAM_KEY}
        GETSTREAM_SECRET: ${self:custom.secrets.GETSTREAM_SECRET}
        OFFER_ROOM: ${self:custom.secrets.OFFER_ROOM}
        WKHTMLTOIMAGE_PATH: ${self:custom.secrets.WKHTMLTOIMAGE_PATH}
        WKHTMLTOPDF_PATH: ${self:custom.secrets.WKHTMLTOPDF_PATH}
        MESSENGER_TRANSPORT_DSN: ${self:custom.secrets.MESSENGER_TRANSPORT_DSN}
        MESSENGER_TRANSPORT_DSN_SQS: ${self:custom.secrets.MESSENGER_TRANSPORT_DSN_SQS}

plugins:
  - ./vendor/bref/bref
  - ./vendor/bref/extra-php-extensions

functions:
    # This function let us run console commands in Lambda
    console:
        handler: bin/console
        role: arn:aws:iam::xx:role/lambda
        vpc:
            securityGroupIds:
              - sg-xx
            subnetIds:
              - subnet-xx
              - subnet-xx
        timeout: 900 # in seconds
        reservedConcurrency: 5 # max. processed in parallel
        layers:
            - ${bref:layer.php-80}
            - ${bref-extra:redis-php-80}
            - ${bref:layer.console}
        events:
         ..
    worker:
      handler: bin/consumer.php
      role: arn:aws:iam::xx:role/lambda
      vpc:
        securityGroupIds:
          - sg-xx
        subnetIds:
          - subnet-xx
          - subnet-xx
      timeout: 900 # in seconds
      reservedConcurrency: 500 # max. messages processed in parallel
      layers:
        - ${bref:layer.php-80}
        - ${bref-extra:redis-php-80}
        - ${bref:layer.console}
      events:
        - sqs:
            arn: arn:aws:sqs:eu-west-3:xx:symfony-prod-newOffer
            batchSize: 1

package:
    excludeDevDependencies: true
    patterns:
        # Excluded files and folders for deployment
        - '!assets/**'
        - '!node_modules/**'
        - '!public/build/**'
        - '!tests/**'
        - '!var/**'
        - '!docker/**'
        - '!.serverless/**'
        # If you want to include files and folders that are part of excluded folders,
        # add them at the end
        - 'var/cache/prod/**'
        - 'public/build/entrypoints.json'
        - 'public/build/manifest.json'

Thanks, Julien

[t-richard]

Your functions are running inside a VPC. Do you have a Nat gateway configured? Otherwise outbound trafic to the internet will not work and could lead to issues like yours.

[pesseyjulien]

Yes I do, my other lambda 'console' works fine and uses the same network config. I'm currently using it to consume messages from doctrine as a temporary solution.

[t-richard]

I see your worker has the ${bref:layer.console}

It should not be the case, maybe that's what is causing your issue?

[pesseyjulien]

I will try without it and let you know, thanks !

[pesseyjulien]

Closing as this was definitely a networking issue :) Sorry !

[t-richard]

Nice you figured it out. Feel free to post some more details that might be helpful for other people to troubleshoot similar issues.

Thanks!