unprocessed ses webhooks

[fibis]

Describe your issue

We are using mailcoach with ses. THe debug page is showing that we have unprocessed web hooks that are SES web hooks. The mailcoach-feedback queue is running., so these web hooks should be proceeded. What could be the problem?

---

Health check:

Environment: production Debug: OFF Horizon: Active Queue connection: OK Webhooks: 5 unprocessed webhooks Schedule: ran 0 minute(s) ago Default mailer: mailcoach Mailcoach mailer: null Campaign mailer: null Transactional mailer: null

Technical details

App directory: /var/www User agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:83.0) Gecko/20100101 Firefox/83.0 PHP version: 7.4.12 MySQL version: 8.0.18-google Laravel version: 8.17.0 Horizon version: v5.6.0@4e21779357d1f1abc0282a940855be635bf4f1be laravel-mailcoach version: 3.5.0 mailcoach-ui version: 1.1.0

[riasvdv]

Could you share your horizon config here and a payload of one of the unprocessed webhooks? (anonymize the data first)

[fibis]

Sure, actually it is the default config after installing mailcoach via composer:

horizon.php

<?php

return [

    /*
    |--------------------------------------------------------------------------
    | Horizon Redis Connection
    |--------------------------------------------------------------------------
    |
    | This is the name of the Redis connection where Horizon will store the
    | meta information required for it to function. It includes the list
    | of supervisors, failed jobs, job metrics, and other information.
    |
    */

    'use' => 'default',

    /*
    |--------------------------------------------------------------------------
    | Horizon Redis Prefix
    |--------------------------------------------------------------------------
    |
    | This prefix will be used when storing all Horizon data in Redis. You
    | may modify the prefix when you are running multiple installations
    | of Horizon on the same server so that they don't have problems.
    |
    */

    'prefix' => env('HORIZON_PREFIX', 'horizon:'),

    /*
    |--------------------------------------------------------------------------
    | Queue Wait Time Thresholds
    |--------------------------------------------------------------------------
    |
    | This option allows you to configure when the LongWaitDetected event
    | will be fired. Every connection / queue combination may have its
    | own, unique threshold (in seconds) before this event is fired.
    |
    */

    'waits' => [
        'redis:default' => 60,
    ],

    /*
    |--------------------------------------------------------------------------
    | Job Trimming Times
    |--------------------------------------------------------------------------
    |
    | Here you can configure for how long (in minutes) you desire Horizon to
    | persist the recent and failed jobs. Typically, recent jobs are kept
    | for one hour while all failed jobs are stored for an entire week.
    |
    */

    'trim' => [
        'recent' => 60,
        'failed' => 10080,
    ],

    /*
    |--------------------------------------------------------------------------
    | Queue Worker Configuration
    |--------------------------------------------------------------------------
    |
    | Here you may define the queue worker settings used by your application
    | in all environments. These supervisors and settings handle all your
    | queued jobs and will be provisioned by Horizon during deployment.
    |
    */

    'environments' => [
        'production' => [
            'mailcoach-general' => [
                'connection' => 'mailcoach-redis',
                'queue' => ['default', 'mailcoach', 'mailcoach-feedback', 'send-mail'],
                'balance' => 'auto',
                'processes' => 10,
                'tries' => 2,
                'timeout' => 60 * 60,
            ],
            'mailcoach-heavy' => [
                'connection' => 'mailcoach-redis',
                'queue' => ['send-campaign'],
                'balance' => 'auto',
                'processes' => 3,
                'tries' => 1,
                'timeout' => 60 * 60,
            ],
        ],

        'local' => [
            'mailcoach-general' => [
                'connection' => 'mailcoach-redis',
                'queue' => ['default', 'mailcoach', 'mailcoach-feedback', 'send-mail'],
                'balance' => 'auto',
                'processes' => 10,
                'tries' => 2,
                'timeout' => 60 * 60,
            ],
            'mailcoach-heavy' => [
                'connection' => 'mailcoach-redis',
                'queue' => ['send-campaign'],
                'balance' => 'auto',
                'processes' => 3,
                'tries' => 1,
                'timeout' => 60 * 60,
            ],
        ],
    ],
];

We are starting horizon in our docker container in our cluster with php artisan horizon.

This is the payload of one of the unprocessed webworks:

{
    "Type": "Notification",
    "Message": "{\"notificationType\":\"Delivery\",\"mail\":{\"timestamp\":\"2020-12-04T08:06:09.664Z\",\"source\":\"news@example.com\",\"sourceArn\":\"arn:aws:ses:eu-central-1:155207742949:identity/news@example.com\",\"sourceIp\":\"***\",\"sendingAccountId\":\"***\",\"messageId\":\"010701762cc97800-a64babda-db6d-41d0-b5cb-506863fc59b9-000000\",\"destination\":[\"other@example.com\"]},\"delivery\":{\"timestamp\":\"2020-12-04T08:06:10.570Z\",\"processingTimeMillis\":906,\"recipients\":[\"other@example.com\"],\"smtpResponse\":\"250 2.6.0 <010701762cc97800-a64babda-db6d-41d0-b5cb-506863fc59b9-000000@eu-central-1.amazonses.com> [InternalId=46574625359717, Hostname=FRYP281MB0079.DEUP281.PROD.OUTLOOK.COM] 103922 bytes in 0.059, 1708.257 KB/sec Queued mail for delivery\",\"remoteMtaIp\":\"104.47.7.138\",\"reportingMTA\":\"b226-226.smtp-out.eu-central-1.amazonses.com\"}}",
    "TopicArn": "arn:aws:sns:eu-central-1:155207742949:**",
    "MessageId": "e81e8855-48eb-5f5d-9d65-d016b8d9c959",
    "Signature": "oD6Ie66PZFeRAyGtVJD96bjJ93J/s4upcYaX+WHZVFI2YYDEf1U/8t/xoAthttA+3+in1yzK1uGy+GzgpL4R+Oos1NZgmKAw4LXmgBLVS1X/H7ZN1wAT5+oPwLfKNFqTfVJ3OetS6nHzthWHFcYiH94FZFQxr2C0uCM+lwl+963awsxdHxtaaGyYXsW2gIahSwpKGFf8NIJqXZYBNJz1fjk7QUaLthTvGWrZ+/l+wd38IywLuV/Gc2apDf8Qxqc8f6S8KcZH1Ba8j2xt3g3dPJjcWs7JtMDHb3OhLcXvPDnyhePpLUDUSxwA2ckYNM5lIMxFolZEZ64i2fi8ZltnZw==",
    "Timestamp": "2020-12-04T08:06:10.615Z",
    "SigningCertURL": "https://sns.eu-central-1.amazonaws.com/SimpleNotificationService-010a507c1833636cd94bdb98bd93083a.pem",
    "UnsubscribeURL": "https://sns.eu-central-1.amazonaws.com/**",
    "SignatureVersion": "1"
}

[riasvdv]

Are you sending emails with that SES & SNS account from somewhere other than that Mailcoach instance?

[fibis]

I found the mistake: Our deployment template uses the old variable name for the queue connection, so the queue driver was sync and no worker has processed the jobs.

[riasvdv]

Glad you found the issue!

[fibis]

So I think I can delete the entries in the web hook table or is there a possibility to process the unprocessed web hooks?

[riasvdv]

There is no way to easily retry processing them, so you can delete them.