Mailer not sending transactional emails using AWS SES credentials

[alifeinbinary]

Version

5.37.2

Operating System

MacOS Monterey 12.6.8

Browser

116.0.3

What are the steps to reproduce this bug?

Assuming you have a verified domain and identity in AWS SES console and satisfied the DKIM, SPF, DMARC compliance within your DNS settings for your verified domain. In AWS SES console, generate SMTP credentials and paste them into the USER and PASSWORD variables as I have done below. I used Thunderbird app to test email sending and confirmed that I am able to send using the SMTP credentials provided by Amazon. If you do the same, be sure to use STARTTLS on port 587 and Plain text authentication using username and password.

Configure your .env file like this, make sure to use port 587 for STARTTLS to work.

# Mailer configuration
WEBINY_MAILER_PASSWORD_SECRET=g8fj4a5485dfGkX12bcca898ea31faa9f2
# required
WEBINY_MAILER_HOST=email-smtp.ca-central-1.amazonaws.com
WEBINY_MAILER_PORT=587
WEBINY_MAILER_USER=AKIAGHFKDOSJSHIXXB
WEBINY_MAILER_PASSWORD=BNAop1UdDvB4MM5EypIzqrBnDznXX/eKMLDNLghWVeV0
WEBINY_MAILER_REPLY_TO=name@domain.com
WEBINY_MAILER_FROM=name@domain.com

In apps/api/graphql/src/index.ts add the following transport to enable logging and debugging in CloudWatch

const transport = createTransport(async ({ settings }) => {
    console.log("settings", settings);
    return createSmtpTransport({
        ...settings,
        from: "name@domain.com",
        envelope: {
            from: "Company <name@domain.com>"
        },
        host: "email-smtp.ca-central-1.amazonaws.com",
        port: 587,
        secure: false,
        auth: {
            user: process.env["WEBINY_MAILER_USER"],
            pass: process.env["WEBINY_MAILER_PASSWORD"]
        },
        authMethod: "PLAIN",
        requireTLS: true,
        name: "domain.com",
        logger: true,
        debug: true
    });
});

Make sure to add transport to the plugins array.

Create a basic contact form within FormBuilder with some basic fields like; first name, last name, email, message, and within the triggers tab add your email to "Email - Submission Notification" and some text to the Subject and Email content fields for the "Email - Thank You Email" trigger. Save and publish the form. Create a /contact page to place the form into it within PageBuilder and publish the page. Navigate to localhost:3000/contact and you should see the contact form.

CloudWatch In AWS CloudWatch console select Live Tail from the menu on the left side. Select the graphql Lambda function linked to your development environment and click filter. If CloudWatch hasn't started the Live tailing process automatically then click "Start".

Return to your contact form, fill in the fields and make sure to include and email that you have access to, and submit.

What is the expected behavior?

It's expected that two emails should be sent; one submission notification and another thank you email and received at the address provided in the "Email - Submission Notification" form trigger and the address that was included in the form submission.

What do you see instead?

While the website app will report that the message was successfully sent, this won't be the case. Looking at the CloudWatch logs we can see where it failed.

2023-08-22T20:58:04.045Z f56ffae3-7f6b-48e3-957c-15033184a759 INFO [2023-08-22 20:58:04] DEBUG [LZyxdJeuXLE] S: 250-AUTH PLAIN LOGIN

2023-08-22T20:58:04.045Z f56ffae3-7f6b-48e3-957c-15033184a759 INFO [2023-08-22 20:58:04] DEBUG [LZyxdJeuXLE] S: 250 Ok

2023-08-22T20:58:04.046Z f56ffae3-7f6b-48e3-957c-15033184a759 INFO [2023-08-22 20:58:04] DEBUG [LZyxdJeuXLE] SMTP handshake finished

2023-08-22T20:58:04.046Z f56ffae3-7f6b-48e3-957c-15033184a759 INFO [2023-08-22 20:58:04] DEBUG [LZyxdJeuXLE] C: AUTH PLAIN AEFLSUFYVlJVTlRKTUVDQjJJMjVNAC5qIHNlY3JldCAqGw==

2023-08-22T20:58:04.057Z f56ffae3-7f6b-48e3-957c-15033184a759 INFO [2023-08-22 20:58:04] DEBUG [LZyxdJeuXLE] S: 235 Authentication successful.

2023-08-22T20:58:04.057Z f56ffae3-7f6b-48e3-957c-15033184a759 INFO [2023-08-22 20:58:04] INFO [LZyxdJeuXLE] User "AKIAGHFKDOSJSHIXXB" authenticated

2023-08-22T20:58:04.057Z f56ffae3-7f6b-48e3-957c-15033184a759 INFO [2023-08-22 20:58:04] INFO Sending message <b36bb36c-be99-3ad4-3e06-6c724007b843@localhost> to <recipient@server.com>

2023-08-22T20:58:04.058Z f56ffae3-7f6b-48e3-957c-15033184a759 INFO [2023-08-22 20:58:04] DEBUG [LZyxdJeuXLE] C: MAIL FROM:<>

2023-08-22T20:58:04.060Z f56ffae3-7f6b-48e3-957c-15033184a759 INFO [2023-08-22 20:58:04] DEBUG [LZyxdJeuXLE] S: 501 Invalid MAIL FROM address provided

2023-08-22T20:58:04.061Z f56ffae3-7f6b-48e3-957c-15033184a759 INFO [2023-08-22 20:58:04] DEBUG [LZyxdJeuXLE] Closing connection to the server using "end"

So, authentication is successful, however, in the third to last CloudWatch log line we can see that MAIL FROM isn't provided by the message context data.

Additional information

According to the nodemailer documentation the transport uses the sendMail function like so:

nodemailer SMTP envelope

let message = {
    ...,
    from: 'mailer@nodemailer.com', // listed in rfc822 message header
    to: 'daemon@nodemailer.com', // listed in rfc822 message header
    envelope: {
        from: 'Daemon <deamon@nodemailer.com>', // used as MAIL FROM: address for SMTP
        to: 'mailer@nodemailer.com, Mailer <mailer2@nodemailer.com>' // used as RCPT TO: address for SMTP
    }
}

transporter.sendMail(message[, callback])

The Webiny implementation of nodemailer doesn't provide acccess to the sendMail function or the message data, so debugging further is challenging without help from the Webiny team.

Possible solution

It appears that the Webiny implementation of nodemailer isn't adhering to the SMTP envelope format, linked above. My next comment links to the points of interest in the Webiny code base and possible solutions. Please advise if you would like me to submit a PR.

[alifeinbinary]

If I had to guess, packages/api-mailer/src/crud/transport/onTransportBeforeSend.ts#L13

Should be:

    .object({
        to: zod.array(requiredEmail).optional(),
        from: zod.string().email().optional()
        envelope: {
            from: zod.string().email().optional(),
            to: zod.array(requiredEmail).optional()
        },
        subject: requiredString.max(1024).min(2),
        cc: zod.array(requiredEmail).optional(),
        bcc: zod.array(requiredEmail).optional(),
        replyTo: zod.string().email().optional(),
        text: zod.string().optional(),
        html: zod.string().optional()
    })

And packages/api-mailer/src/types.ts#L146

Should be:

interface BaseTransportSendData {
    to?: string[];
    from?: string[];
    cc?: string[];
    bcc?: string[];
    envelope?: {
        from?: string[];
        to?: string[];
    };
    subject: string;
    text?: string;
    html?: string;
    replyTo?: string;
}

[blbigelow]

I am running 5.38.3 and running into the same issue.

[Styn]

Running 5.39.5 here, thanks to the very detailed description of @alifeinbinary I've been able to narrow down my problem to the exact same thing...

[brunozoric]

Hi @Styn I added a little bit more info, about the mailer, into our documentation: https://www.webiny.com/docs/overview/features/mailer#the-default-transport-is-not-working-for-me

[Styn]

Thank you @brunozoric I got it to work with your pointers. If anyone wants a starting point for for example AWS SES, I'll share my PoC code below, of course this should be adapted to be less naive and don't do the unnecessary check etc... It's more meant to help anyone who runs into this with getting started.

import nodemailer from "nodemailer";
import { createTransport, createSmtpTransport } from "@webiny/api-mailer";

const transportTest = createTransport(async ({ settings }) => {
    console.log("custom transport");
    console.log("settings", settings);
    const sender = nodemailer.createTransport({
        host: settings?.host,
        port: settings?.port,
        secure: false, // upgrade later with STARTTLS
        auth: {
            user: settings?.user,
            pass: settings?.password
        },
      });
      sender.verify(function (error, success) {
        if (error) {
          console.log(error);
        } else {
          console.log("Server is ready to take our messages");
        }
      });
    return {
        name: "webiny.mailer.sesCustomTransport",
        send: async data => {
            console.log("send data", data);
            try {
                const result = await sender.sendMail({
                    ...data,
                    from: settings?.from
                });
                console.log("sending succeeded", result);
                return {
                    result,
                    error: null,
                };
            } catch (ex) {
                console.log("sending failed", ex);
                return {
                    result: null,
                    error: ex
                };
            }
        }
    };
});

transportTest should be added to the plugins.

[alifeinbinary]

Here's a slightly more robust example with proper logic for the replyTo field depending on whether the Message is a Notification or Thank You

import nodemailer from "nodemailer";
import { createTransport } from "@webiny/api-mailer";

const transport = createTransport(async ({ settings }) => {
    console.log("SES transport");
    if (process.env["DEBUG"]) {
        console.log("settings", settings);
    }
    const name = "Webiny Admin"; // Will aim to retrieve this from Settings > Page Builder > Website > Website Name
    const sender = nodemailer.createTransport({
        host: settings?.host,
        port: settings?.port,
        secure: false, // upgrade later with STARTTLS
        auth: {
            user: settings?.user,
            pass: settings?.password
        },
        requireTLS: true,
        authMethod: "LOGIN",
        name: name
    });
    sender.verify(function (error, success) {
        if (error) {
            console.log(error);
        } else {
            console.log("Server is ready to take our messages", success);
        }
    });
    return {
        name: "webiny.mailer.awsSesTransport",
        send: async data => {
            console.log("send data", data);
            try {
                const result = await sender.sendMail({
                    ...data,
                    replyTo: data.replyTo === undefined ? settings?.replyTo : data.replyTo,
                    envelope: {
                        from: name + " <" + settings?.replyTo + ">",
                        to: data.to?.join(", ")
                    },
                    from: name + " <" + settings?.replyTo + ">"
                });
                console.log("sending succeeded", result);
                return {
                    result,
                    error: null
                };
            } catch (ex) {
                console.log("sending failed", ex);
                return {
                    result: null,
                    error: ex
                };
            }
        }
    };
});

export default transport;