Email Templates

[rallisf1]

Is your feature request related to a problem? Please describe. I use emailengine to send the transactional emails of a Jamstack project. It seems like a huge waste of resources sending practically the same huge request (due to html code) over and over when only a handful of fields change e.g. his name, email and his appointment info (it's a booking app).

Describe the solution you'd like I'd like to be able to create email templates inside each account (standard HTML is fine, MJML is better, grapejs integration will be super for less techies I guess). In the template we can use variables like {name}, {date} etc which will be populated in the request, for example in /v1/account/{account}/submit instead of subject, text and html we can have:

{
    "template_id": "d41f0423195f271f",
    "template_vars": [
        "name": "John",
        "date": "08/06/2022",
        "table": "<table><thead><tr><th>Product</th><th>Quantity</th><th>Price</th></tr></thead><tbody><tr><td>Sample Product</td><td>1</td><td>$28.00</td></tr><tr><td>Sample Product #2</td><td>2</td><td>$12.00</td></tr></tbody><tfoot><tr><td colspan="2">Total:</td><td>$52.00</td></tr></tfoot>"
    ]
}

[andris9]

Templating is in the backlog, but it does not have a high priority for now. I've done something like this in the past with Nodemailer, where templating support was built-in for a while. The issue was that everyone had different expectations on how it should look, which syntax to use etc., and it got really bloated. In the end, I removed that feature completely from Nodemailer's core because there are so many alternatives to rendering a template into an HTML code, while there are not so many good SMTP protocol clients. It made more sense to focus on improving the SMTP side of the library, not template rendering.

Obviously, if there's something that many customers ask for, then I can always reprioritize backlog features. I'm thinking of adding some kind of a voting system to the PostalSys webpage so that it would be possible to nudge some of these topics.

[rallisf1]

The issue was that everyone had different expectations on how it should look, which syntax to use etc., and it got really bloated.

I can only imagine...

It made more sense to focus on improving the SMTP side of the library, not template rendering.

Yup, that's what any wise dev would do.

I'm thinking of adding some kind of a voting system to the PostalSys webpage so that it would be possible to nudge some of these topics.

Cool, don't forget to drop a link.

---

Tbh I'll try to implement this myself and make a pull request when done. Apart from the efficiency issue, for my use-case it is a security issue as well. Sending raw emails from the front-end is like a phising attacker's heaven.

[tomcon]

@andris9 interesting to hear you say it got really bloated, nodemailer that is. Maybe there's a little confusion about what's being requested

AFAICT the only decision to be made is the left and right tag symbol (eg {{ and }} or something else (and even this can be a simple config option as I've seen used elsewhere).

Apart from that, if I take how Mandrill handles this type of functionality: Post to the API with:

1. A list of recipients
2. A html template with embedded tags
3. An array of tag data for each recipient that matches the tags in the html template

And that's it. EE would loop the recipients, merge associated tag data and send. Optionally allow tags in the subject line (mainly for personalisation and better at getting past spam checkers) - even better but not initially required

= poor man's Sendgrid but very useful and would be of interest to many I'm sure

[andris9]

@tomcon, well, your comment is a good example of how different people want different things under the same name. OP requested stored email templates (you send an email in a while, but instead of providing the HTML code, you provide a template ID that gets resolved into HTML on the server-side) while your use case is bulk mail sending where you provide a template HTML and a list of recipients that would get the same email. Both cases are valid and probably will get added to EmailEngine sometime in the future but for now this is not a priority.

[tomcon]

Understood, you're quite right that the OP requested the more complicated case.

However, the simple case is the most needed. Sendgrid and Mandrill/etc just receive a html template with tags. They may offer specifying a particular template but it's nonsense as the caller knows ( and therefore has to specify the html template themselves, so can do so before the final EE call).

Why should you/EE do it rather than the way it is done now? Well, for me at least, you're an Email Engine so that's the job of the engine in my book, but also, otherwise you have 10s, 100s, potentially 1000s of developers writing almost identical code to send these emails in a loop

Please, just support the simple case (ie forget your nodemailer experience as then you were maybe taking the pov of multiple developers)

Just offer (no templates) but basically "mail merge" of a supplied html file with tags and recipients and then you can, quite rightly, announce you have personalized list/bulk/etc emailing support. It's a biggee for sure @andris9 and I'm sure you'll generate a lot of new traffic with just that "simple" feature

[andris9]

@tomcon Yeah, I agree. In fact, I did not want to include any sending at all at first but this was something users actually demanded, so I've been slowly improving these sending features.

Regarding bulk mail, this is slightly complicated. As EmailEngine is supposed to send emails using actual email accounts then the regular SMTP servers are often tuned to block bulk mailing. It seems rather suspicious if an email account, that usually sends out a few emails an hour suddenly tries to send a thousand emails. This might get the account blocked under the suspicion of being hacked and might now be operated by some spammers. Transactional email works better with such accounts.

[tomcon]

👍 I understand this acutely and very much in fact and would always recommend an email platform like Sendgrid in those situations.

Plus anyone using EE in this way and if they did something so stupid as abusing their SMTP account then they deserve what they get!

For a legitimate use case of 1-500 (say) - personalised, not bcc emails - then EE would provide an important function. Even then, if these were in batches, with high opted-in quality of data, it could save a lot of money plus your reporting and other diagnostic sending tools become even more invaluable.

I could be wrong, but adding a single extra sending API call where you are allowed to

1. specify a html template with tags,
2. a list of recipients
3. a list of recipient tag data Is all that's needed and immediately adds yet another strong feature to EE

On Fri, 27 May 2022, 23:00 Andris Reinman, @.***> wrote:

@tomcon https://github.com/tomcon Yeah, I agree. In fact, I did not want to include any sending at all at first but this was something users actually demanded, so I've been slowly improving these sending features.

Regarding bulk mail, this is slightly complicated. As EmailEngine is supposed to send emails using actual email accounts then the regular SMTP servers are often tuned to block bulk mailing. It seems rather suspicious if an email account, that usually sends out a few emails an hour suddenly tries to send a thousand emails. This might get the account blocked under the suspicion of being hacked and might now be operated by some spammers. Transactional email works better with such accounts.

— Reply to this email directly, view it on GitHub https://github.com/postalsys/emailengine/issues/135#issuecomment-1140068140, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAE572K3JIAQRD2M24TTVX3VMFAZVANCNFSM5W2P5TAQ . You are receiving this because you were mentioned.Message ID: @.***>

[rallisf1]

@andris9 I see we've been working in parallel, although I hit a wall as I haven't used redis as a db before, only as a key-value cache. Perhaps what I've done can be of help though.

You can look at my progress here

As I want each account to have its own templates I put everything inside accounts/account, although the code should probably be moved to its own route & model.

Routes, views, api schema and tag replacing are complete. The template redis calls inside lib\account.js though are all messed up, sorry.

[andris9]

@rallisf1 Oh, dang, that's quite a lot of parallel work indeed. There was so much talk about templating that I decided to tackle it right away and moved up the priority. Tag replacing etc can already be tested. Storing templates is halfway done. Though my idea was to use a global registry, not an account-specific registry of stored templates. I can revisit that thought though.

Here's a copy-paste of my comments from Discord about using the templating system that can already be tested:

You can test the new features out with Docker.

$ docker pull andris9/emailengine:latest
$ docker run -p 3000:3000 --env EENGINE_REDIS="redis://host.docker.internal:6379/9" andris9/emailengine:latest

Usage You can use templates to render subject/html/text fields, and you can specify bulk targets so that a separate email would be generated for each such target. And you can obviously mix these features by providing recipient-specific template variables.

You can turn a normal subject/text/html field into a rendered template field by providing an object instead of string value. The format is following: {format, template} where the template is a string template (uses SendGrid flavored handlebars) format is an optional format specifier ('html', 'markdown', 'mjml' for the html field, and 'text' for he text and subject fields). Template variables can be passed with a render:{values:{}} object.

Example Use templating for the subject field, and a markdown template for the HTML field:

curl -XPOST "http://127.0.0.1:3000/v1/account/example/submit" \
    -H "Content-type: application/json" \
    -d '{
      "from": {
        "name": "Andris Reinman",
        "address": "andris@example.com"
      },
      "to": [
        {
          "name": "Ethereal",
          "address": "andris@ethereal.email"
        }
      ],
      "subject": {
        "template": "Hello {{your_name}}"
      },
      "html": {
        "format": "markdown",
        "template": "**Hello from {{my_name}}!**"
      },
      "render": {
        "values": {
          "your_name": "world",
          "my_name": "andris"
        }
      }
    }'

For bulk messages you'd have to provide a bulk array instead of to, cc and bcc fields like this:

curl -XPOST "http://127.0.0.1:3000/v1/account/example/submit" \
    -H "Content-type: application/json" \
    -d '{
  "from": {
    "name": "Andris Reinman",
    "address": "andris@example.com"
  },
  "subject": {
    "template": "Hello {{insert name \"default=Customer\"}}! Thank you for contacting us about {{insert businessName \"your business\"}}"
  },
  "text": {
    "template": "Hello {{insert name \"default=Customer\"}}! Thank you for contacting us about {{insert businessName \"your business\"}}"
  },
  "html": {
    "template": "<h1>Hello {{insert name \"default=Customer\"}}! Thank you for contacting us about {{insert businessName \"your business\"}}</h1>"
  },
  "bulk": [
    {
      "to": {
        "name": "Andris 1",
        "address": "andris@ethereal.email"
      },
      "render": {
        "values": {
          "name": "Andris 1",
          "businessName": "Company 1"
        }
      }
    },
    {
      "to": {
        "name": "Andris 2",
        "address": "andris@ethereal.email"
      },
      "render": {
        "values": {
          "name": "Andris 2",
          "businessName": "Company 2"
        }
      }
    }
  ]
}'

Here's the templating syntax (SendGrid flavored Handlebars): https://docs.sendgrid.com/for-developers/sending-email/using-handlebars#formatdate

btw, here's an example for using a MJML template for html content:

curl -XPOST "http://127.0.0.1:3000/v1/account/example/submit" \
    -H "Content-type: application/json" \
    -d '{
  "from": {
    "name": "Andris Reinman",
    "address": "andris@example.com"
  },
  "subject": {
    "template": "Hello {{insert name \"default=Customer\"}}!"
  },
  "text": {
    "template": "Hello {{insert name \"default=Customer\"}}! Thank you for contacting us about {{insert businessName \"your business\"}}"
  },
  "html": {
    "format": "mjml",
    "template": "<mjml>\n  <mj-body>\n    <mj-section mj-class=\"section-white\" padding-top=\"30px\">\n      <mj-column>\n        <mj-text>\n          <h1>Hello {{insert name \"default=Customer\"}}! Thank you for contacting us about {{insert businessName \"your business\"}}</h1>\n        </mj-text>\n      </mj-column>\n    </mj-section>\n  </mj-body>\n</mjml>"
  },
  "bulk": [
    {
      "to": {
        "name": "Andris 1",
        "address": "andris@ethereal.email"
      },
      "render": {
        "values": {
          "name": "Andris 1",
          "businessName": "Company 1"
        }
      }
    }
  ]
}'

[andris9]

Btw one thing I'll change is variable scoping. Currently if you provide a value name like this:

render: {
    values: {
        name: 'Test'
    }
}

then you can use it directly Hello, {{name}}!.

I will add scoping, so the same thing would become Hello, {{values.name}}!. This is because I want to add some extra variables, eg {{account.email}} or {{service.url}} and I don't want to mix these scopes.

[rallisf1]

Cool, still this doesn't decouple the submit request from the email content which is my intention for both speed and security reasons. In my usecase the templates are saved inside emailengine and I only supply the dynamic values through the API.

[andris9]

That part is halfway done. EmailEngine will start storing templates, so that instead of providing {html, text, subject}, you can provide {template: "template_id"}.

[rallisf1]

That part is halfway done. EmailEngine will start storing templates, so that instead of providing {html, text, subject}, you can provide {template: "template_id"}.

Wow, great! Feel free to use my code if it helps.

[andris9]

Templating support is in master. This includes all the API endpoints and functional templating (you can send out emails with templates). There is no web UI at this point, so you have to edit templates via the API.

[andris9]

Here's an example bash script that uses the API endpoints with different sending options: https://gist.githubusercontent.com/andris9/2472435255db275249a51d90dc291682/raw/5ecf97c0a153887806d36db6b02b2555d43f2c3e/test.sh