Serverless Webhook Client

Why the topic of webhooks?

I will always find webhooks interesting. It's a meaty problem-space and there's a lot to it.

You build a mailbox (or a few) to receive messages, and you'll have to think about:

• Persistence:
  • How can I ensure that webhooks are not lost?
  • If they do get lost, how can we recieve another copy?
• Security:
  • How do I know who sent this?
  • How do I know that this has not been tampered with or altered in any way?
• Failures:
  • How do I handle processing failures?
  • How can I retry or resume processing?
• Duplication:
  • What if I receive more than one copy of a webhook event?
  • At the same time?
  • What about at different times?
• Potential Race Conditions:
  • Does the order of webhooks matter?

Built With

Langauges, Frameworks and Runtimes

• Node

• TypeScript

• SST - framework used to scaffold resources on AWS via infrastructure as code

• ElectroDB - Single-Table design oriented framework for DynamoDB (WIP)

• Golang for our send-webhook lambda function which is used to demo this application.

Infrastructure

• API Gateway
• Lambda - Serverless compute (functions as a service)
• DynamoDB - NoSQL Database
• DynamoDB streams - Publishing events such as a Record Inserted event. We will use this to trigger the initial run.
• SQS - queues containing webhooks that have not been processed successfully. We will use this to trigger re-runs.

Architecture

API

SQS Queues

Lambda Functions

These three lambdas build out the core functionality of the project.

Other Lambda Functions

These lambdas will are used to demo our features.

DynamoDB

We will be using a Single-Table Design to store our webhooks data. Not too familar with this concept but this is a great opportunity to give this a go.

In a real world situation such as an eCommerce store, I would group webhooks and related data as their own collection of entities. And things like Customers, Orders, maybe Products in another multi-entity 'Single-Table'.

Also, I like the thesis of "Storage is cheap, compute isn't". I think this has some merit.

Entities

Example Entities

const webhook = {
  PK: 'WH#57f08d25-a733-453a-be49-8caf04df5169',
  SK: 'WEBHOOK',
  id: '57f08d25-a733-453a-be49-8caf04df5169',
  origin: 'stripe',
  type: 'payment_intent.succeeded',
  created: '2024-05-19T05:27:28+00:00',
  payload: 'json payload',
}

const webhookStatus = {
  PK: 'WH#57f08d25-a733-453a-be49-8caf04df5169',
  SK: 'STATUS',
  id: '57f08d25-a733-453a-be49-8caf04df5169'
  status: 'completed',
  retries: 1,
}

Note: schema is subject to change.

Access Patterns