Unified messaging

[ejsmith]

Benefits

• Job / process / container consolidation
  • We control which message types are handled by a given process
  • Single process can handle many message types
  • No need to create a new job, ECS task, container for each message type like we currently do
  • No need to create a dedicated queue resource (SQS) for every message type
  • Many types of messages handled per resource, we control which ones are in each
  • Will result in large cost savings
• Concept consolidation / consistency
  • Everything is a message
  • Worker queues, pub/sub, RPC (Request/Response)
  • Queue, work items, pub/sub all consolidated
  • Current pub/sub does not have ability to ack messages like queues do
  • Messaging doesn't support correlation for following traces like queues do
  • Queues don't support delayed delivery like pub/sub does
  • Kafka does not align very well to our current queue / messaging abstractions
  • Subscription options will allow specifying stream position and other options
• Performance
  • Receive messages in batches
  • Will allow for much higher overall system throughput
  • Message handler for each message type and messages are pushed to that handler vs pull based queues now
  • Work on multiple messages at once or one at a time
  • Batch processing for efficiency when it's likely a batch of messages will share resources
  • Handle N messages concurrently per process
  • Ability to spawn multiple processing threads per container for handling messages concurrently
  • Ability to route messages based on priority to allow high priority messages to get their own dedicated processors
• Testing
  • Easy to assert that a given test emits exactly what messages you expect it to
  • Protect against issues where changes in code result in new messages being emitted causing accidental big loops / recursion of unnecessary work
  • Generated handlers
  • Remove messaging system out of handlers and allow for much easier testing without mocks for message context
• Resiliency
  • Outbox pattern to store messages and guarantee delivery when message transport is down
• Monitoring
  • Consistent metrics across all types of messages

Scenarios

• Pub/sub for events
  • Fire and forget
  • Message acknowledgement
  • Can be many receivers
• Worker queues
  • Single Worker
  • Round robin workers
  • In process and out of process handling
  • In process can scale out with help of a message store where multiple instances of the application are storing messages to the same topic
• Request/response
  • Form of RPC and can be used instead of HTTP calls
  • Publishes message and then does a single message receive on a topic that is for that exact request and waits the specified amount of time
• Delayed delivery
  • Can schedule delivery
  • Messages are persisted to an outbox message store and a background task polls for messages that are due and then sends them out
• Single handler for multiple message types
  • Wildcard subscription (all messages from a specific namespace)
• Receive message (pull model)
  • Equivalent of current worker queues pulling a single message at a time
  • Ability to receive a batch of messages
• Cross cutting concerns
  • Ability to plug in middleware / interceptors into all message handling

Implementations

• Goal is to create a fairly simple abstraction that can be implemented in various ways
  • Messaging frameworks
  • MassTransit, Jasper, NServiceBus
  • Should be able to fairly easily create an implementation that wraps these systems
  • Direct
  • In memory implementation is necessary for testing
  • RabbitMQ
  • SQS / SNS
• IMessageBus for sending and RPC calls and IHandler for handling messages
  • Foundatio doesn't do any of the hard configuration / routing work
  • Handled by implementations / messaging frameworks

[niemyjski]

For Kafka:

Publishing usually specifies an optional message key (generic but typically a string) which is the partition key. We also need to pass through a context object that we can grab correlation id from. Also, we should be able to specify a message type that we can specify that maps to a specific type in subscribing platform.