RFC: How to better support an Event-Driven Architecture with Amplify DataStore?

[danrivett]

Is your feature request related to a problem? Please describe. We're looking at integrating a DataStore-based AppSync Mobile application with a backend which will be largely integrated using an Event Driven Architecture (EDA) to avoid tight coupling between services as well as provide domain specific events to trigger off rather than needing each integration to inspect model updates themselves to understand if they require any actioning.

EDAs also provide the opportunity for better insight into how the system operates by keeping a journal of events as they occur, as well as allow different materialized view projections of data when helpful (e.g. data indexed in ElasticSearch for it's geo capabilities).

Describe the solution you'd like Below is a possible architecture to support an EDA with AppSync and DataStore. It supports events being generated from changes through DataStore as well as events generated outside of AppSync/DataStore which then triggers updates back into the DataStore models.

We're currently planning on implementing an EDA architecture similar to this ourselves, however I feel this architecture is largely "undifferentiated heavy lifting" to quote Jeff Bezos in his famous "We Build Muck, So You Don’t Have To" keynote, given how popular Event Driven Architectures are.

Therefore I think it's a good candidate for being considered for an AWS-Managed approach in order to support an Event Driven Architecture with AWS AppSync and Amplify DataStore without requiring customers to create a lot of plumbing like the above.

Note: The diagram above provides a simplified view. In the envisaged architecture each Aggregate, in Domain-Driven-Design terms, or Model Type in DataStore terms would be persisted in a separate DynamoDB table, have a separate SQS FIFO queue and be separately integrated with S3 (for archiving and also future new event generation), among other things. So as the number of Aggregates grow, the amount of plumbing increases.

Request for Comment I'm creating this RFC to solicit feedback in general on how AppSync + DataStore can best participate in an EDA; whether this is recommended/discouraged; and seed thought about possible solutions on AWS' end in order to simplify this use case for customers and remove as much undifferentiated heavy lifting as possible.

Additional context Twitter conversation with @undefobj suggesting an RFC discussing this could be raised here.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[hnqso]

This is great @danrivett 👏! Have you made any progress, any findings in the past few months? Or anything relevant you could share? We have a AppSync mobile app and we do want to move more towards an event-driven architecture backend in 2021. At the moment, it does feel like Amplify and EDA aren't made of each other? Thoughts? Thanks 🙏

[danrivett]

@henriquea We've been working towards the architecture documented above this year, and it's been working for us in our development testing but we haven't yet got anything in production using it. It's required a bit of time to create all those pieces and test them. That said, we haven't been working full-time on this though as this project is one of many we currently have, so it would be quicker if worked on full-time.

Also I am hopeful that re:invent starting today will announce something in this space that may help, but we'll see.

But I would say that the architecture above definitely is doable from our experience if you've got the time and the patience to create the various pieces. And it is satisfying seeing granular events being generated and then responded to simply from user interactions with just DataStore in a React Native app.