Question about using servicebus in current architecture.

[CodisRedding]

• frontend angular app
• proxy (frontend proxies through to backend services)
• backend service 1 (REST)
• backend service 2 (REST)
• backend service 3 (REST)
• etc..

We currently rely on each backend service being aware of other backend services. So when a scenario like new user registration happens we:

• frontend makes a request to register a new user
• backend service 1 receives the request and makes several other calls to other backend services in order to aggregate a response directly back to the frontend.

I'm considering using a message broker to handle our backend services communication but I'm unable to wrap my head around how I would do this. If you took the registration scenario above and completely decoupled the backend services via servicebus, how would I go about creating that same aggregate response for the frontend?

[mateodelnorte]

Hi @fourq. I'm working on a site that has examples of all of this, but here's the starting advice I would give.

I typically design systems with an architecture called CQRS. Basically, CQRS means that when a user performs an action in a UI, the web server communicates that action to the backend via a command. Commands are sent directly from the web server to a service responsible for that command's part of the business domain. So if a user updated their personal info, the web server might send a 'user.changeAddress' command to the auth-service. I use servicebus to send these commands so that services don't have to know where others reside. RMQ does the routing.

When auth-service is done updating it's internal state, in it's own local database, it sends another message via servicebus. But instead of doing a direct send to another service, the way commands are done, it publishes a 'user.addressChanged' message that is broadcast to any services which care about that message and implement a listener for it. Any number of services living in domains other than auth might care about that message. A marketing-service, for instance might update update their service's local database with the updated address, so that letters mailed to the customer go to their new office. After auth-service publishes, n number of services will receive the message and so the data propagates it's way through the system, becoming eventually consistent.

There is one special service in CQRS called the denormalizer. He's the special service that subscribes for the far-majority of the events your distributed services will publish. His job is to subscribe to those events and save them in a denormalized form, into a horizontally scaleable database whose sole purpose is to feed your UI. *Note, you can have a single denormalizer serve all your UIs, including APIs, or you can choose to have multiple. The choice is yours, but usually starts with one and goes to many when a certain UI's requirements differ from the others (such as a reporting system).

So, here's how the single-direction flow of the system works:

1. user opens app, application reads from denormalizer db to get current state
2. user clicks on button, causing browser to send command to web application. web app performs any remaining validation and sends a correctly formed command to an appropriate service.
3. backend service retrieves sent command, transactionally updates its internal state or local store. _note that each service can update and save data or perform actions in its own way, and that it's not required that services even follow the same internal patterns or tooling (though it sure is nice when they do!)
4. signifying completion of command, backend service publishes one or more events that signify that certain changes have occurred to one or more local entities, resources, etc.
5. n number of services subcribed to the published events retrieve those events and subsequently update their internal state or local store, repeating steps 4 and 5 with events only ad nauseum until the dance/choreography completes itself. 5a. one of the services listening to the published events in 4 and 5 is the *denormalizer_. as it receives the events, it updates the horizontally scaleable (ready-only, from the client's perspective) database. If you're using something like mongo or reThinkDb then your web application can have op-log style notifications that something has changed, and reactively update. alternatively, you can implement simple polling and timeouts for actions that can't be reactive or don't need to be.

Does that make sense?

[mateodelnorte]

The main thing to consider here is that it's a mental switch from request / response, in a REST-only architecture, to one where services are purposefully pushing data to other services that care about that data, in real time.

There are ways for you to try out this different architecture or step your way toward it.

You could, effectively make your backend service that currently collects responses from other services instead be fire-and forget from the UI's perspective and publish that response to a new denormalizer service which will save that data to your database.

You could also choose to simply publish an event from your current collecting service, and have the downstream services perform their operations, publish when they're done, and have the denormalizer collect that data to be saved to the denormalized db.

Again, these are all steps you can take such that your architecture, instead of requesting data when needed, pushes data to where it will be used, ahead of time, whenever it can.

[eddieajau]

Matt, would something like ElasticSearch typically/possibly fit the bill for the denormaliser?

[patrickleet]

Andrew, you could have a denormalizer service that writes to ElasticSearch. Denormalizers can write to any db that you'd like. It depends how you want your client to consume that data. If you have a client that you would like to perform searches from than ElasticSearch could be a great choice. On Tue, Sep 13, 2016 at 4:33 AM Andrew Eddie notifications@github.com wrote:

Matt, would something like ElasticSearch typically/possibly fit the bill for the denormaliser?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/mateodelnorte/servicebus/issues/70#issuecomment-246586691, or mute the thread https://github.com/notifications/unsubscribe-auth/AAc-y4UePrDu5TJu0mMkazFlrfEDLuWxks5qpkFtgaJpZM4JK1Pd .

[CodisRedding]

So, i'm writing up some ideas to present to engineering. Here's what I have so far. I have a couple questions though.

• Where in this diagram does redis for retries sit?
• Have you seen anyone using REST HTTP API to proxy commands like shown, and do you see any issues with this approach?
• Thanks for looking

[mateodelnorte]

Check out this slide from my talk: https://youtu.be/4k7bLtqXb8c?list=PLTOk-Smvo1l6brny4es2Hprl5TDPMNYQm&t=989

Your system is a little different in that there is a denormalizer for each downstream application in yours. You can make that decision, but note that if all of those denormalizers are listening for the same events from all services you'll be quadrupling the number of messages going over queues via rabbitmq. There are trade offs between having one big denormalizer vs one per downstream application (and you can have a blended approach) but it's something to think about.

One thing to note in the slide I linked is that all of the solid or dotted yellow lines are rabbitmq. So, it's a different way of looking at things. The reason I diagram it that way is to show that the architecture is a unidirectional system. These systems are different than how many people build systems, but in every case I've seen them done - they end up very easily understood.

As for the proxying, I assume you mean that a browser makes a POST to a web application which then sends a command via RabbitMQ to a service? That's the correct way to view it, yes. Similarly, if you created a gateway service whose job is to track some 3rd party's state and bring that data local to your solution, it would work the same. At email service, for instance, might have some web hooks or a websocket connection to SendMailProviderCo. When your sent mail gets opened, their server would send a message to your sendmailproviderco-gateway-svc and update it's local state, then publish an event to the world... which would end up in any interested service, including denormalizers. Denormalizers would then update databases and clients could then refresh, poll, or have the new data streamed to them.

[CodisRedding]

thanks for all that.

I skipped around on that video when I had a chance, and I was wondering if the nycnode (denormalizer, services, etc) code is on Github or not. I did a quick search but didn't see anything.

[CodisRedding]

I just watched your talk...wow! exactly what I needed. Very helpful.

[mateodelnorte]

Spread the word and share 😄