plorenz
commented
1 year ago Notes from Russell:
First design decision I made dealt with granularity and purpose. The MOP's resource change messages are fired every time a property on the resource changes, regardless of whether that particular set of property changes is part of a sequence that compose a more logical change in the resource. For example, creation of a Service might result in two messages where the first is the initial creation message and the second is the update of the zitiId property after it is created in the Ziti Controller.
This level of granularity makes it easy to implement. Every time a change to the resource is persisted, a change message is fired. (Side note, the change messages are bound to the DB transaction. If the transaction is rolled back, the messages are not fired. No false messages.) The downside to this is that it shifts work to the message consumers; they have to be more selective when choosing what messages they should react to since there isn't a single logical "this meaningful thing just happened" message. In my experience, it is difficult and error prone to try and predict a higher level set of logical messages like that. Even in seemingly simple cases, it can quickly unravel as you discover a need for a message half way through what was thought to be a cohesive logical activity sequence, and such discoveries tend to lead to a slippery slope of overlapping, purpose specific messages ... and now you have a messaging anti-pattern, where the message is a substitute for a remote procedure call.
That being said, my design intent with MOP messaging does not exclude potentially creating a higher level, logical event set. For example, (arbitrary and made up) we could decide to define a set of messages that strictly define key points in a Customer's life cycle, like 'changing type, teams to growth' or 'payment status, good standing to in-arrears'. I think the key point is that this kind of logical message design and the aforementioned, generalized entity change message design, should not be colluded.
Second design decision dealt with the message structure. The whole idea behind messaging is to decouple the sender from the observer, and that has implications, particularly with regard to the shape of the message. ie, the sender can't emit XML when the observer is expecting JSON. And perhaps less obvious, the sender can't arbitrarily change the logical content or shape of it and expect observers to cope with it.
In the MOP, resource change messages have a well defined envelope that all messages are expected to conform to. The envelope contains a set of properties (I'll go over them in a minute) which observers can use for much of their decision logic. Since these are part of a universally agreed envelope structure, this ensures compatibility across observers. At the same time, the envelope has support for an arbitrary payload; a 'from' and 'to' version of the resource that has changed. The envelope nor the MOP in general make any guarantee about the structure of those two properties. Thus, it is up to the observer to peer into them, deal with potential unexpected structures, etc. This keeps the coupling of logic as confined as possible. Only the class handling that message contains an expectation of structure, matching that class's purpose.
Third design aspect deals with message routing. While it's possible to just emit a stream of messages, it forces all observers to drink from the same fire-hose. We want / need the ability to declare new observers at will, and for them to be able to express, to some degree, what subset of messages are of interest to them. And it's especially nice if they can offload things like durability, retries, etc.
MOP uses RabbitMQ (a JMS compliant messaging broker; think ActiveMQ or AMQP) to meet these needs. All messages have a set of headers, just like an http request. The most important of which is the routing key (akin to an http request's url path). Rabbit (AMQP?) uses a dot'ed string notation for routing keys, so MOP resource change messages all have a consistent key pattern, like this: ResourceChange.v1.{change-type}.{resource-type}. This gives observers the ability to express interest in a subset of messages by specifying a matching pattern against message keys of that pattern. If the observer want's all messages, not just resource changes, then they subscribe to . If they want only resource change messages (version 1), they subscribe to ResourceChange.v1... In most cases, observers are interested in specific resource types like a Config, in which case the subscribe to ResourceChange.v1..Config or perhaps just when they are created, ResourceChange.v1.CREATED.Config.
Rabbit provides a lot of additional functionality for MOP (not suggesting Ziti use it; just elaborating things to consider), including the ability for an observer to declare if messages matching its interest should be queue'd in the broker when the observer is offline or just dropped, any TTL on handling those messages, what to do if they timeout or fail, and of course support for messaging topologies. After listing that out though ... I'm presuming ziti will 'dump' these messages to one of a small set of supported destinations, even if it's a local journaling file, after which its up to something else (like a 'beat' from Elastic) to capture and pump the message into whatever system is expecting it. ie, all the things I listed in the first sentence are not problems I'd expect ziti to solve.
Fourth and final design topic, and prelude to listing the universal properties in a change message envelope. Those envelope properties (and the routing key parts) contain values which have to mean something to the observer. For example, all MOP resource change messages have a {change-type} in the key and the value can be CREATED, UPDATED, or DELETED. Unless we change the version number, there will never be a DEACTIVATED, REGISTERED, or anything else. If there were, then the limited universal expectation between message sender and the observer would be violated.
In the MOP, I was able to select emerging 'standard' ways of expressing various property values. Fortunately those existed in some form or another, so I was never in the position of making up something new and specific to the ontology of change messages.
So, without further ado, here are the envelope properties of a MOP resource change message:
id - A UUID, generated for the message. Often ignored, but practically required if we ever need to ensure there is no duplication or otherwise want to track a specific message. changeType - One of CREATED, UPDATED, or DELETED. Fairly straight forward. Being an envelope property, this makes it easy for observers to quickly sort the message based on their need. "Contextual Properties" - Within what execution context did this change occur? correlationId - Placeholder for potential client provided "correlation" id value, so that we can trace all things related to that client specified request. traceId - All MOP executions (every time a thread starts business logic), a "trace" is created and its id is placed into all logging, messaging, and even API calls to other services. This lets us filter out all the noise and see only the logs, or messages in this case, which are related to a particular thread of execution. spanId - Related to a trace, whenever a logical thread of execution jumps from one service to another (ala rest request), the trace id stays the same but a new span id is created. So, a span id is attached to all logs, messages, etc. within the scope of a single service (technically we can define our own span scopes), while the trace id will link all of the spans across services. general comment: These values come from our use of emerging standards for distributed tracing. See https://opentelemetry.io/ if you want to dive into the deep end (I see lots of go stuff!) Or, simply look for a way to "instrument" ziti components here: https://opentelemetry.io/docs/instrumentation/go/ and start by figuring out how to get the trace and span ids into logs and events. After that, for actually distributed execution tracing, you'll want to look at "propagators": https://github.com/open-telemetry/opentelemetry-go-contrib/tree/main/propagators which take care of passing trace ids between components (among other info). There are emerging standards for this as well. MOP uses a format referred to as "B3" https://github.com/openzipkin/b3-propagation#single-header but we're migrating to a W3C format https://www.w3.org/TR/trace-context/ initiatingIdentityId - Who triggered the execution which caused the change that this message is about? All things in MOP can be attributed to a MOP identity id, even if it is the MOP service itself; MOP services have a "Service Identity" which is used if they cause a resource change as part of some scheduled process (for example.) "Resource Identity" - There are a set of properties that all answer this question, and each targets a slightly different purpose. As you read these, remember the design goal that these property values have to be understandable to all observers; they have to be expressed in a cross-service, enterprise language. resourceDomain - At the MOP Enterprise level, all services and all of the resource types defined and managed by a service ... they are all organized into 5 top line "domains": management, identity, networking, billing, and auth. We may end up with more, but that's it for the moment. Some of these domains have several micro-services within them, and some have one. Each micro-service defines and manages its own tree of resources. ... and all resource change messages include the domain within which the message's resource is defined. resourceType - Like domain, all resource's have a declared string, technically called the resource type "code". This is that value. Examples are network-controller, service, account, etc. resourcePath - This property contains a URN like value which starts with the resource's domain, and then contains a list of resource type and id pairs. The list is in order from the root of the resource tree all the way down to the resource type and id that this change message is about. This "resource path" structure is what is used by the MOP to perform authorization. MOP's auth is designed around granting permission on resources that subset a given path; for example, permission to update any service resource as long as it is under network-group:333:network:111. So ... from a resource change message perspective, having this property (and trusting its authenticity) lets MOP apply authorization constraints to these messages if/when we allow them to be consumed by MOP clients. resourceId - The literal id, UUID in MOPs case, of the resource. "Envelope Content" - These are resource change messages, so the content is a resource. Since this is a change message, the envelope supports the former and new version, and this is where the polymorphism starts. In MOP code, these two properties are arbitrary JSON blobs. This ensures there are no class dependencies introduced across message senders and receivers, and if the receiver wants to unmarshall one of these properties into a locally defined type, then the onus of dealing with mismatches is on that code. fromVersion - Possibly empty (on create for example), arbitrary JSON version of the resource before the change. toVersion - Possibly empty (true deletion), arbitrary JSON version of the resource after the change. The in code representation of the resource change message envelope provides support for the common task of comparing deep properties of the from and to version (at the same path address.) This is super handy for observers that get all change messages and just want to know if the property they are interested in has changed. occuredAt - Last but not least, a timestamp set by the service sending the message. No guarantee that it is a match for the resource's updated at time, or that all messages will be in order, etc. But otherwise a fundamental, "when did it occur?" time stamp.
dovholuknf
Russell-Allen
Add events when entities are created/updated/deleted.
Need to figure out how we want to format events. Should it based of the DB representation (easiest), REST API representation (+ for consistency), custom events format (most control, but most work and reburies more upkeep)