duglin
commented
3 years ago Therefore a first filtering would occur on data distribution from the DDS and then a second filtering and multiplexing would happen on the UDS side of the ring and even a third in the browser.
The first two steps/layers would each turn into some kind of subscription, right? If so, would it be ok if it was modeled as independent subscriptions or are you looking for a way to have a single subscription encapsulate these multiple layers?
erikerikson
clemensv
Responding to Doug's request for a subscription related use case... For context the broad space of the use case is IoT.
The model that I have have been promoting is a sort of ring on which different domain specialists sit, interacting with the external world (from a system perspective) and with one another. Each of these is actually business logic relating to the domain, packaged in microservices, also containing identity and proof assets, archives of event streams, and materializations of event streams.
Each domain specialist interacts with the sensors and actuators of their domain (e.g. device or users) as sources of event descriptions of observations/measurements/syntheses and commands for actuation (whether display specifications or motor target changes). Each domain specialist instance contains all the security configuration, rights, and subscription details over the set of associated domain entities and acts as gatekeeper to accessing those entities and as publisher of information produced by those devices and more specifically the sensors onboard them.
The next layer of consideration is then those interested in the products of one or more domains which have their subscriptions and control streams over domain entities. Particularly in a device management use case, there is a user domain specialist that presents device (and other) information to users that manage the devices to ensure their proper configuration and continuous proper function. Users view specific devices and summaries of device observations as well as ongoing updates (e.g. alarm activations or clearances, live metric streams, et cetera).
The domain interaction has some interesting cases but is relatively straightforward in terms of entity observations and command issuance but the transmission between ring participants is where the most interesting considerations come up. The device domain specialist (DDS) will have all device information available to it, including some that it may not even process (but still will archive for future unknown but probable business processes and knowledge requirements [we are a resource constrained startup with expectations of increase resource availability as we scale]). A user domain specialist (UDS) will want a subset of all that information (available to the DDS for a subset of devices [e.g. current configuration and telemetry]) for presentation and notification purposes.
Looking closer, specific users will want a subset of that subset of all device information. The most specific subset of devices and specific data is dynamic based on user activity. For maximal efficiency the UDS would continuously push the complete specific information filtering configuration upstream to the DDS (and perhaps devices) in real time but this is probably unwise given the continuous updating of that interest based on the user's direction of their attention as they click through pages. As is my want, I use event order, stream joins, and log processing to achieve consistency in the system and I care that guarantees of these properties are maintained throughout the system. To summarize it, the process of data flowing from devices to users and from users to devices passes through authorization layers and filters but for all authorized data there are multiple levels at which the data is coalesced, filtered, and delivered based on limits of local capacity to efficiently serve dynamicity within known and changing scopes of subscription.
Do we want subscriptions to the Cloud Events Subscription API updated on devices as a user navigates through pages and even as they interact with adjusting a specific device graph? I'd guess not (certainly Azure currently makes this a deploy-time decision) but I think that is one extreme end of the spectrum at play here. The other end of the spectrum is the "degenerate" case described on a call as a behavior of "enterprise developers" who request all data be delivered (until challenged). In this case, all device information, for all devices, would be subscribed to and locally filtered (on the browser if we want to be entirely ridiculous) to just that data which is desired or actionable. Of course, instead of either of these ends of the spectrum, I would expect that the UDS will identify the device data that it supports users in interacting with. It will likely use shared materializations of that data (historically indexed by affecting offset [with TTL]) and subscribe to updates over the data for all devices. It will then manage distributing those updates according to local dynamic server-side filtering based on what users individually attenuate to and additionally in browser based on appropriate knowledge given the current presentation state. Therefore a first filtering would occur on data distribution from the DDS and then a second filtering and multiplexing would happen on the UDS side of the ring and even a third in the browser.
This is a simple statement of a simplest case of direct information flow and management in a domain specialist ring based design. In more advanced cases we get into the tracking of transitive data flow chains where the processing of one or more (i.e. joined) ordered event streams produces a new event stream that can be the subject of subscription/interest (e.g. the synthesis of alarms from telemetry data contingent on device configuration and status; those alarms displayed and updated in the presentations of one or more users). This can be vital to scenarios of effect after cause or its single hop version read after write.