Implement Sequence Diagram (phase 1)

[dannycohen]

1. Based on design discussions in #220
2. Please implement initial design for Sequence Diagram based on latest @sergioc design (see below)

   Designs

01 Normal 02 Saga msg selected 03 Saga msg selected (hover on msg name) 04 Saga msg selected (hover on saga name) 05 Message selected 06 Message selected (hover on name) 07 Saga msg hovered 08 Saga name hovered 09 Timeout hovered 10 Processing error label hovered 11 Endpoint hovered 12 Message handler hovered 13 Message menu 14 Processing failed menu

[dannycohen]

See example of raw data rendering as message flow and sequence diagram in https://github.com/Particular/ServiceInsight/issues/321

[SimonCropp]

@dannycohen considering the number of other important issues in 1.1 can this be pushed out to a later release?

[dannycohen]

@simoncropp - what other important issues for 1.1 ?

According to the prio list for LM2 this is a top prio.

// CC @indualagarasamy

[SimonCropp]

• if an event is published to multiple endpoints how would that be represented? What happens when those multiple receiving endpoints go on to publish multiple events?
• based on the above how will overlapping lines be drawn?
• what happens if we have a significant number of endpoints. just lots of horizontal scrolling?
• how are endpoints ordered?
• what happens when there is a significant vertical space required?

[dannycohen]

if an event is published to multiple endpoints how would that be represented? What happens when those multiple receiving endpoints go on to publish multiple events?

based on the above how will overlapping lines be drawn?

See above.

what happens if we have a significant number of endpoints. just lots of horizontal scrolling? how are endpoints ordered?

We have discussed as follows:

• In phase 1 the ordering will be based on appearance in the conversation.
• Following phases will include the ability to re-order the appearance of the Endpoints
  • (e.g.: Alphabetically asc/desc, manually by drag-n-drop by the users, and - my personal favorite - in descending order of communication frequency: i.e. endpoints that communicate frequently are located next to each other)
• In any case - this is for post phase 1, but the requirement is that the re-ordering will be supported (this is, IMO, an advantage in using Mindscape vs. roll-your-own plain-vanilla WPF)

what happens when there is a significant vertical space required?

Vertical scrolling happens. Same for horizontal scrolling. Also - zoom-on/out and panning. Again - comes OOTB with Mindscape, and can be delayed for post phase 1 if we can't squeeze that in.

[sergioc]

@SimonCropp I'd be happy to visualize a complex real-world scenario in the sequence diagram.

[sergioc]

Updated first comment with links to screen designs.

[dannycohen]

@sergioc - thanks! @indualagarsamy / @andreasohlund / @udidahan - See updated designs on Sequence diagram above.

[dannycohen]

@andreasohlund / @udidahan - are we good to go on this ?

[andreasohlund]

I'm good!

[udidahan]

Yup - good to go.

[andreasohlund]

@modul8com please review and comment on the updated design (see the issue description)

[modul8com]

Ok,I know what to do after this week of sunshine, pools, water slides, etc. Nothing I see seems impossible, though it might take long, since I heard I have to implement my own layout routine, etc to make it look as we want too, but all that might facilitate the timeline after all. Other things like the hovers, highlighting is already in my try outs.

[dannycohen]

@andreasohlund / @modul8com - Sounds good. Do we want to schedule review meetings with @sergioc and I to make sure we're on the right path ? (implementation wise and schedule ETA's, as well as making sure there's nothing missing in the requirements side)

[distantcam]

Ok, I've had a chance to evaluate the Mindscape Diagram controls with regards to this issue.

For me the Mindscape controls only get us about 70% of the functionality wanted for this issue, and only by heavily customizing it anyway. And that's visual functionality. It doesn't provide any of the interactive functionality.

• The endpoints are not part of the scroll surface, so they remain at the top of the view. This is not supported at all by Mindscape.
• The endpoint lines are also not part of the "diagram" so they would need to be drawn separately.
• The endpoint headers and lines must be kept in sync with the vertical scroll of the diagram.
• The Mindscape control doesn't manage reordering of the endpoint lanes. I don't know why you thought they did.
  • We could use grouping to control the layout of endpoint columns. But layout must be customized.
• We'd have to create a custom layout system to handle all this. Documentation is bare, so this would take a while.
• There's a lack of documentation with the Mindscape controls. The docs haven't been updated in 3 years and are basically autogenerated.
• The scroll/zoom functionality is broken in my opinion, and treats the diagram as a box on an infinite canvas, which means we can scroll "outside" the data, with no way to know how to get back to our view.

[dannycohen]

OK, so you're saying that the most efficient way to go is by avoiding Mindscape and starting from scratch ?

[distantcam]

@dannycohen I'm still investigating the options. All I'm saying for now is that the Mindscape diagrams are about the same amount of effort as doing it from scratch.

[dannycohen]

@distantcam - Understood.

FYI: One of the main reasons why we had seen Mindscape as a possibility is due to the relative ease of development of the ServiceMatrix Canvas and SI s message flow, which is, at least from a 30,000 feet view, similar to Sequence diagram (i.e. it also has "lanes" and dynamic connections that can change locations).

Either way - it would be good to have a fresh view on the possibilities + compare with SM Canvas to understand the Mindscape's limitations.

[distantcam]

That makes sense. The SI message flow is perfectly suited to what the Mindscape control is doing. The design control is great for unstructured layouts of nodes and connections. The problem comes in when you want to have a rigid custom layout, and elements that are not part of the design (like the endpoint lines, etc).

Also the design control is doing a lot that we're not using, like for example it's designed so the user could create their own flowchart. Here we're just using it to display our data. It makes more sense to use it in SM that way.

I'll keep investigating.

[dannycohen]

@distantcam -

The problem comes in when you want to have a rigid custom layout, and elements that are not part of the design (like the endpoint lines, etc).

We were also concerned about this, but what convinced us - at the time - was the easy ability in ServiceMatrix to implement lanes in which the endpoints live:

In the canvas these lanes are quite small - about 10px - and they allow both horizontal and vertical movement, but it is possible to place an endpoint within a rigid grid.

We assumed that the sequence diagram is a special case of the Canvas (i.e. make the Sequence layout options more restricted and less dynamic than the Canvas').

[modul8com]

I can only agree with what Cameron says in it's initial comment... after trying hard I managed to get some stuff done, but it's hard and at the end it always came down on creating your own layout system, which seemed like an easy task in theory, until I started with that... in practice it's bitching hard and because of the lousy documentation you just keep on trying and often failing. Like I always said, it feels more like forcing a square peg through a round hole... with brute force and loads of patience you'll get there... Starting with something custom developed is, in my humble opinion even worse - since you need to take care of a lot of tiny things, if you find the right Lego blocks to play with it might be a less ridged task to achieve.

[distantcam]

@sergioc @dannycohen How should it look if a message sends to the same endpoint it comes in on? Where should the name and popup for that message appear since there won't be a horizontal line for that message?

[distantcam]

Another question. What happens when a message sends to say 2 endpoints, one on the left, and one on the right of the originating endpoint? Draw a line to each? Where should the message dropdown part appear in that case?

[dannycohen]

@distantcam - good points.

Here's my view, while eagerly awaiting @sergioc response:

How should it look if a message sends to the same endpoint it comes in on?

I guess a line connecting to itself (not round-ish like timeouts, but with clear 90 degree breaks, that do not reach the adjacent endpoint's "area")

What happens when a message sends to say 2 endpoints, one on the left, and one on the right of the originating endpoint? Draw a line to each?

Yes.

Where should the message dropdown part appear in that case?

I think preferably on both (with the on-hover highlighting assisting in understanding context)

[sergioc]

@distantcam

How should it look if a message sends to the same endpoint it comes in on? What happens when a message sends to say 2 endpoints, one on the left, and one on the right of the originating endpoint? Draw a line to each? Where should the message dropdown part appear in that case?

Labels for messages sent to same endpoint should "attach" to send line and left align with saga label (if applicable). If you're wondering about the gap between the horizontal line and Message 4, it's there to make room for a possible saga label under Message 3 + give room to things to breathe (i.e. whitespace).

Note: This specific visualization may not cover all possible scenarios, it'd only feel right if all messages were sent at the same time and processed at the same time. @dannycohen Would it be desirable to differentiate in a more complex scenario e.g.:

• message 2 being sent after message 1 was was sent
• message 3 being sent after message 4 was processed
• etc. ?

[dannycohen]

@sergioc - focusing on messages 4 & 5 -

I assume you wish to vizualise 2 messages (or more) sent from endpoint X to endpoint X.

In the sample it sems like they are both processed together, which is not the case (separate processing per each message).

[sergioc]

@dannycohen How about:

[udidahan]

That doesn’t look good.

How about a “loop” back to the endpoint – down, left, down, right, all at right angles. Trying some ASCII art:

|

---

--->

[SimonCropp]

remember any approach needs to cope with n messages. So choose an approach that works for 100 messages as well as 1 or 2

[dannycohen]

Proposal - parallel lines, one for each message, separated by a pixel or two. Use on-hover highlighting to focus and get context.

This ought to scale relatively well, and provide context on-demand. It will also easily indicate which are the most prolific handlers (that send many messages). On Jul 4, 2014 10:10 AM, "Simon Cropp" notifications@github.com wrote:

remember any approach needs to cope with n messages. So choose an approach that works for 100 messages as well as 1 or 2

— Reply to this email directly or view it on GitHub https://github.com/Particular/ServiceInsight/issues/233#issuecomment-48014707 .

[sergioc]

@udidahan not sure what you mean there. Do note @simoncropp's comment.

@simoncropp this would scale to as many messages as you want, just widen the gap between endpoints. Can visualize that later.

@dannycohen how do you see that working with message labels?

[distantcam]

What problem is this diagram trying to solve? How does it help more than the current saga flow diagram?

[dannycohen]

@sergioc - Regarding:

how do you see that working with message labels?

For the "multiple lines emanating from a single message" labels will be displayed on-hover describing the currently highlighted line.

From a usability point:

1. When a message sends out more than 1 (and possibly many messages) our visual priority in Sequence diagram changes from "visualizing the message label details" to "visualizing the very fact that there are many messages coming sent by this handler"
2. This "visualization of multiple message" is problematic mainly when the sent messages are processed by the same endpoint
   • (for messages processed on different endpoints - not a problem since the lines do not criss-cross each other)
3. When there are many such messages - we can compromise on showing the message label on-hover
   • we can improve on that by making it "sticky" (user clicks on the label to make it visible even when not hovering above it) and / or in a form of a "callout" (with a thin line connecting to the message's line)

My bottom line is that adding more data (i.e. indication of multiple messages) is more important than the label being visible at all times for all such messages.

[dannycohen]

@sergioc - Possible visual:

[dannycohen]

@distantcam -

What problem is this diagram trying to solve? How does it help more than the current saga flow diagram?

Excellent question! The current message flow focuses on a sort of "narrative" of the messages themselves. Its visualization of endpoints is secondary, and problematic.

For example, using only / mainly the message flow, you can't (easily, or even at all, practically speaking) answer the following questions:

1. Which endpoints generate the most messages ?
2. Which endpoints are conducting very chatty interactions ?
   • at what point in the process do these interactions become chatty ?
3. Which endpoints are rarely (or never) involved ?
4. Which endpoints serve as "bottlenecks" in the process ?

All of the above questions (and many other we did not yet think about) have - potentially - significant impact on how the endpoint are structured, deployed, and which handlers should be deployed on which.

P.S. once we add the time dimension to sequence diagram, these questions become even more interesting, especially the "bottlenecks" issue.

[distantcam]

@dannycohen That's why I asked the question. :)

So, within the context of this diagram.

How easy is it to answer the following questions?

1. Which endpoints generate the most messages?
2. Which endpoints are conducting very chatty interactions?
3. Which endpoints are rarely (or never) involved?
4. Which endpoints serve as "bottlenecks" in the process?

[dannycohen]

@distantcam - IMO, quite easy to answer.

Q: Which endpoints generate the most messages? A: the messages are quite well distributed across endpoints. There's no endpoint serving as a "central hub", just the classic frontend to backend and back to front end interaction

Q: Which endpoints are conducting very chatty interactions? A: None. Which indicates a good design for the process flow (both message interactions and endpoints organization)

Q: Which endpoints are rarely (or never) involved? A: None. Each has well defined and scoped responsibilities

Q: Which endpoints serve as "bottlenecks" in the process? A: The bottleneck is in interaction and/or in processing. Since there's no obviously visible "central hub" endpoint there's no interaction bottleneck. When we add a "timeline stretch" feature (extend connections to indicate length of time to deliver / process) then the processing bottleneck aspects will also be easily visible.

Does this make sense ?

[distantcam]

Q: Which endpoints are rarely (or never) involved? A: None. Each has well defined and scoped responsibilities

What about the CustomerRelations and Operations endpoints? I'd consider them rarely used.

[dannycohen]

That endpoint has a clear role that abides by SRP so that's debatable.

But whether this specific endpoint is rarely or often used is not the point.

The point is that you and I can start discussing this without diving into heaps of code, simply by looking at the sequence diagram (and it would be hard / impossible to answer these questions based on message flow, which has other, more limited goals). On Jul 6, 2014 3:33 PM, "Cameron MacFarland" notifications@github.com wrote:

Q: Which endpoints are rarely (or never) involved? A: None. Each has well defined and scoped responsibilities

What about the CustomerRelations and Operations endpoints? I'd consider them rarely used.

— Reply to this email directly or view it on GitHub https://github.com/Particular/ServiceInsight/issues/233#issuecomment-48110723 .

[distantcam]

@dannycohen Excellent. That's clarified things for me, thanks.

Here's a standard sequence diagram for the same scenario.

Why can't we use a single straight line for the message line, and a loopback line like for the timout for messages on the same endpoint?

I personally think the current design is visually cluttered. If we instead simplified the lines and showed important information in the message section over the line that would be enough.

[dannycohen]

@distantcam -

Why can't we use a single straight line for the message line ?

You mean:

vs.

In essence: the simplified "generic" sequence diagram displays designed and intended behavior, while the sequence diagram we have in mind for SI focuses on actual runtime behavior that provides information (visual and textual) on real behavior.

Specifically, the addition of the "delivery line" and the "processing rectangle" (both highlighted below) allows the display of additional data on the actual processing of the message.

For now (v1 of Sequence diagram), the "processing rectangle" will display basic processing statistics:

In the future, we would like to add more support for visualizing the time dimension by extending the "delivery line" and the "processing rectangle" based on how long it took the message to be delivered and processed, allowing for a much easier understanding of how the process behaved over time.

and a loopback line like for the timeout for messages on the same endpoint?

A loop back line is not a bad idea. I believe the proposal above (https://github.com/Particular/ServiceInsight/issues/233#issuecomment-48106578) is quite close to a loopback (for multiple message).

But note that the generic sequence diagram does not depict the connection of the message sent and the handler that sent it. Tying the "message loopback" to the sending handler provides valuable information on the endpoint's behavior (i.e. "the message emanated from a specific piece of code that I can debug!")

This again goes back to the theme of having a sequence diagram display real information about runtime behavior (i.e. with handlers and performance info included and depicted) as opposed to an amorphous design.

Make sense?

[sergioc]

Assuming several loopback msgs are possible, but extremely unlikely, I think a slightly tweaked version of Danny's proposal could work:

We'd lose the ability to see message labels, but have it as uncluttered as possible too. Major drawback is having to scroll a lot if there's several loopback messages but, again, I'm assuming such scenario is very unlikely.

Thoughts?

[udidahan]

Keep in mind that the messages an endpoint sends itself could be processed at the same time by different threads.

[SimonCropp]

also worth considering that in many simpler NSB implementations a single endpoint scenario is uses. This has been the case on several project i have been involved in. So in these cases the single endpoint sends all messages to itself.

[dannycohen]

@sergioc and I were discussing this.

One possible approach - which may make the implementation simpler as well ( @distantcam ?) is to allow multiple "threads" to be displayed within an endpoint, allowing it to "swell" (become wider) as needed.

Something like this:

As I see it, this has some advantages:

1. It resolves the issue of visualizing multiple concurrent threads within the same endpoint (with or without timeline stretch)
2. It can scale quite well (with the endpoint "region" becoming as wide as required in order to depict the number of messages involved within a single endpoint
3. It provides a clear indication of "busy-ness" of an endpoint (which may, depending on circumstances, be considered "a design smell")

For now, I suggest we leave the message labels as displayed only on-hover. Later we can add them on-demand (e.g. toggle button to "Always show labels") or find some other approach to improve UX.

Thoughts ?

[distantcam]

Won't each of those lines down be a different message?

[dannycohen]

@distantcam -

Won't each of those lines down be a different message?

Yes, a different message send by the same handler and processed withing the same endpoint. (in the above visual, we have one message that was procesed by a handler that sent 4 messages that were processed by the same endpoint as the sending handler).

Theoretically - and that's Simon's point (https://github.com/Particular/ServiceInsight/issues/233#issuecomment-48243480) - we can have an entire system with a single endpoint, processing all the messages. This may be a "design smell" but it is technically possible (and sometimes leads to interesting support cases :-) )

[sergioc]

@distantcam those 4 lines going down are 4 different messages.

More complex scenario (including same endpoint sending message to itself again):

@dannycohen your last image is nearly what I've been recommending the most so far. Added bonus, btw, is possibly simpler implementation (@distantcam ?) since there's less need to calculate line collisions and re-routing. Re-worked this option a bit, see below:

[indualagarsamy]

@dannycohen - I don't think its a design smell. In fact if you design it right, it can start with one e/p and then scale to multiple e/ps, if needed.

• Also, the vertical lines look super confusing. What if other messages originated from these?
• And the bar growing horizontally also looks weird in a sequence diagram.
• What about a simple loop back?

[dannycohen]

@sergioc - I like your latest design except for the width of the sending handler:

I think it is not clear that this specific handler sent all these 5 messages. Making it wider (and less "handsome") will make it clearer: