NicolasNewman commented 4 months ago

Proposal

Proposal: Cloud Architecture Diagram

Motivation

The recent addition of Block diagrams greatly helped in allowing users precise control over where nodes (blocks) are placed. While they are useful for representing API or Network diagrams, it proves to be unweidly for cloud architecture diagrams such as (1) or (2)

Terminology

Service: a node within a diagram. Represents services such as from AWS, Azure, Docker, etc
Group: a collection of related services (similar in presentation to subgraphs for other diagrams). This could be a VPC on AWS, different APIs which talk to one another, etc

Syntax

A group can be defined as group {id}[{title}] (in {group_id})? where id is the value to reference the group and title is the text to display along othe border. (in {group_id})? can optionally be used to position a group within another group
A service can be defined as service {id}({icon})[{title}] (in {group_id})? where id is the value to reference the service, icon is the svg resource to use for the node (more on this later), and title is the text to display under the service. Like with groups, (in {group_id})? can be used to declare which grouping this service falls under. If no group is defined, it will default to be free floating and the renderer will decide where it goes
Services and groups can be related with the syntax {id_1} (<)?(L|R|T|B)-([{title}])?-(L|R|T|B)(>)? {id_2} where id_1/2 are the two services/groups to connect and title is the optional text to display along the line. The positional relation between services / groups is declared using L|R|T|B on both sides of the arrow. For example, L on the LHS means the line will come out of the left of the service id_1

Example

The diagram shown in (2) can be represented as:

architecture

    group rest_api[Rest API]

    group web_console[Web Console]

    group image_pipeline[Image Pipeline]
    group codepipeline(codepipeline)[AWS CodePipeline] in image_pipeline

    group testing_engine[Load Testing Engine]
    group vpc(vpc)[Amazon VPC] in testing_engine

    service cf(cloudfront)[AWS Cloudfront] in web_console
    service store(s3)[Amazon S3] in web_console

    cf <L--R> store

    service api_gateway(apigateway)[Amazon API Gateway] in rest_api
    service lambda(lambda)[AWS Lambda] in rest_api
    service congnito(amzn_cognito)[Amazon Cognito] in rest_api
    service iam(amzn_iam)[AWS IAM] in rest_api

    api_gateway <L--R> lambda
    api_gateway <B--T> congnito
    congnito <L--R> iam

    web_console <B--T> rest_api

    service store_2(s3)[Amazon S3] in testing_engine
    service db(dynamodb)[Amazon DynamoDB] in testing_engine
    service sqs(amzn_sqs)[Amazon SQS] in testing_engine
    service lambda_2(lambda)[AWS Lambda] in testing_engine
    service ecr(ecr)[Amazon ECR] in testing_engine
    service cw(cloudwatch)[Amazon CloudWatch] in testing_engine

    store_2 <R--T> lambda_2
    db <R--L> lambda_2
    sqs <R--B> lambda_2

    service fg(fargate)[AWS Fargate] in vpc

    lambda_2 <R--L> vpc
    vpc <R--L> ecr
    vpc R--L> cw

    service docker(docker)[Taurus Docker Image] in image_pipeline
    service store_3(s3)[Amazon S3] in codepipeline
    service codebuild(codebuild)[AWS CodeBuild] in codepipeline

    docker R--L> codepipeline 
    store_3 R--L> codepipeline
    image_pipeline R--T> ecr

Considerations

Should N/S/E/W be supported alongside or instead of L/R/T/B?
group and service are names I quickly came up with. I'm sure there are better options.
Alternate ideas for positioning services? I haven't thought of this from the renderer's perspective so I currently don't know if there are flaws with this approach.

SVG Icons

As mentioned earlier, the icon field for a service declaration will map to a SVG icon. Currently to use external SVG icons in Mermaid, HTML tags need to be used which points to the icon. This method isn't practical for a digram in which every node references an icon. Since we likely wouldn't want to host the icons within Mermaid for common services (either due to legal requirements or increasing the bundle size), I belive a standardized system should be implemented where icons can be defined and registered in the initialize() step. A standard API can additionally be exposed so users can create their own 3rd party modules which contains bundles of icons.

Since there are many ways to go about implementing this functionality, I'd love to hear others thoughts. If approval is given for begining work on this diagram type, svg handling should be moved to its own issue and potentially implemented for another diagram for testing before moving forward with the architecture diagram.

Relevent issues: #3358, #3746, #3124, #1723

Use Cases

No response

Screenshots

No response

Syntax

No response

Implementation

I will try and implement it myself.

sidharthv96 commented 4 months ago

Love the idea in general! This would be a very powerful addition to mermaid, after Katex 🚀

We can't sadly use < in the syntax, as that'll be interpreted as an opening html tag, when mermaid is written in HTML. What are your plans regarding layout libraries? We have Cytoscape and Dagre as 2 options currently. (We are doing some work to decouple layout engines/algorithms, mainly to move on from Dagre, so would love to hear your thoughts on this)

Please refer to the following PRs on how to use Langium to add a new diagram grammar.

NicolasNewman commented 4 months ago

@sidharthv96

If < can't be used do you think (/) is the next best choice? I've also already written the grammer is jison. Would it be ok to keep it as that or would you rather it be converted? For layout libraries I don't have experience with either so I'll look into Cytoscape if that's what's being pivoted too.

And what are your thoughts on displaying SVG icons?

sidharthv96 commented 4 months ago

The move away from Jison is finalized, and the first version with Langium will be v11. So adding a new diagram in jison is taking on extra work on migrating it few months down the road. So I would strongly suggest using langium before merging the PR, but won't be a hard blocker. I'll update the contribution guidelines to reflect this change as well.

As you've already written the grammar, let's work on the rest of the stuff before converting it to langium. Keep the < as well for now (it should definitely be changed later), let's see what the other maintainers have to say as well.

Bundling the icons with the diagram is going to be a maintenance issue, so I like your solution. But, the problem is that, in that case the diagram cannot be used in most places, unless people explicitly add support (GitHub/GitLab/etc). Github is already way behind our release schedule, so I don't think they are going to add extra icon packs easily. Maybe we should identify a subset of generic components, and include the SVGs in the diagram, with the ability to add extra packs as well.

NicolasNewman commented 4 months ago

I'll convert it to Langium once the render is finished if the plan is to backport all the other diagrams.

Another potential solution for icons is also letting web urls be used for although I don't know if there will be issues that could arise from things such as CORS settings. Given I can't directly link an image via [img](https:/.../img.png) here I'd assume that won't work for GitHub/GitLabs. That may be less of an issue for OS software that uses Mermaid such as Joplin and they could benefit from having url support.

I do like the idea of including some generic icons. I'm not a graphics designer or a legal expert so do you know where existing icons can be found?

Ronid1 commented 3 months ago

From my understanding, the < syntax works as long as it's followed by a > tag, so it should be fine for the syntax proposed.

I really like the arrow direction flexibility (RL/ TB) you proposed here, and think this could be valuable in other Mermaid diagrams. I would suggest using a grammar that is both easy to borrow into existing diagrams and will not break their current functionality. Maybe adding the direction before or after the arrow? something along the lines of TB--> or <-->(RL)? I think this kind of syntax would be easier to copy while leaving the direction optional for already implemented diagrams.

NicolasNewman commented 3 months ago

From my understanding, the < syntax works as long as it's followed by a > tag, so it should be fine for the syntax proposed.

I really like the arrow direction flexibility (RL/ TB) you proposed here, and think this could be valuable in other Mermaid diagrams. I would suggest using a grammar that is both easy to borrow into existing diagrams and will not break their current functionality. Maybe adding the direction before or after the arrow? something along the lines of TB--> or <-->(RL)? I think this kind of syntax would be easier to copy while leaving the direction optional for already implemented diagrams.

I can definitely adjust it for that. I've started work on the renderer and basic directionality has been added. I added the package cytoscape-fcose as it provides options for fixing nodes horizontally and vertically from one another for the LR/TB syntax.

A file has also been added for basic icon registration but for now it's a temporary solution so it doesn't block progress. It currently relies on importing resolvers that takes a root SVG node and injects a string containing the svg. For example:

import { Selection } from "d3-selection";

export default (parent: Selection<SVGGElement, unknown, Element | null, unknown>) => {
    parent.html(`<g id="Icon-Architecture/64/Arch_Amazon-DynamoDB_64" stroke="none" stroke-width="1" fill="none" fill-rule="evenodd">
    <g id="Icon-Architecture-BG/64/Database" fill="#C925D1">
        <rect id="Rectangle" x="0" y="0" width="80" height="80"></rect>
    </g>
    <path d="..." id="Amazon-DynamoDB_Icon_64_Squid" fill="#FFFFFF"></path>
</g>`)
    return parent;
}

These get added to the object map in svgRegister.ts which can then be used like so for adding the icons to a diagram.

NicolasNewman commented 3 months ago

architecture
            group vpc[Private VPC]
            group vpc2[Public VPC]

            service s3(s3)[S3 Bucket]
            service rds(rds)[RDS DB]
            service ddb(dynamodb)[DynamoDB]
            service ec2(ec2)[EC2 Server] in vpc
            service gateway(api_gateway)[API Gateway] in vpc
            service docdb(documentdb)[DocumentDB]
            service lambda(lambda)[Lambda] in vpc

            service serv1(ec2)[Public Server] in vpc2
            service serv2(ec2)[Private Server] in vpc2

            serv1 L--R serv2

            s3 L--R rds
            s3 B--T ddb
            s3 R--L docdb
            gateway L--R ec2
            gateway T--B rds
            gateway B--T lambda

update: groups are done

sidharthv96 When you get the chance could I get some feedback on my current implementation? My two main concerns are adding cytoscape-fcose (119.4kB minified) and how svg icons are currently handled (see previous comment).

sidharthv96 commented 3 months ago

cytoscape-fcose is not a major concern, as our diagrams will be lazy loaded. We're also adding a mermaid.tiny.min.js file, which will not include heavier features like elk, cytoscape, katex, etc.

Can you please raise a draft PR so it's easier to review?

Some features to note

Groups within groups
Groups with icon, can copy service syntax group id(icon)[text]
Links to & from groups
Copy styling syntax from flowchart for lines, background, etc.

Re icons, ZenUML has support for icons, but it's currently an external diagarm. You can check out how they are adding the icons (I think it's with SVGs in repo).

Some thoughts on grouping syntax, OTTOMH

Combined declaration and nesting.

architecture-beta
  service rds(rds)[RDS DB]
  service ddb(dynamodb)[DynamoDB]
  service docdb(documentdb)[DocumentDB]

  group aws[AWS]
    service s3(s3)[S3 Bucket]

    group vpc[Private VPC]  %% group in group
      service ec2(ec2)[EC2 Server]
      service gateway(api_gateway)[API Gateway]
      service lambda(lambda)[Lambda]

    group vpc2[Public VPC]
      service serv1(ec2)[Public Server]
      service serv2(ec2)[Private Server]

Declare, then organize.

architecture-beta
  service ec2(ec2)[EC2 Server]
  service gateway(api_gateway)[API Gateway]
  service lambda(lambda)[Lambda]
  service serv1(ec2)[Public Server]
  service serv2(ec2)[Private Server]
  service s3(s3)[S3 Bucket]
  service rds(rds)[RDS DB]
  service ddb(dynamodb)[DynamoDB]
  service docdb(documentdb)[DocumentDB]

  group vpc[Private VPC]
    ec2
    gateway
    lambda

  group vpc2[Public VPC]
    serv1
    serv2

  group aws[AWS]
    vpc %% group in group
    vpc2 %% group in group
    s3

Some questions on layout

How would multiple nodes like this on same side be handled?
Can we make the LRTB arrow modifiers optional and let mermaid do automatic layout? I'm not saying we should not have the modifiers, but follow the mermaid ethos of just write code, don't worry about the layout.

Comments on code

Please use ImperativeState and avoid using multiple global variables for state. You can refer sequence diagram as an example.
Avoid using export { functionName }, and instead add export to the function declaration.
Avoid mixing camelCase and snake_case

Arnab-Developer commented 1 week ago

When can we expect this feature to be generally available to all users?

NicolasNewman commented 1 day ago

When can we expect this feature to be generally available to all users?

I've been quite busy the past month so I unfortunately haven't had the time to finish implementing this feature. I'm hoping to be able to continue development and have it ready for review within the next 2 weeks but there is no guarantee. After that it needs to go through review and I don't have enough knowledge of the process to provide a timeline of that. You can view a more updated discussion on the draft for this feature.

mermaid-js / mermaid