Closed derberg closed 2 years ago
I am trying to get myself familiar with parser-js
for now, since I believe that would be the main tool we would be using and building this idea on top of. Would try to resolve some good first issues and get familiar with the codebase.
@derberg Also can you please point out to some already implemented async-api
collection so that I can test this idea on those files manually for PoC.
@arjungarg07 we do not have anything out of the box. These either would have to be created from scratch or you can try out https://solace.com/try-it-now/ that gives an event portal where you can design a system in a kind of design tool. You first create objects, connect them to explain how they are related and then you can export AsyncAPI files from there. Decide on your own what is better, play with some tool that already supports AsyncAPI so you learn how tooling can support the spec, or you learn the spec by just creating few files for some fake system, and try to "connect" them (I can of course help).
you can also wait few days and there is Alvaro that works on a blog post on a real system based on AsyncAPI and he has AsyncAPI documents there, he just needs to do some fixes to that. I can keep you posted, especially that you will anyway be busy with the onboarding tasks from the parser
Yeah, thanks for sharing the design tool. I would try to build AsyncAPI files as per the use cases described in the idea and play with it to draw some relations manually first. Sure, I will do some onboarding tasks from the parser till Alvaro has those documents ready. Thanks.
@arjungarg07 Alvaro's blog post is out https://www.asyncapi.com/blog/building-async-flight-notification-service and the project where he has AsyncAPI files is here https://github.com/amadeus4dev/amadeus-async-flight-status
@derberg I build a prototype in which I tried to identify relations between the AsyncAPI docs on the Rideshare microservice (provided by solace) and model the output as shown in the image. This is just a basic structure and can be changed according to the requirements. Link to the source code: https://github.com/arjungarg07/relation-finder-prototype
Rideshare Graph View
Basic output structure
Please give your feedback. Thanks๐
@derberg I did some research about what more functionalities we can provide by the library:
service
subscribe/publish on channel
which would help in visualizing the architecture if any of the puzzles is removed or added.Please share your opinion on this and also throw some light on what more functionality we can provide by this library. Thanks๐
Loving it so far mate, I thought about this idea for very long but never had time to code it ๐ awesome to see what you're doing here and ideas that pop up in your head, really ๐๐ผ
Do not worry about the structure much because as you noted, this can be modified once the actuall work on the library starts. And the structure will change for sure, will be more complex, with more data. I'm more interested at the moment what you think about making library extendable in a way that someone can provide a plugin for the library, so the output is not our custom Map but class diagram syntax from mermaid.
Lemme just list all comments 1by1
Rideshare
files is the only sample project to use here. These files do not have servers
, which at the end means that there is no validation if given channel from App A and the same channel from App B are connecting to the same server or not. So what I mean is that we can discover relation that is not necessarily there. Now, this doesn't mean you did something wrong, because servers
are not mandatory in AsyncAPI, we should definitely support a flow without servers. Just the main flow we need to support is the one with servers
matching across files (so I guess the example from Alvaro). So once servers
are there, do matching, if not, don'tFilters to highlight relations that match a criteria given by the user
I thing this would be a great thing. As I understand this would be besically a set of helpers that someone could use in frontend. So instead of writing custom code to traverse through found relations, frontend dev can just use a helper to get relations only of a given application, or only relations where given message is in play. That would be awesome, much easier then to do some "click to narrow view" magic ๐
option to add/remove service subscribe/publish on channel which would help in visualizing the architecture if any of the puzzles is removed or added.
you mean in the UI already? we are talking about library here. Of course it is good if some features are driven by how we see it in the UI, just need to know what you mean exactly and what would be the purpose of add
for example? what I add
could later be saved to a file or just returned as string?
option to provide output format as grouped events which are common to two services
I'm not sure what you mean here? oh wait, events are messages for you I guess, right? so you might mean here what I wrote above ๐ if yes, then yes, but it is not option on output but simply another function available imho
showcasing all the events referencing the same Schema
you mean not focusing on channels but on messages? find where they are used and show what schemas they have? Like it, it will for sure help to check if there are some inconsistencies maybe
Loving it so far mate, I thought about this idea for very long but never had time to code it ๐ awesome to see what you're doing here and ideas that pop up in your head, really ๐๐ผ
@derberg Thank you for the kind words, all thanks to you๐
I'm more interested at the moment what you think about making library extendable in a way that someone can provide a plugin for the library,
I'm not sure what you mean here? Did you mean that someone can use this library as a plugin I guess, right?
so the output is not our custom Map but class diagram syntax from mermaid.
Yeah, we can provide the output in a way that can match the syntax from mermaid, also we can search some more related tools which use the same syntax as mermaid from this collection: awesome-d3
- Just the main flow we need to support is the one with
servers
matching across files (so I guess the example from Alvaro). So onceservers
are there, do matching, if not, don't
Got it, I will look into it.
you mean in the UI already? we are talking about library here. Of course it is good if some features are driven by how we see it in the UI, just need to know what you mean exactly and what would be the purpose of
add
for example? what Iadd
could later be saved to a file or just returned as string?
It would also be just another helper function, which can be used by developers to visualize the change in the architecture.
Like: add(<channel name>, <subsrcibe/publish>, <message name>)
, can be used to detect how a change in one microservice is going to impact others.
you mean not focusing on channels but on messages? find where they are used and show what schemas they have? Like it, it will for sure help to check if there are some inconsistencies maybe
Yeah!
@derberg Shall we solidify all these functionalities? also please tell me if we can provide any more features by this library as per your opinion so that I could jot them down in my official proposal with the workflow of how I will be implementing them๐
I'm not sure what you mean here? Did you mean that someone can use this library as a plugin I guess, right?
I think it would be cool to make it possible to provide plugins for this library. So by default, it outputs some custom format, but one can also install an additional plugin and use it with the library to get mermaid (as an example) format. Of course in first version we do not have to have pluggability, just a thought. We can also have a switch in first version where you can just pass option and say that instead of default you want mermaid.
awesome-d3
definitely, mermaid is on my head only because you get UI out of the box, but other formats would be great too
It would also be just another helper function, which can be used by developers to visualize the change in the architecture. Like: add(
, <subsrcibe/publish>, ), can be used to detect how a change in one microservice is going to impact others.
ok, so you import 4 files, get a visualization of architecture and then can add new element to the "visualization" to see how it will look like if some new service will publish to some existing channel, good, makes sense. Just keep in mind that if we make it possible to add something to the structure, we should be able to later export added element into AsyncAPI - just making it clear so you do not extend the scope of the task to much, so you have enough time to finish it. We can make add
as a nice to have feature
I think all is clear and you can start drafting a proposal
I think it would be cool to make it possible to provide plugins for this library. So by default, it outputs some custom format, but one can also install an additional plugin and use it with the library to get mermaid (as an example) format. Of course in first version we do not have to have pluggability, just a thought. We can also have a switch in first version where you can just pass option and say that instead of default you want mermaid.
Got it๐๐ฝ
ok, so you import 4 files, get a visualization of architecture and then can add new element to the "visualization" to see how it will look like if some new service will publish to some existing channel, good, makes sense. Just keep in mind that if we make it possible to add something to the structure, we should be able to later export added element into AsyncAPI - just making it clear so you do not extend the scope of the task to much, so you have enough time to finish it. We can make
add
as a nice to have feature
Sure, will keep in mind, it could be added if the project finishes before time.
I think all is clear and you can start drafting a proposal
Sure. Thanks!
I'm also looking for some UI to discover relationships in a "mesh" of services. Great to see that you made something so quickly. I'm looking for quite some time for something to document event driven systems in a better way than linked confluence pages ;-)
In my mind there is a web UI+API where you can register asyncAPI files and then you can browse all these in a format similar to the docs generated in https://playground.asyncapi.io/?load=https://raw.githubusercontent.com/asyncapi/asyncapi/master/examples/2.0.0/simple.yml but with links between services that share events/messages.
I'm not very fond of JS, I'm more a Java guy, so my first instinct is to create a nice Spring boot app to do the heavy lifting and a small vue.js UI to display everything. But that takes time and effort and when something else is developed somewhere, I could use the time more productively ;)
@arjungarg07 repo ready and as you can see, the initial issue is also transfered
@derberg Thanks, will soon be opening issues for further discussions ๐
Some discussed points:
Map
as a default output and would be configuring it to obtain output following mermaid syntax.
- it should show that App A has a link with App B through some message, but the relation goes through a given server
@derberg Since the Apps relation goes through a given server/broker, in the case of multiple servers described in a single AsyncAPI file, which one do we have to consider?
@arjungarg07 Hello! At the moment we cannot define in spec which server exposes some channels, but currently we have PR for that https://github.com/asyncapi/spec/pull/531 so for your question: you should consider all servers, which will be logical, it means: the application A can be hosted by 2 servers then you are considering 2 servers through which application B can communicate.
cool tool to consider https://reactflow.dev/ I admit I have not much knowledge about it, but assume that whatever is used to render flows, probably accepts some specific format, and this is the format that could be an output here
@magicmatatjahu Thanks for clearing it out.
So we would distinguish relations between files on 3 levels.
1st level
: server URL
2nd level
: server protocol
3rd level
: channels to which services subscribed or publishing
We could club 1st and 2nd levels too
Please let me know your thoughts on this ๐ค
cool tool to consider https://reactflow.dev/
@derberg yeah! i checked it out yesterday and it's super cool, would love to experiment with it but first focussing on MVP ๐
but first focussing on MVP
๐๐ผ
We could club 1st and 2nd levels too
same URL with different protocols would be super unusual so you can definitely assume this is always referring to the same server
@arjungarg07 Hello! At the moment we cannot define in spec which server exposes some channels, but currently we have PR for that asyncapi/spec#531 so for your question: you should consider all servers, which will be logical, it means: the application A can be hosted by 2 servers then you are considering 2 servers through which application B can communicate.
I'd like to throw into the ring that structurewise it doesn't really matter is a service is running in 1, 2 or 200 instances, it shouldn't appear with a cardinality of 200 in the flow. The main point of the message approach is that you don't care on the sender side how many workers/consumers there are. It may be an important fact for the consumer to be for example single instanced, or it is stateful, but that has IMHO nothing to do with the message flow documentation.
@pcornelissen Thanks for your insights :) I almost agree with you, but there is some situation when pinned channel only for particular server can change the rest of flow. We can use the relationship (as a user/client) to check which flow will be "created" when for example I as a user/client will send message to create a new order in e-shop. Sending the order to this same channel, but in two different server can change flow. This situation is rare, but still possible. Of course, we should in first implementation go with simple solution which will cover almost cases, and then go with more complex and rare.
If we don't understand each other, it may be because I look at the current problem (app relationship) from the perspective of the user/dev who wants to send a message to a given channel and know what operations will be performed after sending the message/event, such as creating an order, adding loyalty points etc. on the example of an e-shop.
@arjungarg07 Please don't focus on first weeks on output shape (also using shape of ReactFlow isn't a good idea, we should be agnostic here). We can change it later, because currently most important part is making, as you wrote, some MVP :)
same URL with different protocols would be super unusual so you can definitely assume this is always referring to the same server
Yeah! but I would like to play it safe and not make any assumptions๐
@arjungarg07 Please don't focus on first weeks on output shape (also using shape of ReactFlow isn't a good idea, we should be agnostic here). We can change it later, because currently most important part is making, as you wrote, some MVP :)
Sure thing.
@derberg @magicmatatjahu @jonaslagoni I ran the script for on Alvaro's flight service with a slight change of adding one more server in one of the spec. Generating default output as:
Map(2) {
'mqtt://localhost:1883,mqtt' => Map(2) {
'flight/update' => {
sub: [ 'Flight Monitor Service' ],
pub: [ 'Flight Notifier Service' ]
},
'flight/queue' => {
sub: [ 'Flight Subscriber Service' ],
pub: [ 'Flight Notifier Service', 'Flight Monitor Service' ]
}
},
'mqtt://163.124.11,mqtt' => Map(2) {
'flight/update' => { sub: [ 'Flight Monitor Service' ], pub: [] },
'flight/queue' => { sub: [], pub: [ 'Flight Monitor Service' ] }
}
}
So the mermaid diagram of this output should be like this:
Any feedbacks are much appreciated.
@arjungarg07 I love it! Really :) As I see everything is good, 'publish' operation go to channel, from channel you link to service/app's 'subscribe' operation. At the moment is good, but honestly it's a hard to understand in first look what is connecting to what, I mean, you should look on diagram and see that you can publish something to (eg) FlightNotifierService when you will publish to flight/update
channel and etc. It's not a problem of your code, but probably with our "description" of publish/subscribe in spec, but overall, great job! :) What do you want to do next?
but honestly it's a hard to understand in first look what is connecting to what
Yup! I mean the whole thing is to visualize the relations and the first look at this image feels somewhat complex :( I would try to add the config theme in the script to visualize it much easily than now
What do you want to do next?
will start coding the script for generating mermaid diagrams syntax asap ๐
Do we have to add metadata information in the default output itself or have a separate API for it?
@arjungarg07 Do you mean metadata like payload's schema of message etc? You operate on parsed spec, currently you save sub/pub only by name to array for appropriate channel name, to save metadata you can change the array to Map, so you have:
'flight/update' => {
sub: [ 'Flight Monitor Service' ],
pub: [ 'Flight Notifier Service' ]
},
but you can change it to Map<string, ChannelMetadata>
'flight/update' => {
sub => {
'Flight Monitor Service' => Channel metadata,
'Flight Notifier Service' => Channel metadata,
}
}
Of course you can save as Channel metadata
whole object of appropriate channel of given app as reference :)
... in the default output itself or have a separate API for it?
About separate API we can think in the next weeks, at the moment we can add metadata by default to default output/syntax or add option to enable adding metadata, which will be by default set to true.
Do you mean metadata like payload's schema of message etc?
Yeah, message and schemas too.
About separate API we can think in the next weeks, at the moment we can add metadata by default to default output/syntax or add option to enable adding metadata, which will be by default set to
true.
Sure. I will then update the metrics ๐๐ฝ In the next weeks maybe we could imply these metrics as class models and extend the APis from there itself.
@derberg @jonaslagoni @magicmatatjahu Consider we need to get the metadata information like message and schema for each channel on this spec.
channels:
flight/update:
description: |
Provides updates from a subscribed flight
subscribe:
summary: Inform about the status of a subscribed flight
message:
$ref: '#/components/messages/flightStatus'
flight/queue:
description: |
Queues a flight in order to retrieve status
publish:
summary: Subscribe about the status of a given flight
message:
$ref: '#/components/messages/flightQueue'
I can't find any API available in parser-js
through which I can get the served message information on a particular channel, like
const messageInfo = doc.channel('flight/update').message();
Same for schemas. Although we do have APIs doc.allMessages()
and doc.allMessages()
but it doesn't give info about on which channel it is being served.
Please correct me if I am missing something.
@arjungarg07 Because message is in the operation
item of channel - publish
or subscribe
. You must use in your case doc.channel('flight/update').subscribe().message();
What about saving all channel's metadata (with description etc) as metadata - not only message - in relation?
EDIT: We have also corresponding hasSubscribe()
and hasPublish()
functions to check if given operation is available in channel. There is a full API of parser-js https://github.com/asyncapi/parser-js/blob/master/API.md
Because message is in the
operation
item of channel
Ohhk, this I missed :) Thanks!
What about saving all channel's metadata (with description etc) as metadata - not only message - in relation?
TBH, I didn't discuss it with Luksaz about what all to save in metadata coz i thought of asking it along the development phase. I would suggest to consider all channel's metadata as you said.
EDIT: Also, do we need to have payload as it is for metadata or just some specific parts of it?
EDIT: We have also corresponding
hasSubscribe()
andhasPublish()
functions to check if given operation is available in channel. There is a full API of parser-js https://github.com/asyncapi/parser-js/blob/master/API.md
Yeah, I have been checking from here itself :)
EDIT: Also, what do we need to have payload as it is for metadata or just some specific parts of it?
Payload is a shape of (usually) json which should be sent to channel or will be subscribe from channel, so payload is a complete whole that cannot be divided into smaller parts :) If you think about different problem, please describe it in different way, because probably we don't understand each other :)
@magicmatatjahu Yeah, what I mean to say is that since we have great support for traversing schemas with support for circular references, when we will extract the metadata information from let's say message payload do we have to consider some specific fields from it or just include the whole payload itself like this:
message: {
payload: {
type: 'object',
'$id': 'PlayerItemPickupPayload',
additionalProperties: false,
properties: {
pickupTimestamp: {
type: 'string',
format: 'date-time',
description: 'The timestamp the item was picked up',
'x-parser-schema-id': '<anonymous-schema-4>'
}
}
},
'x-parser-original-schema-format': 'application/vnd.aai.asyncapi;version=2.0.0',
'x-parser-original-payload': {
type: 'object',
'$id': 'PlayerItemPickupPayload',
additionalProperties: false,
properties: {
pickupTimestamp: {
type: 'string',
format: 'date-time',
description: 'The timestamp the item was picked up'
}
}
},
schemaFormat: 'application/vnd.aai.asyncapi;version=2.0.0',
'x-parser-message-parsed': true,
'x-parser-message-name': '<anonymous-message-1>'
}
}
Considering the whole payload would be a lot.
Also, I was thinking of wtiting tests after implementing the metadata extraction funcitonailty. What do you think? ๐ค
@arjungarg07 I see your problem but you also put example on which I can describe my thinking :)
You can treat only payload
as metadata, but we must also know that dev can put inside message object specific information for given protocol, this same as in given server, so for me this is also very important metadata which we should show to user. Also channel can have specific protocol information in bindings
object https://github.com/asyncapi/spec/blob/master/spec/asyncapi.md#channelBindingsObject You can have protocol info in message
, operation
(publish/subscribe), channel
and server
object. If we only consider message and it's value, someone can say that headers
are also very important as metadata. I would like to say that whole data in a channel is very important metadata, so I think that treating whole channel as metadata would be the best option for our case, so I will remind you in which way I would like to do that:
'flight/update' => {
sub => {
'Flight Monitor Service' => Channel metadata,
'Flight Notifier Service' => Channel metadata,
}
}
and then you don't focus which part of message is important, because you treat whole channel as metadata.
Also:
Yeah, what I mean to say is that since we have great support for traversing schemas with support for circular references
No, we don't, we only handle simple circular refs :) I created issue for that https://github.com/asyncapi/parser-js/issues/293 so please have it in mind when you will use traverse
function :)
What do you think about treating channel as metadata. It's easiest solution for our problem and will give us as many possibilities for the future.
No, we don't, we only handle simple circular refs :) I created issue for that asyncapi/parser-js#293 so please have it in mind when you will use
traverse
function :)
Yeah, I just tried to iterate over it and it caused recursive stackoverflow๐
What do you think about treating channel as metadata. It's easiest solution for our problem and will give us as many possibilities for the future
I agree with the explanation that the metadata's importance could change for every dev and treating the channel as metadata itself would be the best possible solution ๐๐ฝ
'flight/update' => {
sub => {
'Flight Monitor Service' => Channel metadata,
'Flight Notifier Service' => Channel metadata,
}
}
Here flight/update
is the channel and since channel's metadata would be same for every service(Flight Monitor Service
,Flight Notifier Service
, etc...), won't be mapping the same metadata with every service, instead we should have it like this:
<channelName> => {
sub: [ ],
pub: [ ],
metadata: {}
}
EDIT: I get it, you are right! channel's metadata could change for every service, sorry :)
@magicmatatjahu Also, I think treating channel as metadata would have a huge ton of info to put in mermaid class diagram. How we would deal with that ๐ค
Also, I think treating channel as metadata would have a huge ton of info to put in mermaid class diagram. How we would deal with that ๐ค
Yes, you're right about that, but partially :) For mermaid
you can only use payload
, headers
and bindings
metadata from channel, for another syntax you can only use payload
etc. Of course, It will be "huge" data for default
syntax, but we can read/reuse/change it for another syntaxes. You should think about it as a bridge to another syntax.
@arjungarg07 they all feel kinda the same tbh, I mean the same principle behind as in case or mermaid and plantUML. As you wrote, I think more important are tests now, and a way to test all those outputs not only from unit tests point of view but also integration tests.
And then something more beautiful........ React Flow โค๏ธ
@derberg @magicmatatjahu I was playing a bit with React Flow ๐
and it turns out that they don't have any playground online where we can just paste the syntax and render the diagram. Then I thought of generating an HTML page coz I don't want to set up a whole react project just for a simple diagram, but for that(HTML file) I am not able to find and cdn/browser package for react-flow-renderer
so that I can include it in that HTML file and code it further. I think if I could get this cdn package which I could include in HTML code, we could use puppeteer to automate opening the HTML page and taking a screenshot of the page, and save it in the system.
Any workarounds/suggestions for this?
function for react flow should just generate the relations, the code responsible to show them, not entire website. This is a library, I integrate it however I want in my system, generate relations and enable in my HTML. So basically the goal is to focus on react flow code that shows relations between nodes, the HTML to present it can just be an example, part of test or a demo. Does it make sense?
Yeah, so this is the simplest flow for ReactFlow.
import React from 'react';
import ReactFlow from 'react-flow-renderer';
const elements = [
{ id: '1', type: 'input', data: { label: 'Node 1' }, position: { x: 250, y: 5 } },
// you can also pass a React Node as a label
{ id: '2', data: { label: <div>Node 2</div> }, position: { x: 100, y: 100 } },
{ id: 'e1-2', source: '1', target: '2', animated: true },
];
export default () => <ReactFlow elements={elements} />;
So our library function for react flow should just generate the elements
array which could be directly passed by the user, right?
@arjungarg07 Yes, elements should be the exact output from function.
@arjungarg07 Important thing: If you will use inside output the VDOM nodes, like in your example <div>Node 2</div>
you should write them with React.createElement()
function, not as JSX, because if you use the JSX, you need then transpile your package by babel.
This issue has been automatically marked as stale because it has not had recent activity :sleeping: It will be closed in 60 days if no further activity occurs. To unstale this issue, add a comment with detailed explanation. Thank you for your contributions :heart:
@arjungarg07 Can we close the issue?
we can definitely close it, purpose of this issue was to discuss initial idea and the scope.
Reason/Context
Would be awesome to build a library that as the input gets an array of AsyncAPI documents and analyzes them to get information about Applications described by those files:
As an output, we should get a format that can be used to generate a diagram of relations between applications with additional metadata information about them (message, schemas).
Use case:
Visualization of the architecture and understanding what happens with the system if one of the puzzles is removed. Later on, we could use this library in AsyncAPI CLI and also the AsyncAPI Studio.
Required knowledge: There is no need to have any experience in event-driven architecture. You do not even need to know AsyncAPI spec too much. It is a pure coding task where you will need to find matching patterns between different documents and return this information. This should be an independent library that can be later reused in AsyncAPI CLI or AsyncAPI Studio, or by others in their tools.
Description