search

event-catalog / eventcatalog

An open source documentation tool to bring discoverability to your event-driven architectures
https://eventcatalog.dev
MIT License
1.64k stars 135 forks source link

EventCatalog v2 - What's new, thoughts and plan #505

Closed boyney123 closed 1 month ago

boyney123 commented 4 months ago

Hello folks,

EventCatalog started about 2 years ago as a random idea focused on one thing, helping people document their event-driven architectures.

Over the past couple of years, the project has naturally grown with 13k catalogs created, 44 contributors, 600 discord members and 1.3k stars. More and more folks are looking to EventCatalog to help them document their EDA, great!

This community has grown and it's been great to watch, and just want to thank you all for using EventCatalog ♥️

As I continue to evangelize EDA, governance is becoming more and more important and area of concern for many folks. The ability to discover events, document them as well as services, domains, commands and much more!

I have had time to think about EventCatalog, what we have learnt over the past couple of years, and how some refactoring and new features can help move this project further!

Here are some initial thoughts, your feedback is welcome!

Moving to a container based application

Many folks are using EventCatalog as a static site generator, and many folks have asked in the past about key features that would require some kind of server to be there. Also many folks are trying to host EventCatalog (statically) on their own infustructure or in the cloud, for many this seems to be a challange based on the routing cofiguration in the app etc.

I want to make it as easy as possible for people to host EventCatalog whilst also providing some new feature support (integration with their brokers, API support, single sign on etc), looking into the future of features for EventCatalog, I believe it makes sense to focus on container support, allowing anyone to host EventCatalog anywhere.

EventCatalog will remain to be focused on markdown generation and ease of use, it just means many more options are open to for us as a community to support.

Keeping things simple

EventCatalog has evolved over the past 2 years, with support for domains and services, 3D node graphs and much more. Adding support for things like 3D node graphs is cool, but requires third party dependencies that may not be required. So I want to bring EventCatalog back to basics this means:

Extra features

There are many more features that EventCatalog can support, and with the move to a server based application (using containers), this opens the door to some more interesting features and things to help you document your EDA. But some initial features will be:

Long term feature ideas

What this means for you?

The project will still be focused on markdown files that power your catalog. Moving to V2 will require some structure changes to your catalog, but hopefullly migration docs and tools will be there to support you. Nothing too bad (hopefully!).

EventCatalog has proven to add so much value to people around the world, and in ways I feel it's just getting started. Hopefully you are on board with this vision and changes, and I would love your feedback below.

Governance is a big problem, but I believe we can fix this, work together and build something cool along the way! ♥️

donaldpipowitch commented 4 months ago

static site generation will still be possible, correct? Container deployment is an optional feature?

XaaXaaX commented 4 months ago

Hey @boyney123 I think in cases, the domains folder scaffolding was useful, when a domain has many services representing too many events, this will be hard with a flat way , but having markdown side by side for each service seems ok

boyney123 commented 4 months ago

static site generation will still be possible, correct? Container deployment is an optional feature?

Hey @donaldpipowitch need to think about that, I think with some server support much more could be done to help govern things. How much would this effect you?

Hey @boyney123 I think in cases, the domains folder scaffolding was useful, when a domain has many services representing too many events, this will be hard with a flat way , but having markdown side by side for each service seems ok

Hey @XaaXaaX yeh should be able to do whatever you want to do. So if you want all events in /events thats fine, if you want events in domains like /domains/mydomain/events/* this should fine. The glob system will fetch all types regardless where they are as long as they would be in an events folder. That's my thinking anyway. So quite flexible!

debae commented 4 months ago

Would be great to have support for multiple environments, I guess most people or companies are having for example DEV / TEST / ... / PROD as environments. Would be great if you could see which versions of events / schema's are active on which environment.

schrepfler commented 4 months ago

Backstage plugin/easy way to integrate with IDPs.

thim81 commented 4 months ago

A REST API to manage events, services, ... would open-up extra integration options.

boyney123 commented 4 months ago

Would be great to have support for multiple environments, I guess most people or companies are having for example DEV / TEST / ... / PROD as environments. Would be great if you could see which versions of events / schema's are active on which environment.

Yeah need to figure this out, but 100%.

Backstage plugin/easy way to integrate with IDPs.

Yeah be interested to connect with people that want to integrate with Backstage, EC plugin here, not done much with backstage, do people see EC working along side it or backstage replacing EC?

A REST API to manage events, services, ... would open-up extra integration options.

Yes! Any ideas on the kind of things you would want to do? Things that come to my mind are download schemas, upload new versions of schemas etc,

thim81 commented 4 months ago

For the REST API, it would be useful to provide CRUD operations for the users, domains, services, and all the aspects of events. For events specifically:

People could then build their integration/service/pipeline (next to the plugins), in case they have special transformation needs or certain schema registry tool that they want to combine and centralize in EventCatalog.

kernwig commented 4 months ago

Right now EventCatalog is an easy sell to clients and development teams. They see the value, the learning curve is low, and tossing it up in an S3 bucket and CloudFront is simple and generally costs nothing. (Pennies at most.)

I don't want to lose that. There are hosted platform that cost $10/user/month; I don't even try to justify those to clients.

donaldpipowitch commented 4 months ago

Hey @donaldpipowitch need to think about that, I think with some server support much more could be done to help govern things. How much would this effect you?

It would be awesome, if static export would still be possible. Similar reasons as @kernwig said.

Would be great to have support for multiple environments.

We create an event catalog instance per stage. This seems to be a good approach - especially if you would work in a company where new stages can be dynamically created and removed.

A REST API to manage events, services, ... would open-up extra integration options.

If event catalog wants to support cases like this an approach similar to next-auth might make sense. You create your own Next application and then you just mount any page/api routes offered by event catalog that you need/want.

I personally would like to see better support of existing features in v2. Like getting mermaid support in domains (https://github.com/boyney123/eventcatalog/pull/407) or custom React component usage (similar to how it's done in Storybook).

andy-g commented 4 months ago

@boyney123 thanks so much for all your work on EventCatalog.

One of the design aspects that complicates things for us in the current version is the fact that if you want to add consumers you must do this in the Event markdown files. Personally I like the idea that an Event is owned by a producing service, but as in most event systems, it doesn't really care who consumes it. The consumer itself needs to be aware of the event though. In light of that, would it be possible to define the events a consumer is subscribed to in the consuming service itself?

We were looking into the idea of allowing each event producing service to define their own EventCatalog service and the events they produce in their git repo, and similarly allow each consuming service to define their service definition in their own repo (which would ideally include references to the events it consumes). Then in our CI/CD we could pick these configurations up as a artefacts and push it to our central EventCatalog.

If for example a consumer is updated to consume another event, we only need to change the consumer configuration in its repo, this will get pushed to the central EventBridge repo, which can then add this service as a new consumer of the event, without needing to update the event configuration itself - which should really be owned by the event producer.

I hope that makes sense.

black-snow commented 2 months ago

I second what @andy-g said. Producers should be oblivious to who consumes the events. Hence, the docs should reflect that and adding another consumer shouldn't touch the producer docs.

Also +100 for an API to use it as a schema registry. It basically is a sort of registry atm but without an API. Was expecting to be able to grab the schemas from a corresponding endpoint.

IsmaelMartinez commented 1 month ago

Hi @boyney123 , is there any way to get early access to this? At the company that I work, we would be more than happy to help evaluate EventCatalog if that is of any use.

Regarding other integrations that might be of interest, integration with Pact will be great. Also happy to help on that area if at all possible.

boyney123 commented 1 month ago

Hey @IsmaelMartinez

Yeah sure you can start testing it here https://discordapp.com/channels/918092420338569216/1248605310520922162/1253724516484517979 (message on Discord). Let me know if you don't have access.

Yeah Pact is an interesting usecase, maybe if you have some ideas for it you can raise an issue/feature request explaining your thoughts and ideas would be useful.

IsmaelMartinez commented 1 month ago

Hi @boyney123 , I don't seem to have access. My username in discord seems to be ismael786655 in case that helps. Thanks!

Pakisan commented 1 month ago

Hey @IsmaelMartinez

Yeah sure you can start testing it here https://discordapp.com/channels/918092420338569216/1248605310520922162/1253724516484517979 (message on Discord). Let me know if you don't have access.

Yeah Pact is an interesting usecase, maybe if you have some ideas for it you can raise an issue/feature request explaining your thoughts and ideas would be useful.

Hi! Looks like link is broken. Can you share new one to test v2?

boyney123 commented 1 month ago

Hey @Pakisan try this https://discordapp.com/channels/918092420338569216/1248605310520922162

Pakisan commented 1 month ago

Hey @Pakisan try this https://discordapp.com/channels/918092420338569216/1248605310520922162

Nope, shows empty state that tell that I don't have access to text channels or none are available on given server

boyney123 commented 1 month ago

Thank you all for your feedback and patience

EventCatalog v2 is planned for release this week, will close this issue :)