Add Spring Integration graph XML

[dinizssa]

Missing support for Spring Integration message flow built in XML.

[martinlippert]

To clarify: this is about a graphical representation of the message flow that got defined in Spring Integration XML files, right?

Are you looking for a graphical view only? Or do you desperately need editing features on the graphical side, too? I am asking this because having a graphical representation with full editing capabilities is a much much bigger effort and would require a whole lot more time than "just" the graphical representation of the flow defined in XML.

[dinizssa]

That's correct. Just the graphical representation of the message flow that got defined in Spring Integration XML files would be good enough. No editing capabilities from the graphical side is required, at least my team and I prefer editing from the XML itself.

[martinlippert]

To distinguish this feature request from the one in #213, this is all about reading and "understanding" the Spring Integration XML files and generating a graphical representation from that. This is NOT related to the GraphController that is able to serve the necessary data from a running SI application and render that graphically. That is done in #213.

[martinlippert]

There are a number of concerns about going into the space of parsing the XML data and interpreting it from the tooling side in order to create a nicely rendered graph from it. Therefore, before we dive into this, we should discuss the use cases on the one side and the technical limitations on the other side first.

Therefore, I would like to invite @dinizssa (and others) to provide some concrete sample projects for this (in order to talk about concrete use cases) and describe what you would like to gain from a graphical representation (and how that should roughly look like).

On the other side, I would like to invite @garyrussell to tell us more about the technical limitations that come with this approach of reading and "interpreting" the Spring Integration XML files to turn them into a graphical representation. Concrete examples and/or code snippets about be welcome here, too.

[dinizssa]

My team and I are still using the XML graphical representation during the design of new integration projects. That also help us on discussing the proposed message flow with other parties involved. Meaning that we start using the graphical representation before any coding. That said, whenever the coding starts we use the same XML message flow as per discussed before. That's the point in my case. In my humble opinion, the "abuse" of Builder Pattern compromise the readability of the message flow and make the code even more complex to change later. Thanks for looking into this.

[martinlippert]

@dinizssa Thanks for the additional details. Would you be able to share some sample projects? That would be awesome.

@garyrussell Anything you would recommend for this use case? Any limitations that you are aware of in this situation? Looking forward to your input here.

[dinizssa]

Unfortunately I'm not allowed to share the projects that I have, but you can find some at this location https://github.com/spring-projects/spring-integration-samples/tree/master/applications/cafe.

[garyrussell]

Hi @martinlippert , sorry I didn't reply earlier.

The biggest problem I see with the XML parser is just that (only XML - it can't render EIP annotated methods or DSL, which is becoming increasingly popular).

However, I understand that XML is still very popular for SI applications and the graph was quite useful for small applications (IIRC, it could only render a single file and didn't deal with <import/>s - but I could be wrong).

For these reasons, we decided to take the approach of rendering the SI app at runtime because it gets past all these limitations. The IntegrationGraphServer builds a runtime graph of the application components which can be rendered as JSON. It works with all configuration mechanisms, including XML.

As you know, there is ongoing work to further improve that approach by enhancing the graph to provide for help for UI rendering.

@dinizssa There is an example for an early prototype shown in the README for this sample.

[martinlippert]

@dinizssa Do I understand the use case correctly: You create Spring Integration XML config files in more or less empty projects and visualize them using the STS3 SI graph visualization. Then you discuss them, talk about the graphs, change the XML definitions while doing that, etc. And once you are "done" with this, you go ahead and implement the project which should use that SI XML config file. Is that correct?

Since we are already working on a graph visualization for the SI data from the running apps, I am wondering what would be necessary to re-use that technology for the above described use case - so what would be necessary to fire up an SI app that serves the "real" data using the XML configs that @dinizssa works on in the "design phase". Maybe we could so something like that from inside of the IDE (under the hood). Would allow us to nicely re-use the graph visualization for the "live data" without having another parser for the SI XML config files. What do you think, @garyrussell ? Would that be possible somehow?

@dinizssa The examples that you referenced above are ready-to-use examples that I could start up and see the graphical visualization for. How would your projects differ from that?

[dinizssa]

That's correct. Regarding the examples provided, there is no real difference, it's the same idea but the message flows will differ of course (e.g. read from JMS queue, validations, filters, transformations, splits, aggregations, etc and send out the payload towards downstream applications - in many ways - and so on).

[martinlippert]

@dinizssa So looking at the sample it sounds like you might be able to use the new graphical visualization for exactly your purpose by hacking the XML config, quickly starting up the simple plain app with that XML config, and see the graphical representation (and discuss it). And whenever you want to change the config, edit the XML, restart the app, and refresh the browser view of the graph? What do you think?

[dinizssa]

It is not the same thing because depending on the endpoint a valid Java class (bean) will be required otherwise the application will not even start. If this is too much trouble we can live without this feature (at least very old users like myself). I'm sure that Spring Integration lost visibility and new users willing to experiment it due to this - I might be wrong. Again, thanks for looking into this.

[garyrussell]

@dinizssa You can achieve effectively the same with some small modifications.

Assuming you have <service-activator ref="someBean" method="foo"/>, you indeed would have to add an empty implementation of someBean.foo() for the app to start.

However, you could instead use <service-activator expression="dummy" />; the app will start and the graph will render at runtime. Obviously you won't be able to send any messages through, but that's not important.

Furthermore you can set auto-startup="false" on inbound endpoints so they don't attempt to connect to whatever external message source they are configured to talk to.

When you are happy with the design, you can then reconfigure the XML as needed.

[dinizssa]

I'm happy to apply this workaround too. Cheers!

[martinlippert]

@dinizssa So the workaround that Gary described in combination with the live graph visualizer sounds like a viable solution to you moving forward? I would love to hear about your experiences using this approach so see what we could improve / make easier. Do you think you will have a few cycles to give this a try and report back? That would be awesome!!!

[dinizssa]

Sounds like a plan. Let me play around a bit with the runtime and report back to you.

[martinlippert]

@dinizssa Awesome, looking forward to hearing about your experiences.

[BoykoAlex]

@dinizssa If you're trying the suggested workaround for the SI Graph and would like to see the Live SI Graph visuals we are working on the prototype of the SI graph viewer. Below are some instructions how launch the SI Viewer app, configure your SI app to serve the graph json and use SI Viewer app to see the graph frpm the live running SI app.

Launch SI Viewer

1. Run ./mvnw clean install
2. Launch the attached JAR: java -jar sprotty-si-view-0.0.1-SNAPSHOT-exec.jar
3. Navigate to https://localhost:8877
4. Sample graph shown (from a json file)

Enable Live Integration Graph Endpoint for SI App

Spring Integration application needs to have @EnableIntegrationGraphController(allowedOrigins = "*") somewhere on a class annotated with @Configuration annotation (or @SpringBootApplication). The annotation is available if the project depends on spring-integration-http and requires Spring Web or Spring Reactive Web starter. If Maven is used then the following should be in the POM:

    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-web</artifactId>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>org.springframework.integration</groupId>
      <artifactId>spring-integration-http</artifactId>
      <scope>compile</scope>
    </dependency>

Now when the SI app is started https://localhost:8080/integration endpoint should be available and respond with the graph in JSON format.

See more information about enabling and using @EnableIntegrationGraphController here: https://github.com/spring-projects/spring-integration/blob/master/src/reference/asciidoc/graph.adoc

Live SI Graph from the Running SI App

Assumes:

1. SI Viewer app is started
2. SI App is configured for serving Graph JSON from https://localhost:8080

SI Viewer app https://localhost:8877 endpoint accepts the following parameters:

• url - SI App Graph endpoint URL (typically https://localhost:8080/integration or https://localhost:8080/integration/refresh)
• refresh - time period in ms after which the SI graph JSON is refreshed

Thus, in order to see the SI graph from the running SI app navigate to URL: https://localhost:8877/?url=https://localhost:8080/integration

Zipped JAR file is attached. Note it's just an early prototype. If it blows up rendering your particular graph please let us know - I'm sure we'll be able to fix it promptly. Feedback is very welcomed.

sprotty-si-view-0.0.1-SNAPSHOT-exec.jar.zip

[realwuffi]

Hi all, we are using the current STS 3 graph viewer as well during development phase. Sometimes when new developers come in the team or when you have a look at some older projects, its a good possibility to get a quick overview of the current situation. (Yes I know, some other documentation may help there as well...)

We use Integration mainly for communication with ERP systems like SAP, Manhattan etc. Also we developed a small plugin for eclipse to show our own components like a FileEndpoint or a PersistentChannel, which persists the messages an headers in our Oracle database.

Sometimes the application gets quite heavy weight, so starting the whole thing just for having a quick look on smaller changes would be a bit time consuming. As we decided to stay with the XML context for the Integration part, the whole Graph would be stored in an XML. And the connection to the rest of our software is done via ServiceActivators. One possibility I could think of, would be to feed the IntegrationGraphServer with ClassPathXmlApplicationContext to render this context. Then we could have a small application like your SI Graph Viewer app to have a look on the design of the SI context. Would not be as convenient as the Eclipse plugin, but may be a bit more save in future, if Eclipse is really not any longer supported. We have to maintain and extend applications sometimes for 15 years, so that standalone viewer might be a handy thing to have.

If I can support you in any way, please let me know. Wolfgang

[martinlippert]

Hey Wolfgang, thanks a lot for sharing the details, much appreciated. And we will come back to your support offer for sure once we get back to this topic sometime soon. Thanks a lot in advance!

[swiss-chris]

@BoykoAlex thanks for your instructions above at https://github.com/spring-projects/sts4/issues/167#issuecomment-557170957. They still work, and of all the instructions I've tried to follow to get one of the Spring Integration Graph Visializing projects projects working (e.g. Spring Flo), this seems to be the only one where the instructions still work. There's only one correction I would make, the mvnw clean install doesn't make much sense to me in the context of your post and I think you can remove it.

That said, what I really wanted to ask is: whats the current status of officially supported Spring Integration Graph Visualization Tools ? It would be great to have something like what you shared as a JAR, but ideally allowing me to configure an entry points so that if I have multiple flows inside the same Spring Boot Application, the visualization only covers the flow/application I am interested in.

[VaibhavS94]

@BoykoAlex Is there any update on the SI View App, Is anyone working on it?

[BoykoAlex]

Nope... Don't think this is planned yet...

[clfsoft]

@BoykoAlex would you mind shareing the code in the comment . it's outdated with the latest SI(lack integrationPatternType,integrationPatternCategory)

[BoykoAlex]

@clfsoft I hope this is it: https://github.com/spring-projects/sts4/tree/sprotty/headless-services/sprotty-si-view It is the sprotty branch of this repo.