RobotLocomotion / drake

Model-based design and verification for robotics.
https://drake.mit.edu
Other
3.17k stars 1.24k forks source link

javascript rendering of system diagrams #13874

Open RussTedrake opened 3 years ago

RussTedrake commented 3 years ago

(I wrote this in an email for someone potentially willing to help with the cause; but realized it deserves to be here in an issue for anybody that might want to help).

Quick background:

I think a few hours of trying some of the js diagramming libraries and deciding what works would result in an immense improvement in the user/student experience. I think one key feature is the ability to expand/collapse nested subdiagrams.

RussTedrake commented 3 years ago

I spent a little more time looking through the options yesterday, but haven't found one that is obviously good.

To address #13975, I realized pydot could create svg. For moderate size diagrams, that is a reasonable solution for diagram visualization, too.

I went through the entire list of javascript packages in https://codefreezr.github.io/awesome-graphviz/#language-bindings , as recommended by @jwnimmer-tri in #13705. Those links took me to https://github.com/dagrejs, which is the most promising path I've seen from graphviz to a javascript rendering. But even this package doesn't appear to support collapsing/expanding subdiagrams, so I feel the additional value it adds over the python-based svg rendering is not worth the trouble.

Because of the existing SVG solution, I am going to upgrade the feature request of collapsing/expanding here to a "must have". I honestly haven't found a good gojs alternative. I'm almost of the opinion that we should just write our own in d3. I bet I could do some pretty nice graph layout optimization in mathematicalprogram (probably a MILP).

I'm also ok if it takes a different form from collapse/expand. It might be possible to use a combination of max_depth on the graphviz and links in something like dagrejs to open subdiagrams.

rpoyner-tri commented 2 years ago

Seems like someone with js chops (not me) could do something decent with https://github.com/magjac/d3-graphviz (BSD 3-clause license). If it is possible to enumerate the set of expanded/collapsed graphs ahead of time, it may not be too hard to provide some navigation among them.

jwnimmer-tri commented 2 years ago

Does @trowell-tri fit that bill?

RussTedrake commented 2 years ago

There is one deprecated example of using GenerateHtml left in https://deepnote.com/project/Ch2-Lets-get-you-a-robot-j4YXK7WXTOubrZLRGsemzA/%2Frobot.ipynb which shows the old feature that we are trying to replace.

jwnimmer-tri commented 2 years ago

The recent https://github.blog/2022-02-14-include-diagrams-markdown-files-mermaid/ caught my eye. Possibly its https://github.com/mermaid-js/mermaid#readme will be a winner in terms of a durable graph modeling language for us to build upon.

RussTedrake commented 2 years ago

mermaid looks great, but it doesn't look to support collapsable subdiagrams. I think that's a required feature for this.

shensquared commented 2 years ago

I was just searching for interactive (i.e. collapsable) mindmap/diagram libraries myself and came across this issue. https://github.com/gera2ld/markmap (their demo) fit my bill. Not too sure about the specific needs here, but sharing fwiw.

shensquared commented 2 years ago

Chatted with Russ a bit yesterday and I understand the needs better now. https://js.cytoscape.org/ and its plugin https://ivis-at-bilkent.github.io/cytoscape.js-expand-collapse/ looked promising.

Building on top of those (and some other auxiliary libs, all MIT license), I cobbled together https://shensquared.github.io/diagram-proto/demo.html. The interaction isn't intuitive yet, so here's a gif. diagram

Comparing with the one in https://manipulation.csail.mit.edu/intro.html#example3 that Russ made with GoJS, I think we'd still need:

Both feel doable. I think I actually know how to tweak the listener/event. For the visual, cloning Russ's graph is for sure possible; in fact, Cytoscape's nodes/edges API looks pretty similar to GoJS (JSON based, uses keys like from, to, group, etc). I do need to look closer if their layout also has a smart ‘auto’ mode like GoJS, but even if it's not built in, seeing that they have a lively community (building extensions, plugins, etc), I’m optimistic.

Before we pursue this further though: wonder if I missed anything on the wish list?

RussTedrake commented 2 years ago

Thanks @shensquared ! I'm not so concerned about how "polished" the collapse/expand looks; as long as you can connect multiple ports in one diagram and get somewhat reasonable layouts, then this looks great!

shensquared commented 2 years ago

Sounds good. My to-do plan then, in order:

Will let you know if/when the first two (the essential ones) are done.

shensquared commented 2 years ago

@RussTedrake Updated the demo towards the first 2 items on my list. Does it look like what you had in mind?

In parallel, I plan to start trying auto-generate a real Drake diagram (instead of using 'Lorem ipsum' nodes/edges). Maybe from e.g. a .yaml? Wonder how/where did you get the port names and connections for your GoJS diagram?

RussTedrake commented 2 years ago

Looks great!

On Tue, Mar 15, 2022 at 10:26 PM, Shen Shen @.***> wrote:

@RussTedrake https://github.com/RussTedrake Updated the demo https://shensquared.github.io/diagram-proto/demo.html towards the first 2 items on my list.

— Reply to this email directly, view it on GitHub https://github.com/RobotLocomotion/drake/issues/13874#issuecomment-1068666089, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABRE2NAYFNALL3Y53SNLCMDVAFBETANCNFSM4QAFRCSA . You are receiving this because you were mentioned.Message ID: @.***>

shensquared commented 2 years ago

Sorry it's been a while.

Below is a mimic of Russ's manip_station diagram: The demo; it's based on Cytoscape.js, and line 24-34 toggles different edge styles and layouts.

Some shortcomings when compared with gojs:

As I was trying to improve these, questions/answers like this makes me a bit less optimistic about the base library.

So while I'm still trying, I'm also keeping an eye for other libraries. I'm also happy to bring this one in as is into GenerateHTML first (e.g. if we wanna first see how other Drake diagrams might look like with this?)

@RussTedrake any preference? thanks!

shensquared commented 2 years ago

The demo looks/works a lot better to me now.

The rendering is via Cytoscape; the collapse/expand is via cytoscape.js-expand-collapse; the layout is done by elk and then wrapped/parsed via cytoscape.js-elk. While not perfect, this combo is my favorite so far :D

Fwiw, other libraries that didn't quite work out but probably would be great depending on the need (so 'good-to-know's):

RussTedrake commented 2 years ago

That demo does look great. Thanks for doing it! Is there any chance that the input ports / output ports for a single system (e.g. the plant in this demo diagram) can be left/right justified?

shensquared commented 2 years ago

seems possible in elk though the wrapper has some lost-in-translation; I'll look more into it.

shensquared commented 2 years ago

Is there any chance the input ports / output ports for a single system (e.g. the plant in this demo diagram) can be left/right justified?

Check this out?

Also added in that demo:

Feel free to let me know too if these are irrelevant/counterproductive? Thanks.

zapplelove commented 10 months ago

I think the WebGME is a good choice, it can achieve the rendering of system diagrams and drag/drop modeling. But unfortunately, I'm not good at coding. Can anyone achieve it? image @RussTedrake @shensquared @rpoyner-tri @jwnimmer-tri