Extend Mesh Plugin for Line Data

[dmadisetti]

The Mesh Plugin seems like a great place to add 3D graph embedding visualizations (Although, now I'm writing this, projector might be a better fit? Let me know what you think- projections for higher dimensional graphs would be pretty cool)

To do this, I see a few changes:

• [ ] Add a content type for line
• [ ] Provide lines kwarg in api + the data massaging into the summary
• [ ] Switch off points, lines, faces when rendering
• [ ] Make some lines!

I'm happy to implement this, but I need to find the time- I just wanted to check if this would be a welcome change, or if I should just make my own plugin

edit: Implementing for projector instead looks a bit hairier. Let me know a good way to approach this if this is the preferred method

[stephanwlee]

Greetings! Projector definitely is not the most ideal place to make this change. Wondering, is there a reason why you specifically want the line? Can you have a mesh that can form a line instead? While I am generally favorable to your change, I am trying to gauge whether the change is absolutely required.

Btw, when I used THREE.js's LineBasicMaterial, I found it to misbehave and does not really let you customize width so we, for other visualization, ended up using bunch of triangles instead.

[dmadisetti]

Sure, so I first experimented with hacking together lines, e.g.

    vertices = [[1,1,1], [0,0,0], ...]
    faces = [[0,1,0], ...]
    summary = mesh_summary.op('mesh', vertices=vertices, faces=faces,
        colors=tf.zeros_like(vertices)
    })

This doesn't seem to produce an output, which is expected (maybe there's ThreeJs params I can pass in that I'm not seeing?). I tried giving my triangle some depth at the midpoints, relative to the line length, but this looks pretty ugly (maybe I should get over it).

Most importantly, I'd ideally like to see the relevant nodes. As such, avoiding the hack, and implementing LineMaterial seems the best way to do this.

I agree that Projector is not the easiest place to put this feature, but as ML for graph processing becomes more common place, you may hear more calls for this feature.

[dmadisetti]

I think a separate plugin may be the best option. It would allow for graph particular embeddings (e.g. force directed)

[stephanwlee]

Would you be able to inform as to what a 3D graph embedding is? I am still not sure why there would be DAGs in an embedding. If you could point us to a blog, paper, or just about any material, we'd definitely appreciate it.

In the meantime, if you think this would not be a great fit for current TensorBoard plugin and benefit the machine learning community, yes, I will invite you to create a new plugin.

[dmadisetti]

By 3D graph embedding, I just mean a graph embedding in R^3.

I am interested in bidirectional, potentially cyclic, graphs- I'm not sure where I said DAG here. Here's a stackexchange answer that I think frames graph embeddings and their importance https://datascience.stackexchange.com/a/24083 and the paper it references: https://arxiv.org/pdf/1705.02801.pdf

Recent Google particular examples:

• Alpha Fold which considers a molecule a spatial graph
• DeepMind's Graph Net library Note to vizulize resultant graphs, they;ve just hacked something together with matplotlib

[dmadisetti]

Would you want this plugin back upstream? I can close this and make a new issue.

Or I can create a separate plugin, liberally license it, and let it be integrated as you see fit

[stephanwlee]

Sorry, "DAG" is something that I added. There, I wanted to say a graph with edges, or a simply graph :).
The word "graph" means many things to us so I wanted to get that cleared up.

It looks like you simply want to have some visualization for things like GNN and a general graph, not TensorFlow's GraphDef. It is not yet clear to me is whether the word "embedding" is meaningful--I have read the abstract of the Arxiv paper but it does not seem to require much specialized feature other than one for a simple graph in R^{2,3}-- but, unless I am mistaken, the definition is not too important for time being.

Now, about the plugin. We generally ask contributors to distribute their plugin on PyPI themselves which can be loaded onto TensorBoard via "entry_points" as is described here: https://github.com/tensorflow/tensorboard/blob/master/ADDING_A_PLUGIN.md. Hope the material in the ADDING_A_PLUGIN.md can get you started.

[dmadisetti]

Great thanks! I'll take a look at some point.

Maybe the term embedding muddied understanding, but graphs don't inherently occupy a vector space or manifold- so I was just trying to make the distinction that I wanted pretty graphics. Appreciate the help!