Show all links between children of two nodes

[swhalemwo]

Cool project! I just came across it and I'm really intrigued by the idea of only showing the relevant links as well as the keyboard navigation. When I started playing around with it, I however wondered if it is possible to show all the relations between two kinds of nodes.

For example, if over some time I have connected tasks and people (or scientific theories and cases, or actors and movies etc), it would be nice to get an overview of the structure by showing all the links between the two kinds of nodes. However I cannot really imagine how such relations could be visualized in plain text, especially when nodes get multiple links.

Do you have any ideas on this?

[Kungsgeten]

Glad you like it! I don't quite understand what you mean with "two kinds of nodes". Lets take your example with movies and actors, what is it you want to visualize then? Every actor and every movie at the same time? It might just be that I don't quite understand the terminology of nodes/links here.

[swhalemwo]

Yup indeed! I'd like to see all actors and all movies (and the links between them) at the same time to get idea of the structure. What I have in mind would look like a bipartite graph. But I don't really know if that's possible in plain text.

[Kungsgeten]

Its probably hard to visualize in text, since a lot of lines would cross each other etc. Maybe this is a task for something like graphviz? Some has requested that org-brain could export to graphviz, but I haven't figured how to do that and make it look good.

Another alternative could be that you don't see the connections, but when you "select" an entry on the left side (by placing the point over it) the linked entries on the right side are highlighted, and vice versa.

[ahmed-shariff]

I think having a query like system would be a more reasonable starting point. For example, what are the list of entries that have two particular nodes as their parents.

[MarioRicalde]

I think that @ahmed-shariff is onto something with this one, if there's some sort of query system you can pretty much list them in some more 'detailed' view.

That way you could see more information about the note you're looking at, for instance, a node would tell you something like:

This is a child of 16 entries. (listed) This is a friend of 5 entries. (listed)

[Kungsgeten]

@ahmed-shariff @MarioRicalde One thing I've thought about is to look into org-ql by @alphapapa and use it inside org-brain. That would require a lot of refactoring though. If org-ql is used I think it would be possible for org-brain to expand the query language and add some extra functionality specific to how org-brain works.

One thing I've tried to avoid in my packages is using external packages (which aren't included in Emacs). I'm starting to change my mind here though, since I distribute my packages on MELPA anyway.

[swhalemwo]

little update: I've been hacking some visualization together which looks somewhat like I had in mind originally; it's nowhere near finished but what's there i've uploaded here

regarding the selection/query language: for now i'm using the pinned notes to select which entries/links to display which works ok-ish. overall i quite like not having to type a query (similar to the magit interface).

it also came to my mind that the data/relations in concepts mapping seems to be somewhat less coherent/clearly defined/more quick-and-dirty than in a DBMS, which is where i know query languages from. for more analytical queries a query language would certainly be very useful, but I think it's somewhat less relevant in most of the day-to-day interaction with org-brain (just my 2 cents. in any way, implementing it is way beyond my abilities).

[Kungsgeten]

@swhalemwo I don't really understand what I'm seeing, but it sure looks cool 😄 I haven't heard of graph-tool before, maybe I'll have a look. I haven't tried EAF myself, mostly because I use Emacs on both Windows and Linux and I want the same experience on both.

@MarioRicalde I did some experimenting with org-ql and I've actually added it as a dependency to org-brain. It improves the performance of org-brain quite a bit. At the moment it is only used internally, but it would be nice to expose some functionality to end users so that entries can be queried.

[bepolymathe]

Hi,

Your discussion is interesting. I've been going around this for some time with the double idea of org-ql + data visualisation.

I rather like the intuition of orgqda which proposes to play on the tags to export in .csv which is importable in VUE to use results with the visualization.

It could be wonderful to be able to use Org-brain to archive, classify, tag... then export the results of the relationships to help analysis. As a researcher, this would be very useful!

[bepolymathe]

The proposal of @swhalemwo seems to me to be an excellent lead!

[swhalemwo]

@Kungsgeten Thanks! graph-tool is doing the layout calculations, i just listed it separately from the other python libs because it can be quite a hassle to install (it was for me, at least). I also just found pymacs, might have a look at it.

@bepolymathe yup, visualizing the relationships between concepts for research was is one of my motivations as well! i hope having them in digital form beats my paper scribbles as they tend to clutter quickly. I had a look at VUE; I like the visuals and would certainly like to not have to re-invent the wheel (it was quite an effort for me to write the drawing functions for rudimentary nodes and links in PyQt) but it doesn't really seem "hackable"? it's possible to import csv via the GUI, but I couldn't really find a way to e.g. update layouts or add/remove nodes in an automated kind of way (i don't really know java tho).

[bepolymathe]

@swhalemwo I understand the effort and I wouldn't have been able to do it ;-) I talked about VUE because I have friends who got in touch with the developers to try to make it evolve. But on my side, using a lot Emacs and Org-brain, I'm looking for solutions to go further knowing that I'm not a developer.

For VUE the import of csv is bad at this stage but the idea of orgqda is not bad. If you can do better with Python it's good too!

[theottm]

@swhalemwo thanks a lot for implementing this ! I will try it out.

Have you seen someone mentioned using D3.js through an export to CSV in #50 ? Would it it be easy to do this ? What data-format do you use to pass all the graph data to graph-tool in python ?

I am also looking for a way to visualize my knowledge graph. I would be glad to contribute to what seems promising.

There seem to be a less complete but maybe simpler feature here #105. I thought this could be useful if it could be quickly inserted in an org-mode buffer.

Maybe it would be nice in the future to unify the visualization to one command and select the background rendering engine from there.

Very excited to try this all out and see where I can help improve visualization of the brain!

[swhalemwo]

@theottm glad to see there's interest! I think I came across d3.js (not issue #50 tho) when looking for graph visualization, e.g. here which tbh looks quite fancy. I think I didn't pursue that further because I don't know JavaScript and wanted to get some result (I have more experience with python so I first tried graph-tool and pyqtgraph, but they were computationally expensive and not as configurable as I liked so I switched to low-level PyQt stuff). Overall I'm not very well-versed in visualization/GUIs, so I'll still have to see what issues/tasks wait there.

Generating the data is more straightforward, atm I'm using org-brain internal functions to generate an edge list that is sent to python (I think I used some of the code of #105 as a general inspiration for this but iirc the focus there was on the hierarchical structures extending from one parent, not e.g. friend relationships). I could convert that edge list to e.g. csv, but EAF allows to directly send it so I'm not really writing anything to files. The issue here is rather to effectively select the nodes/links for visualization; atm I'm using pins but maybe there are better ways (perhaps it also depends on the use cases/org-brain structurations of different people).

So I'd say it's the visualization backend and the node/link selection in org-brain that can use improvement. atm i think it's less about specific tasks, more about trying out things and see what works :) If you have more ideas on those topics (like using d3.js, which I maybe should have a look at now that you mentioned it; perhaps this post is relevant as well) I'd very much like to hear them!

[Kungsgeten]

@swhalemwo @theottm @bepolymathe I've started a new repository named org-brain-export. I think discussions focused on exporting could be moved there. A starting point could be: https://github.com/Kungsgeten/org-brain-export/issues/2

[michaelsjackson]

Yup indeed! I'd like to see all actors and all movies (and the links between them) at the same time to get idea of the structure. What I have in mind would look like a bipartite graph. But I don't really know if that's possible in plain text.

Now having obvz, what would be the right way of achieving this visualization? Using cls_movies and cls_actors?