Core Network Dataviz

[harlantwood]

@jacksenechal @adamapollo and I have been working on some MIT licensed dataviz using the famous.org JS viz framework over here: https://github.com/nodesphere/graphviz/tree/ipfs

-- it's not usable through the gateway quite yet, so I'm including some screenshots for the moment below. This (roughly) shows entering a hash of the root of go-ipfs, and navigating through the Godeps into the github.com directory... @jbenet enjoy : )

[harlantwood]

For more of the vision of what we're working on, see http://www.nodesphere.org/ -- for now we support only IPFS : )

[whyrusleeping]

thats pretty... o.o

[cryptix]

Looks awesome! Can't wait to try it out!

[jbenet]

this looks amazing!! @harlantwood would you mind pushing even a dev version to the branch? we can keep it as an ongoing PR. would love to play with it!

[jbenet]

moved to #5

[harlantwood]

@jbenet glad you like it! You can play with it locally until the gateway is ready:

ipfs daemon --unrestricted-api  
open http://localhost:5001/ipfs/QmZdNvHk7kpMbyJed5yoeApJQYnWaBdCn7eJXkaboBRHie/

[harlantwood]

I'm re-opening this and closing #1 -- which is actually the one that you merged in in #5 -- this is a whole separate thread.

[harlantwood]

If you want to play with the source code:

git clone git@github.com:nodesphere/graphviz.git
git checkout ipfs
npm install
# make changes if desired
npm run build
ipfs add -r dist
ipfs daemon --unrestricted-api  # in another window
open http://localhost:5001/ipfs/<your hash>

[harlantwood]

Thanks for nice comments @whyrusleeping @cryptix :smile:

[jbenet]

Oh sorry, i meant #6

[harlantwood]

@jbenet said:

this looks amazing!! @harlantwood would you mind pushing even a dev version to the branch? we can keep it as an ongoing PR. would love to play with it!

I think the best way to play with it is as described above... I could copy the source code into this repo, but that seem like a duplicate to keep up to date...

[jbenet]

@harlantwood yep, that works :) no worries

[harlantwood]

Currently broken on gateway, eg:

http://gateway.ipfs.io/ipfs/QmZdNvHk7kpMbyJed5yoeApJQYnWaBdCn7eJXkaboBRHie/

Not quite sure why, will debug when I have some spare cycles.

[rht]

Limitations of graph visualizations (especially on semantic web, no matter how beautiful it looks): http://swui.semanticweb.org/swui06/papers/Karger/Pathetic_Fallacy.html

[jbenet]

Yeah, certainly most huge graphs turn into hairballs, but I think graph vis and interactive exploration can be very useful. The trivial graphviz graphmd tool already helped me find a bugs. Vis is very hard to do, and yes has limitations, but it can help humans traverse large data sets quickly. Also some dags are way easier than regular graphs 

— Sent from Mailbox

On Wed, Aug 26, 2015 at 3:12 PM, rht notifications@github.com wrote:

Limitations of graph visualizations (especially on semantic web, no matter how beautiful it looks): http://swui.semanticweb.org/swui06/papers/Karger/Pathetic_Fallacy.html

Reply to this email directly or view it on GitHub: https://github.com/ipfs/dataviz/issues/4#issuecomment-135009402

[harlantwood]

@jacksenechal I think the CORS issue we experienced on gateway might be this one: https://github.com/ipfs/infrastructure/issues/77

[harlantwood]

This is now working on the gateway; this link is the code visualizing itself:

http://gateway.ipfs.io/ipfs/QmVKFoisa75PALvkHte6AdA8zVej8xrQ4rmT2kHQF3Sbu3/#QmVKFoisa75PALvkHte6AdA8zVej8xrQ4rmT2kHQF3Sbu3

There are still a number of issues, both usability and technical, I'll keep you posted on any major improvements.

Thanks go out to @jacksenechal @adamapollo and @dukedorje!

[harlantwood]

@rht I agree, it's a very hard problem, and we are certainly making only baby steps so far.

If I were to imagine looking backward from the future at our present navigation systems ("the web"), I imagine it looking like a really really primitive graph visualization. So poor that you could only see one node at at time, and the links to other nodes were mostly just different-color words.

Why not galaxy-scale visualizations like 100,000 Stars?

Why not immersive holographic interfaces like those depicted in Iron Man?

We have a long way to go to get there, but if we don't try, we'll just be looking at simulated pieces of paper for the rest of our lives.

[jbenet]

This is now working on the gateway; this link is the code visualizing itself:

Very cool! works great! Here's go-ipfs: https://gateway.ipfs.io/ipfs/QmVKFoisa75PALvkHte6AdA8zVej8xrQ4rmT2kHQF3Sbu3/#QmbXyMDLz2RZgZK3kb3mpru7mBe7xajiJ3oVqgeyn8WLsm

Feedback:

• wish i had a way to "visualize going back". So of course a merkledag has no "back", but perhaps a traversal may move from A -> A/B instead of A -> B. (i.e. the # link would be /ipfs/<hash>/<path>, not just <hash>).

More ideas:

• more inspiration: http://dhumanities.appspot.com/
• next up: could do "tree circle-packing" like this, and make a very useful vis for arbitrary dags.

Why not galaxy-scale visualizations like 100,000 Stars?

I've wanted an "explorable graph viz" exactly based on this for so, so, so, so long.

We have a long way to go to get there, but if we don't try, we'll just be looking at simulated pieces of paper for the rest of our lives.

+9001.

@harlantwood might be useful to schedule a weekly/fortnightly hangout on dataviz.

[harlantwood]

I made a little fix and published to IPNS to have a more stable URL:

http://ipfs.io/ipns/QmWdprFxhCWzjJ6D9Tw9tj5FyWFauhYuGtDQigVvwfteNv/

wish i had a way to "visualize going back". So of course a merkledag has no "back", but perhaps a traversal may move from A -> A/B instead of A -> B. (i.e. the # link would be /ipfs//, not just ).

Totally agree, I was thinking the same.

next up: could do "tree circle-packing" like this, and make a very useful vis for arbitrary dags.

Yup! We plan to develop this data viz (Core Network) to include several/many viz types, and circle packing may in fact be next! There is an issue for this too: #2.

Why not galaxy-scale visualizations like 100,000 Stars?

I've wanted an "explorable graph viz" exactly based on this for so, so, so, so long.

I know, right???

@harlantwood might be useful to schedule a weekly/fortnightly hangout on dataviz.

Yeah, sounds awesome. I'm deep in wedding/honeymoon mode here, will be back in circulation in early October. How bout Wed Oct 7 @ 2pm Pacific for the first one?

[jbenet]

I'm deep in wedding/honeymoon mode here,

Congratulations!!

How bout Wed Oct 7 @ 2pm Pacific for the first one?

Works for me! we need a community calendar

[harlantwood]

Thanks Juan 8D

For anyone interested, join us on Wed Oct 7 @ 2pm Pacific:

http://g.co/hangouts/rlan.me/ipfs-dataviz

[rht]

@harlantwood very sorry for the late reply to the dataviz discussion. It is because I haven't had the time to uncloud my view yet (not that it is any clearer now). And that my recent interactions with the computer & the web are almost exclusively in the form of 'saying' instead of 'gesturing' (I find GUI to be extraneous compared to the almighty REPL, ¯(ツ)/¯).

As stated in the RDF pathetic fallacy paper, the default representation of a graph data should be determined on a case by case basis (though I admit "ultimately everything is case by case" is a tired argument). Otherwise this could lead to sensory overload.

1. html traversal. I can't speak of things that are beyond the blue links and simulated paper. After a consensus has been reached on what the knowledge tree structure should be, then perhaps a subjective web that adapts to one's perception that is more fine-grained than lang localization (private lang?)? Will likely happen sooner in code, e.g. an RFC spec that translates to native expressions in prolog or APL. But maybe this is still primitive by the 'far future' viewpoint.
2. local file or source code traversal, as in the example. I find it easier to tell the computer directly of a file's name and code pattern instead of remembering its location or redoing the clicks on the graph every time. find+grep instead of tree.
3. dags with one edge type (source code (excluding loops) may count as one). The paper says this should be graphed because it is 'simpler', so it partially admits that bird-eye view is sometimes needed.
4. star chart. This could mean a tool to do interstellar navigation or astronomy. Dunno for the former, but for the latter, afaik, science tools suck, especially in obscure fields.
5. ironman HCI. While the mixed reality part is in the near future (now?), I wonder of the building/designing part?
6. knowledge dependency graph. Don't have a reason for why it should be "graph", but I wish this exists.

(note that: https://en.wikipedia.org/wiki/Syntactic_Structures#cite_note-26)

[harlantwood]

Thanks @rht for your comments. No particular responses, just thanks for your interest in the topic :)

[harlantwood]

We have switched technologies: Core Network is now using three.js instead of famous.

Note that this is not interactive yet, just an early demo version.

https://ipfs.io/ipfs/QmUPgdm6CtPAtFb4N2W9RDhWH4Mm2dBrLoJ8hFpTk4uJET/#Qmb3YqwpnxFgRdsUbnh4Y7Xsx4BbLrk3JWZQsCDfhznBZA

source: https://github.com/core-network/client-threejs

screenshots: