search

covid-19-net / covid-19-community

Community effort to build a Neo4j Knowledge Graph (KG) that links heterogeneous data about COVID-19
MIT License
118 stars 77 forks source link

Neo4j graph visualization or plugin for Jupyter Lab #1

Closed pwrose closed 3 years ago

pwrose commented 4 years ago

A Jupyter Lab plugin is urgently needed to visualize and interactively explore Neo4j KG.

Here is a list of JS libraries: https://neo4j.com/developer/tools-graph-visualization/

See also: https://github.com/neo4j-graph-analytics/graph-algorithms-notebooks/blob/master/notebooks/DegreeCentrality.ipynb

https://ipython-cypher.readthedocs.io/en/latest/

https://nicolewhite.github.io/neo4j-jupyter/hello-world.html

martinRenou commented 4 years ago

@jtpio Do you think we could help with this one? You definitely know how to make a JupyterLab plugin. And I'm happy to help as well.

martinRenou commented 4 years ago

cc. @wolfv as you said you were interested in this

This might be helpful: https://github.com/neo4j/neo4j-javascript-driver but I guess it requires back-end logic.

jtpio commented 4 years ago

That should definitely be possible by wrapping one of the existing graph libraries.

Wondering whether this should be a widget or a pure lab extension, since there is some authentication to access the Neo4j database.

pwrose commented 4 years ago

I think you can assume that Neo4j has been authenticated and you have access to the graph object. See ExampleQueries.ipynb.

@jtpio @martinRenou @wolfv It would be great if you could help with this project. I'll post some requirements for the graph visualization soon.

If you can, sign up as a GraphHack contributor (https://medium.com/neo4j/graphhackers-lets-unite-to-help-save-the-world-graphs4good-2020-fed53562b41f) and join the kick-off meeting:

Friday, March 27th at 11am ET | 15:00 GMT to Zoom. Link: https://neo4j.zoom.us/j/4601817416 Meeting ID: 460 181 7416 Find your local number: https://neo4j.zoom.us/u/abpGfRX3KS

seangrant82 commented 4 years ago

This doesn't have all the functionality of D3 but could be a good start at least: http://compbio.ucsd.edu/bringing-interactivity-network-visualization-jupyter-notebooks-visjs2jupyter/

pwrose commented 4 years ago

@seangrant82 visjs2jupyter cannot display multiple relationships with opposite directions. I haven't tried it on Jupyter Lab.

pwrose commented 4 years ago

What are the minimal requirements for a graph vis. in Juypter Lab?

Here are some thoughts:

Nice to have:

wolfv commented 4 years ago

Hi @pwrose we have worked on a graph visualization tool using ipywidgets (so that it works in Jupyter and Jupyter Lab).

It's based on cytoscape.js (https://github.com/quantstack/ipycytoscape) I am sure we could extend it to support all functionality you desire. Let me know if you want to chat about this cc @martinRenou @marimeireles

sonygreen commented 4 years ago

@pwrose I think we can help with this too. Our GraphXR software covers all of the need to haves (and 2 out of 3 nice to haves) that you listed.

pwrose commented 4 years ago

Hi @wolfv @martinRenou @marimeireles, I'd be happy to chat. I can set up a zoom call for Tue. Contact me at pwrose@ucsd.edu.

pwrose commented 4 years ago

@sonygreen do you want to add an example notebook to this repo to show how it works? Note, we are interested in open source solutions that can be installed from conda or pypi.

sonygreen commented 4 years ago

@pwrose Ah, my apologies. GraphXR is free to use but not open source

wdyang commented 4 years ago

@pwrose GraphxR is not opensource. However, we made it free for people to use with minimum restrictions. My workflow often involves using Python+Jupyter for data preparation, analysis, and bring to GraphXR for doing things you listed.

As a dedicated visual analysis tool, it can handle much more nodes, much easier to shift perspectives in visualization, as well as integration with other tools through API. Let me know if opensource is a must have for you. If you are open to use tools that are available for free, I'd love to chat with you.

pwrose commented 4 years ago

@sonygreen and @wdyang, let's chat in a week or two once we have some initial data in our KG and then see how GraphXR could be used for visuals analysis. It sounds like it would be complementary to what we do with Jupyter Notebooks.

marimeireles commented 4 years ago

Hey @pwrose so I'll be testing locally until we have release the new version of ipycytoscape and you'll be able to test it locally too. (Should take a couple of days, maybe this Friday).

To have a minimal working version:

It'd be nice if we could:

Plus all of the things you mentioned in the https://github.com/covid-19-net/covid-19-community/issues/1#issuecomment-605287835 and we still don't have

Regarding the interactive aspects like:

It's currently doable, the "materials" are all there, but there is not an API to do it. So we have two options here, either we implement something specific for your user case or try to create something more generic and add it to ipycytoscape. I think the later is doable and would be actually pretty cool, but might take a little longer and it's a bit more complicated. If anyone wants to help here and wants to discuss, I'm open to ideas.

As I said in the meeting I'm not really sure if we can add specific colors to nodes in graphs with the cytoscape API. If anyone has an idea on how to do that, it'll be welcomed =)

marimeireles commented 4 years ago

@wolfv @jtpio did I miss something?

pwrose commented 4 years ago

Also, since the Cytoscape team is also here at UCSD, we can connect with them for specific questions or feature requests as well.

On Thu, Apr 2, 2020 at 9:36 AM Mariana Meireles notifications@github.com wrote:

@wolfv https://github.com/wolfv @jtpio https://github.com/jtpio did I miss something?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/covid-19-net/covid-19-community/issues/1#issuecomment-607955915, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA53AEH3GEH4TD5R35HMDC3RKS5ILANCNFSM4LRFYWOA .

pwrose commented 4 years ago

Here is an example of how to get the data for a graph object: https://github.com/nicolewhite/neo4j-jupyter/blob/master/scripts/vis.py

marimeireles commented 4 years ago

@pwrose if it's this Cytoscape team they may not be aware of what's going on on CytoscapeJS which is the implementation I'm basing the widget in.

marimeireles commented 4 years ago

Hey @pwrose, I remember seeing some examples in your screen in the meeting where you showed some uses of networx, but I can't seem to find them anymore. Could you please link it to me? Thanks!

pwrose commented 4 years ago

@marimeireles The examples are here: https://nicolewhite.github.io/neo4j-jupyter/hello-world.html

marimeireles commented 4 years ago

Oh, thanks! Was looking in this repo.

pwrose commented 4 years ago

@marimeireles if you have any prototypes you want me to review, please let me know.

marimeireles commented 4 years ago

Hi @pwrose, sorry for taking some time for getting back to you. I prepared a small reproducible notebook example using the same database that you're using. This one.

I also found out that there is a straightforward way of changing nodes color, apparently you just have to add a style to it. I think you could iterate through a column of the df and change the colors of them to make a group. But I haven't tested it yet, lemme know if you need any help to get this working.

I really couldn't think of many different examples on the kind of data visualization you want to have. If you're struggling with anything let me know and we can build a solution together.

I also tested ipycytoscape with neo4j stuff, including the outputs of Cypher. I don't have an example right now, but I can write one for you if you need it, the results from NetworkX extracted from Cypher are pretty broken though, they're not able to form connections out of the box with the data that they're used in the repo you linked here.

To have access to this new features please download the new version of ipycytoscape, 0.2.0, make sure you have it as the latest in your environment =) A pip install ipycytoscape --upgrade should do it or conda install -c conda-forge ipycytoscape. You can check with pip list or conda list if the version is correct.

pwrose commented 4 years ago

Thanks @marimeireles !

I ran into a couple of issues:

  1. ipycytoscape is not on conda-forge, yet, so I used the version from PyPi
  2. It works in Jupyter notebook, but in Jupyter lab I get this error: Error displaying widget: model not found
  3. To reproduce this, I created the following repo: https://github.com/pwrose/neo4j-ipycytoscape There are two Binder links to try this out, one for Jupyter Notebook and one for Jupyter Lab. I added the Jupyter Lab extension here: https://github.com/pwrose/neo4j-ipycytoscape/blob/master/binder/postBuild#L19
  4. I created a fully functional deployment of Neo4j for Binder. Launch the Neo4j notebook to see how to create a small graph and list the nodes and edges. Can you add the graph visualization there? I put some notes there: https://github.com/pwrose/neo4j-ipycytoscape/blob/master/notebooks/Neo4j.ipynb

Please let me know if you have any questions.

marimeireles commented 4 years ago

@pwrose

  1. Was fixed, thanks.
  2. Could you please try with the new version? It's working for me in a new environment. I'll update the Readme as I find it a bit confusing.
  3. Okay, I gave a look at your notebook examples and they're using the input directly from neo4j. The alterations I did were based in other examples you sent me, they were manipulating the data from dataframes, and that's what the new function I added here is doing. I made a fast google search and apparently it's possible to make this conversion without much work. I'll work on an example for you.

Thanks for the repo with examples, they made it much clearer for me.

pwrose commented 4 years ago

@marimeireles I've updated the environment.yml file to use ipycytoscape 0.2.1 from conda-forge, see: https://github.com/pwrose/neo4j-ipycytoscape. Click the launch binder link for Jupyter Lab and you'll see the error: Error displaying widget: model not found

pwrose commented 4 years ago

@ery010 is now working on this issue.

marimeireles commented 4 years ago

I'm so sorry I'm not making progress on this! I'll keep you updated when I finish what you sent me by email. Good luck!

pwrose commented 3 years ago

Added a prototype using [ipycytoscape] (https://github.com/covid-19-net/covid-19-community/blob/master/notebooks/analyses/GraphVisualization.ipynb)