search

MaayanLab / clustergrammer

An interactive heatmap visualization built using D3.js
http://maayanlab.github.io/clustergrammer/scrolling_tour
MIT License
230 stars 85 forks source link

port to python >= 3.5 please! #4

Closed paul-shannon closed 8 years ago

paul-shannon commented 8 years ago

Hoping this can go on your to-do list.

cornhundred commented 8 years ago

It's on my list now :)

paul-shannon commented 8 years ago

A Python3 clustergrammer notebook widget will be VERY useful!

On Jul 22, 2016, at 3:15 PM, Nicolas Fernandez notifications@github.com wrote:

It's on my list now :)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

cornhundred commented 8 years ago

Yeah it would be great to get it working as a Jupyter notebook widget (maybe as a part of the bokeh project). I'm not familiar with how to make these widgets. Do you have any experience with this Paul?

The closest thing I have done is this IFrame hack described here: https://github.com/MaayanLab/clustergrammer/blob/master/README.md#jupyter-notebook-example-workflow

But this requires uploading to the web app.

I'll keep you updated on any updates of this kind.

Nick

paul-shannon commented 8 years ago

Hi Nick,

I worked on this, got through my initial confusion. Here are my notes. Maybe you will find them helpful.

They show how to create the simplest notebook widget, following the full MVC, backbone.js, server/browser division of responsibilities.

Based on code and README’s from these three websites:

http://ipywidgets.readthedocs.io/en/latest/examples/Custom%20Widget%20-%20Hello%20World.html https://github.com/ipython/ipywidgets https://github.com/jupyter/widget-cookiecutter

pip install cookiecutter
cd ~/s/examples/jupyter cookiecutter https://github.com/jupyter/widget-cookiecutter.git cd ~/s/examples/jupyter/jupyter-widget-example/ pip install -e . jupyter nbextension install --py --symlink --user ipywidgetexample jupyter nbextension enable --py --user ipywidgetexample jupyter notebook

within notebook: from ipywidgetexample import * HelloWorld() # displays simple text "Hello World!" in the "widget space" below the input cell

--- now change the message, the simplest possible modification ~/s/examples/jupyter/jupyter-widget-example/ipywidgetexample/example.py now reads

@widgets.register('hello.Hello') class HelloWorld(widgets.DOMWidget): """""" _view_name = Unicode('HelloView').tag(sync=True) _model_name = Unicode('HelloModel').tag(sync=True) _view_module = Unicode('jupyter-widget-example').tag(sync=True) _model_module = Unicode('jupyter-widget-example').tag(sync=True) value = Unicode('Hello Universe!').tag(sync=True)

--- rebuild

pip install -e .
jupyter nbextension install --py --symlink --user ipywidgetexample
jupyter nbextension enable --py --user ipywidgetexample
jupyter notebook

On Jul 22, 2016, at 5:22 PM, Nicolas Fernandez notifications@github.com wrote:

Yeah it would be great to get it working as a Jupyter notebook widget (maybe as a part of the bokeh project). I'm not familiar with how to make these widgets. Do you have any experience with this Paul?

The closest thing I have done is this IFrame hack described here: https://github.com/MaayanLab/clustergrammer/blob/master/README.md#jupyter-notebook-example-workflow

But this requires uploading to the web app.

I'll keep you updated on any updates of this kind.

Nick

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

cornhundred commented 8 years ago

Paul,

Thanks those notes were very helpful. I'll let you know if I can get the clustergrammer widget working.

Nick

cornhundred commented 8 years ago

Paul,

Here's a link to a pre-release version of clustergrammer_widget (Python 2.7) for IPython/Jupyter notebook. I'll let you know when I have upgraded to 3.5 and when a more stable version is released.

https://github.com/MaayanLab/clustergrammer-widget

Thanks again for your help with the widget.

Best, Nick

paul-shannon commented 8 years ago

Hi Nick,

Very nice work. I cloned the repo, gave it a look. Sorry if this seems ungrateful, but I think I will wait till you support python 3.5 before digging into actual use. Very cool to have this ready for notebook use.

One minor typo in the README.md

git clone https://github.com/maayanlab/clustergrammer_widget.git

I think this should be

“clustergrammer-widget.git”

On Oct 18, 2016, at 2:47 AM, Nicolas Fernandez notifications@github.com wrote:

Paul,

Here's a link to a pre-release version of clustergrammer_widget (Python 2.7) for IPython/Jupyter notebook. I'll let you know when I have upgraded to 3.5 and when a more stable version is released.

https://github.com/MaayanLab/clustergrammer-widget

Thanks again for your help with the widget.

Best, Nick

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

cornhundred commented 8 years ago

Thanks, I'll let you know when 3.5 is available. -Nick

cornhundred commented 8 years ago

Paul,

I used the 2to3 conversion tool to update to Python 3.5 compatibility. Everything should be working for Python 2 and 3. The Jupyter notebook widget should also be working for 3.5. You can try using pip3 to install both clustergrammer.py and clustergrammer_widget. I was able to get the pip-installable widget is working in mac but not windows yet.

Best, Nick

paul-shannon commented 8 years ago

Cool! I will try this later today. Thanks, Nick.

On Oct 19, 2016, at 12:15 PM, Nicolas Fernandez notifications@github.com wrote:

Paul,

I used the 2to3 conversion tool to update to Python 3.5 compatibility. Everything should be working for Python 2 and 3. The Jupyter notebook widget should also be working for 3.5. You can try using pip3 to install both clustergrammer.py and clustergrammer_widget. I was able to get the pip-installable widget is working in mac but not windows yet.

Best, Nick

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

cornhundred commented 8 years ago

Hey Paul,

I'll close the issue since Python 3.5 seems to be working, but let me know if you have any issues.

Best, Nick

paul-shannon commented 8 years ago

Hi Nicolas,

That’s great news. I have not been able to use your notebook widget yet, having been preoccupied with (among other things) making a prototype jupyter widget for cytoscape.js. That now works, sort of, and I will try out clustergrammar next week. I am excited to do so!

On Nov 4, 2016, at 8:34 AM, Nicolas Fernandez notifications@github.com wrote:

Hey Paul,

I'll close the issue since Python 3.5 seems to be working.

Best, Nick

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

cornhundred commented 8 years ago

Paul

Cool, if you want to share your cytoscape widget I would like to try it out too. Hopefully, I can make a few small updates to the widget over the weekend.

-Nick

paul-shannon commented 8 years ago

Hi Nick,

The cytoscape widget is in its earliest stages, not yet fit for anyone to use (or even look at!). I will keep you posted as it matures.

I just tried but had trouble with the ceustergrammer widget install. Can you see what I am doing wrong? Trace below.

Thanks!

On Nov 4, 2016, at 11:27 AM, Nicolas Fernandez notifications@github.com wrote:

Paul

Cool, if you want to share your cytoscape widget I would like to try it out too. Hopefully, I can make a few small updates to the widget over the weekend.

-Nick

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

cornhundred commented 8 years ago

Hey Paul,

You had a typo in spelling 'clustergrammer', but you can try the pip installation (instead of the developer installation), which should be quicker (link). You might need to use pip3 for Python3. Let me know if that works.

Best, Nick

paul-shannon commented 8 years ago

Thanks for the quick response, Nick. The typo, as it turns out, was only in the email, not in the bash command itself. However that may be (I may have done something equally dumb…), here is a fresh try using the link you sent. Any ideas?

book.~> pip install clustergrammer_widget Requirement already satisfied: clustergrammer_widget in ./github/clustergrammer-widget Requirement already satisfied: ipywidgets>=5.1.5 in ./anaconda/lib/python3.5/site-packages (from clustergrammer_widget) Requirement already satisfied: widgetsnbextension>=1.2.6 in ./anaconda/lib/python3.5/site-packages (from ipywidgets>=5.1.5->clustergrammer_widget) Requirement already satisfied: ipython>=4.0.0 in ./anaconda/lib/python3.5/site-packages (from ipywidgets>=5.1.5->clustergrammer_widget) Requirement already satisfied: ipykernel>=4.2.2 in ./anaconda/lib/python3.5/site-packages (from ipywidgets>=5.1.5->clustergrammer_widget) Requirement already satisfied: traitlets>=4.2.1 in ./anaconda/lib/python3.5/site-packages (from ipywidgets>=5.1.5->clustergrammer_widget) Requirement already satisfied: notebook>=4.2.0 in ./anaconda/lib/python3.5/site-packages (from widgetsnbextension>=1.2.6->ipywidgets>=5.1.5->clustergrammer_widget) book.~> jupyter nbextension enable --py --sys-prefix clustergrammer_widget Unrecognized JSON config file version, assuming version 1 Unrecognized JSON config file version, assuming version 1 Enabling notebook extension clustergrammer_widget/extension...

On Nov 8, 2016, at 5:04 AM, Nicolas Fernandez notifications@github.com wrote:

Hey Paul,

You had a typo in spelling 'clustergrammer', but you can try the pip installation (instead of the developer installation), which should be quicker (link]. Let me know if that works.

Best, Nick

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

cornhundred commented 8 years ago

Hey Paul,

I will have to look more into those errors. I found this about the Unrecognized JSON config file version, assuming version 1 error.

You could try double checking that you have the latest version of Jupyter (probably your best bet is to use Anaconda 'conda' package manager to install/update Jupyter) and that you have the latest version of clustergrammer_widget with the command

pip install clustergrammer_widget --upgrade

It can be a little tricky to make sure you are in the right conda environment (activating environments).

I'll let you know if I find anything.

Best, Nick

cornhundred commented 8 years ago

Paul,

I updated the installation instructions. The problem might be that you need to run the middle line also:

# enable widgetsnbextension
jupyter nbextension enable --py --sys-prefix widgetsnbextension

Nick

paul-shannon commented 8 years ago

Thanks, Nick. For my own widgets, and my own eduction,I have been working my way through all the options of the nbextension commands. Progress is a little slow.

Following your suggestion, ceustergrammer is now apparently properly installed (but see below). Its installation and structure will be a good case study to learn from.

After what appears to be a successful installation, I load your Python 3.5 Test notebook, then excuse the first (and only) cell:

from clustergrammer_widget import * net = Network() net.load_file('rc_two_cats.txt') net.make_clust() clustergrammer_notebook(network = net.export_net_json())

It executes with only a warning (maybe you know the answer to that as well :}). missing sysctl warnings appear in some other notebooks, but without apparent effect:

/Users/paul/anaconda/lib/python3.5/site-packages/numexpr/cpuinfo.py:76: UserWarning: [Errno 2] No such file or directory: 'sysctl' stacklevel=stacklevel + 1):

I will try to eliminate the warning - must be some anaconda misconfiguration or installation ambiguity.

Thanks for your considerable patience!

On Nov 9, 2016, at 10:49 AM, Nicolas Fernandez notifications@github.com wrote:

Paul,

I noticed that you ran the command

jupyter nbextension enable -py --sys-prefix clustergrammer_widget

where you put clustergrammer_widget at the end instead of widgetsnbextension

jupyter nbextension enable -py --sys-prefix widgetsnbextension

That might be the problem.

Best, Nick

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

cornhundred commented 8 years ago

Paul,

Okay, I will look into the warning and thanks for your help too. Were you able to get a visualization despite the warning?

paul-shannon commented 8 years ago

No visualization, alas. Probably something else I am doing, or failing to do.

Is there any lightweight quasi-standard way to add tests to a notebook? So that (especially for demo notebooks) a test cell can be executed, or all cells run in order, with some “asserts” called to make sure (for instance) that the output cells have sensible values?

On Nov 9, 2016, at 11:30 AM, Nicolas Fernandez notifications@github.com wrote:

Paul,

Okay, I will look into the warning and thanks for your help too. Were you able to get a visualization despite the warning?

• Nick — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

cornhundred commented 8 years ago

Thanks for the quick reply. I would say maybe try restarting Jupyter notebook if you have not already done so (after running that last nbextension command). You can also try separating each line into a separate cell to see which parts are running without errors.

Also, if you want you can also try installing a different instance of Python (maybe 2.7) and see if starting from scratch works.

Nick

paul-shannon commented 8 years ago

Thanks, Nick. I’ll try these things later today.

On Nov 9, 2016, at 11:47 AM, Nicolas Fernandez notifications@github.com wrote:

Thanks for the quick reply. I would say maybe try restarting Jupyter notebook if you have not already done so (after running that last nbextension command). You can also try separating each line into a separate cell to see which parts are running without errors.

Also, if you want you can also try installing a different instance of Python (maybe 2.7) and see if starting from scratch works.

Nick

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

paul-shannon commented 7 years ago

Hi Nick,

(Sorry to be sending this via github issues. Is there a more appropriate email address?)

As I think I mentioned yesterday, I have been putting in time trying to understand the (to me, rather complex) architecture of jupyter widgets.

I bet you can explain to me one puzzle I have - which is, after modifying javascript in (for example) js/src/example.js, how do I assemble that new javascript into the webpack bundle or npm module that gets deployed, and visible to a notebook server?

I am using an old-fashioned makefile at the root of a newly installed cookiecutter directory containing the simple HelloWorld widget:

default: pip install -e . jupyter nbextension install --py hello --symlink jupyter nbextension enable --py --sys-prefix hello jupyter notebook basicDemo.ipynb

My changes to js/src/example.js simply never show up.

Got any tips for me?

Thanks -

On Nov 9, 2016, at 11:47 AM, Nicolas Fernandez notifications@github.com wrote:

Thanks for the quick reply. I would say maybe try restarting Jupyter notebook if you have not already done so (after running that last nbextension command). You can also try separating each line into a separate cell to see which parts are running without errors.

Also, if you want you can also try installing a different instance of Python (maybe 2.7) and see if starting from scratch works.

Nick

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

cornhundred commented 7 years ago

Paul,

Yeah, we can move the conversation to clustergrammer-widget. But they are using webpack to build the main script from the source code. It's a little confusing, you have to run webpack from the js directory where the package.json is located.

Nick

On Nov 10, 2016 6:38 PM, "Paul Shannon" notifications@github.com wrote:

Hi Nick,

(Sorry to be sending this via github issues. Is there a more appropriate email address?)

As I think I mentioned yesterday, I have been putting in time trying to understand the (to me, rather complex) architecture of jupyter widgets.

I bet you can explain to me one puzzle I have - which is, after modifying javascript in (for example) js/src/example.js, how do I assemble that new javascript into the webpack bundle or npm module that gets deployed, and visible to a notebook server?

I am using an old-fashioned makefile at the root of a newly installed cookiecutter directory containing the simple HelloWorld widget:

default: pip install -e . jupyter nbextension install --py hello --symlink jupyter nbextension enable --py --sys-prefix hello jupyter notebook basicDemo.ipynb

My changes to js/src/example.js simply never show up.

Got any tips for me?

Thanks -

  • Paul

On Nov 9, 2016, at 11:47 AM, Nicolas Fernandez notifications@github.com wrote:

Thanks for the quick reply. I would say maybe try restarting Jupyter notebook if you have not already done so (after running that last nbextension command). You can also try separating each line into a separate cell to see which parts are running without errors.

Also, if you want you can also try installing a different instance of Python (maybe 2.7) and see if starting from scratch works.

Nick

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/MaayanLab/clustergrammer/issues/4#issuecomment-259840401, or mute the thread https://github.com/notifications/unsubscribe-auth/AH90SDLfheTSkAE6NfEMRENK4VD6RiRhks5q86r3gaJpZM4JTLHH .

paul-shannon commented 7 years ago

Hi Nick,

I finally got the build-and-deploy process to work, running out of a makefile:

bash> cat makefile default: (cd js; webpack --config webpack.config.js) pip install -e . jupyter nbextension install --py hello jupyter nbextension enable --py --sys-prefix hello jupyter notebook basicDemo.ipynb

The webpack command which did the trick. It appears to be a closely held :} secret, unmentioned in any of the jupyter widget docs. Or maybe I just missed it. Along with some judicious use of

touch timestamp

find . -newer timestamp I was able to see a bit into the effects of the various commands, and dispel some of my confusion. Today I will get back to clustergrammer. I am excited to put it to work with my collaborators here at the ISB. Before I get back to jupyter-cytoscape.js, I want to package up a small and simple d3 interactive widget - this is the tutorial I wish I had when I started. The ipywidgets tutorials and recommended examples leap from helloWorld and a datepicker, all the way up to large and complex examples using 3js, for instance, leaflet and plotbq. The group expressed some interest in my proposal of an intermediate example, using a tiny bit of d3. I started out with a notebook-only version using javascript magic, totaling < 100 lines of code. Now the same code is going into the extension framework - something which now I may actually know how to do. I will offer this to the ipywidgets group, who may still be interested. The I urn back to jupyter cytoscpe.js, working in a github repo. Perhaps that will be of some use to you and your group. If so, I’d be glad to get your feedback as the project proceeds. - Paul > On Nov 10, 2016, at 6:59 PM, Nicolas Fernandez notifications@github.com wrote: > > Paul, > > Yeah, we can move the conversation to clustergrammer-widget. But they are > using webpack to build the main script from the source code. It's a little > confusing, you have to run webpack from the js directory where the > package.json is located. > > Nick > > On Nov 10, 2016 6:38 PM, "Paul Shannon" notifications@github.com wrote: > > > Hi Nick, > > > > (Sorry to be sending this via github issues. Is there a more appropriate > > email address?) > > > > As I think I mentioned yesterday, I have been putting in time trying to > > understand the (to me, rather complex) architecture of jupyter widgets. > > > > I bet you can explain to me one puzzle I have - which is, after modifying > > javascript in (for example) js/src/example.js, how do I assemble that new > > javascript into the webpack bundle or npm module that gets deployed, and > > visible to a notebook server? > > > > I am using an old-fashioned makefile at the root of a newly installed > > cookiecutter directory containing the simple HelloWorld widget: > > > > default: > > pip install -e . > > jupyter nbextension install --py hello --symlink > > jupyter nbextension enable --py --sys-prefix hello > > jupyter notebook basicDemo.ipynb > > > > My changes to js/src/example.js simply never show up. > > > > Got any tips for me? > > > > Thanks - > > - Paul > > > > > On Nov 9, 2016, at 11:47 AM, Nicolas Fernandez notifications@github.com > > > wrote: > > > > > > Thanks for the quick reply. I would say maybe try restarting Jupyter > > > notebook if you have not already done so (after running that last > > > nbextension command). You can also try separating each line into a separate > > > cell to see which parts are running without errors. > > > > > > Also, if you want you can also try installing a different instance of > > > Python (maybe 2.7) and see if starting from scratch works. > > > > > > Nick > > > > > > — > > > You are receiving this because you authored the thread. > > > Reply to this email directly, view it on GitHub, or mute the thread. > > > > — > > You are receiving this because you modified the open/close state. > > Reply to this email directly, view it on GitHub > > https://github.com/MaayanLab/clustergrammer/issues/4#issuecomment-259840401, > > or mute the thread > > https://github.com/notifications/unsubscribe-auth/AH90SDLfheTSkAE6NfEMRENK4VD6RiRhks5q86r3gaJpZM4JTLHH > > . > > > > — > > You are receiving this because you authored the thread. > > Reply to this email directly, view it on GitHub, or mute the thread.
cornhundred commented 7 years ago

Paul,

Sounds good, let me know if you need help getting the clustergrammer_widget working. I think I figured out that webpack was needed because I saw the webpack.config.js file in the js directory, but I'm not sure if it is in the docs. I would also be interested in helping out with a smaller d3 example (when I'm not so busy :) ) so send me a link to the repo if you want.

paul-shannon commented 7 years ago

send me a link to the repo if you want.

Hi Nick,

You will see straightaway that these two github repos are for beginners, not for experienced jupyter widget developers like you. But perhaps you will see where improvements can be made.

I’ll definitely get back to ceustergrammer next week.

Cheers!

Two repos, both present a simple jupyter widget: a d3 canvas on which circles are drawn in response to python function calls, or by directly clicking on the d3 canvas.

The first demo (possibly useful as a tutorial also) is simple, with all code self-contained in one notebook:

https://github.com/paul-shannon/jupyter-widget-demo-all-in-notebook

The second demo - to my mind, pretty complex - should be useful if you want to learn how to make installable notebook widget extensions and have little prior understanding of the mechanisms involved:

https://github.com/paul-shannon/jupyter-widget-demo-nbextension

Suggestions, advice, criticism: all are welcome.

cornhundred commented 7 years ago

Paul,

Sounds great. Thanks for the links I'll have a look.

Nick

On Nov 11, 2016 8:49 PM, "Paul Shannon" notifications@github.com wrote:

send me a link to the repo if you want.

Hi Nick,

You will see straightaway that these two github repos are for beginners, not for experienced jupyter widget developers like you. But perhaps you will see where improvements can be made.

I’ll definitely get back to ceustergrammer next week.

Cheers!

  • Paul

Two repos, both present a simple jupyter widget: a d3 canvas on which circles are drawn in response to python function calls, or by directly clicking on the d3 canvas.

The first demo (possibly useful as a tutorial also) is simple, with all code self-contained in one notebook:

https://github.com/paul-shannon/jupyter-widget-demo-all-in-notebook

The second demo - to my mind, pretty complex - should be useful if you want to learn how to make installable notebook widget extensions and have little prior understanding of the mechanisms involved:

https://github.com/paul-shannon/jupyter-widget-demo-nbextension

Suggestions, advice, criticism: all are welcome.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/MaayanLab/clustergrammer/issues/4#issuecomment-260093604, or mute the thread https://github.com/notifications/unsubscribe-auth/AH90SMvefOLVUZDaMkPN2ayOb267GrX-ks5q9Rs0gaJpZM4JTLHH .