'IGV Browser not ready'

[sooheelee]

Hi Jim,

I'm attempting to run igv.js-jupyter in a FireCloud Jupyter Notebook. I'm using the example notebooks you provide and I get the 'IGV Browser not ready' message that does not resolve over time. Is there something I should be doing to get past this?

Here are my specs:

Thank you, Soo Hee

[jrobinso]

Did you install and enable the nbextension? What are the steps you used to install?

On Thu, Feb 14, 2019 at 2:11 PM sooheelee notifications@github.com wrote:

Hi Jim,

I'm attempting to run igv.js-jupyter in a FireCloud Jupyter Notebook. I'm using the example notebooks you provide and I get the 'IGV Browser not ready' message that does not resolve over time. Is there something I should be doing to get past this?

Here are my specs: [image: screenshot 2019-02-14 17 10 27] https://user-images.githubusercontent.com/11543866/52820959-75faeb80-307b-11e9-91ad-b1590e22f38e.png

Thank you, Soo Hee

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/igvteam/igv.js-jupyter/issues/23, or mute the thread https://github.com/notifications/unsubscribe-auth/AA49HIMnJO79SQV99MhCq5iY5O-ST4mXks5vNd8WgaJpZM4a8irj .

[sooheelee]

I used pip install igv.

[jrobinso]

On the firecloud server? That should be enough but you might have to enable the extensions

jupyter nbextension install --py igv jupyter nbextension enable --py igv

I'm actually a bit confused myself on the process for installing an extension on a share Jupyter server. @tmtabor should know, Thorin can you comment? The symptom Soo Hee is seeing leads me to believe the extension is not installed. @sooheelee Can you describe the Firecloud Jupyter setup a little, is it a shared server, or does the server and client both run on your machine?

[sooheelee]

I'm using the Jupyter Notebooks tab of FireCloud, which allows you to run a notebook on a standalone compute engine instance, specifically Dataproc. I just noticed https://github.com/igvteam/igv.js-jupyter/pull/21, but this was merged 26 days ago and the last release is from 28 days ago. Currently, here is what I'm seeing in the notebook:

[sooheelee]

What I ultimately need is to be able to visualize BigWig files in a FireCloud workspace, towards illustrating the GATK gCNV workflow.

[sooheelee]

I am trying to debug and really appreciate the help. Towards debugging I found https://github.com/Calysto/notebook-extensions/issues/4 and if I jupyter nbextension list, I see:

Not sure if helpful but I thought I'd share.

[jrobinso]

Did you pip install this on the Firecloud Jupyter server, or on your client machine, or both?

[sooheelee]

My client machine, a Google Compute Engine instance. I have a script that pip install igv plus GATK when the instance is created.

[jrobinso]

That won't be enough, it needs to be installed on the server. Having the admin run pip install igv on the server might be enough, but let's wait for @tmtabor to respond before trying that.

[sooheelee]

Ok. Here's some more information on the Jupyter Notebook (aka Leonardo) - FireCloud relationship: https://github.com/DataBiosphere/leonardo

[jrobinso]

Who administers this? IGV is a Jupyter extension, with python (server) and javascript (client) parts. The admin will definitely need to be involved in the installation and should be looped into this conversation.

[sooheelee]

Yes, I have asked @MatthewBemis and @kyuksel to weigh in.

[tmtabor]

I have seen this sort of error before. It usually happens when Jupyter has been installed as root, but you're trying to install a Jupyter extension from a user account.

What happens is that when you install Jupyter, it creates the directory /usr/local/share/nbextensions/ to contain its extensions. But if you're installing Jupyter as root, user accounts aren't going to have write permission to that directory. So when you try to install an extension there, it fails. (Seen in this screenshot.)

The real fix is to either install the extension as root, reinstall Jupyter from your user account or make /usr/local/share/nbextensions/ writable. I don't know FireCloud's particular setup, but I suspect this will require the system administrator.

[rtitle]

@tmtabor @sooheelee would this work?

!pip3 install igv
!jupyter nbextension install --py igv --user
!jupyter nbextension enable --py igv --user

I think this should install the extensions in user space?

Otherwise, Terra notebooks support plugging in "user scripts" which allow you to install/run things as root when the notebook server launches. That also might be a path forward here.

[sooheelee]

Thanks @tmtabor for the insight and @rtitle for a potential solution. I tried adding --user but I still get some Xs when I jupyter nbextension list:

[sooheelee]

I didn't realize the custom script is run with root privileges, so I am waiting for a new cluster to form where I've added the extension-install to the script.

[jrobinso]

@sooheelee @rtitle I think this extension will be of broad interest to Firecloud users. Could it be installed as part of the image for everyone?

[sooheelee]

Well, even with the custom script containing:

set -e
pip install igv
jupyter nbextension install --py igv
jupyter nbextension enable --py igv

on a fresh cluster, I get the same errors I show in the last screencapture above (https://github.com/igvteam/igv.js-jupyter/issues/23#issuecomment-464159151).

What can I try next? Really appreciate all the support here. Definitely @jrobinso @rtitle I think the igv.js-jupyter extension is of great interest to researchers.

[tmtabor]

In theory, the X's for _nbextensionsconfigurator shouldn't affect igv since that's a different nbextension.

But you can probably fix them by installing it as root (or with --user) as well.

[rtitle]

Yeah I'm not sure those red X's from nbextensions_configurator matter. I did the following as root:

pip3 install igv
jupyter nbextension install --py igv
jupyter nbextension enable --py igv

and this in a notebook:

No cell errors but I do see errors in browser console:

Any idea what this means?

[rtitle]

Is there a serverextension that needs to be installed in addition to a nbextension?

[rtitle]

I think this extension will be of broad interest to Firecloud users. Could it be installed as part of the image for everyone?

First we need to get it working. :) But yeah then we can consider installing it by default.

[jrobinso]

@rtitle this is both a server and nbextension, but pip install should install both. I'll see if I can reproduce this and get back to you. Stay tuned.

[jrobinso]

@sooheelee @rtitle @tmtabor OK, Helga was able to reproduce the "IGVBrowser not ready" with the "igvcomm not in registry error", but explicitly installing the nbextensions resolved it. @helgathorv correct me if I'm wrong here. The explicit install/enable should not have been neccessary, we followed instructions here, but obviously that wasn't working for everyone. So I've removed the auto-install and pushed a new version, 0.9.5, to pipy along with updated instructions. I tested with a fresh conda virtual environment, I'm going to list everything I did below for completeness but creation of the virtual environment would not apply in your case. However, if it still doesn't work with firecloud could you try a fresh environment as below?

• Create the virtual environment

conda create -n test python=3.7.1
conda activate test
conda install pip
conda install jupyter

• Install IGV -- note that you would not include the --sys-prefix flag if not in a virtual environment

pip install igv-jupyter
jupyter serverextension enable --py igv --sys-prefix
jupyter nbextension install --py igv --sys-prefix
jupyter nbextension enable --py igv --sys-prefix

• Run the notebook

jupyter notebook

[sooheelee]

Thanks, @jrobinso @helgathorv for these instructions. They work great on my laptop. It's a geat feeling to be able to interact with the data via the embedding. I hope @rtitle can enable this functionality on FireCloud Notebooks soon.

[sooheelee]

Update and follow-up question

With permission, I've moved my secondary analyses off of FireCloud for now. I need a heatmap visualization of germline copy-number variants derived from WGS samples, where the heatmap coloring corresponds to copy number. Yesterday I tested out three file formats towards visualization. I am hoping you can tell me if I've missed some way of getting to where I need in igv.js-jupyter (IGV). I'd really appreciate your insight.

1. VCFs of structural variants, where the POS field indicates the start of a CNV and the INFO field's END= annotation indicates the end of the same CNV event. Depending on the source of the callset, the extent of copy gain/loss is either denoted in the ALT field, e.g. with <CN0> or in a FORMAT level annotation, e.g. CN annotation is 0 in the sample field.
   • IGV marks the starting POS but does not extend the variant representation to the END postion. I know block-representation is possible in desktop-IGV for GVCF visualization. I tried changing the file-extension to .g.vcf but this did not change the visualization.
   • IGV coloring is based on GT. Need to color by CN state.
   • Feature visibility is independent of zoom-level, which is great.
2. SEG file where segment mean reflects copy number, e.g. 2 is diploid normal.
   • IGV fully represents the start and end of the variant and colors appropriately (blue-white-red)!
   • Feature visibility is dependent on zoom-level. Features are often not visible until zoomed in considerably, making us dependent on also loading the VCF.
3. BW bigwig file, which is performant for the size of the data (24 WGS samples). I had been using this format and desktop-IGV's heatmap coloring capability so far, based on the two different result formats of GATK4 gCNV. One of the formats gives a feature for every bin of data collected, e.g. every 1000bp of a WGS dataset.
   • IGV displays data as a graph, much like a histogram, which is useful.
   • Cannot convert to heatmap format like in desktop-IGV. Is this possible, and how do I go about it?

I would really like to be able to compare heatmaps that have full feature-visibility even at full-contig view levels. Thank you @jrobinso @tmtabor.

[jrobinso]

Seg files do not have visibility window restrictions, they even work at whole genome, so I'm not sure what is going on here. Possibly the segments get too small to see. Please open an issue in igv.js on this issue and attach an example seg file that reproduces the problem. I'm closing this issue since the "browser not ready" problem is resolved.

[sooheelee]

I suspect the diploid segments (in white) are given equal visibility as the non-diploid segments of interest (in blue and in red). I will see if removing CN=2 segments allows for better representation at the contig level before I open a new issue. Would it be okay for me to open an issue requesting heatmap representation of bigwigs in igv.js-jupyter?

[jrobinso]

@sooheelee The request should be to igv.js, this is not about jupyter, but that is not likely to get prioritized anytime soon.

[HBegali]

@sooheelee @jrobinso Thanks for ur comments