IGV-WEB: custom data sources

[rjpbonnal]

Hello, igv-web is a great tool, easy do deploy, immediate for the end-user and provides a very good user experience from those coming from igv standalone.

One feature I really like is the possibility to browse ENCODE data from a table. Following the ENCODE idea, giving the user the ability to browse, filter and select the tracks from one or multiple tables (or Datasources) become natual; when the lab geneates a lot of data is crucial.

At the moment only ENCODE is supported, quoting the documenations:The type property can be used in lieu of a track list to trigger pre-defined lists of tracks for special data sources. Currently the only recognized value is "ENCODE". This property can be used to populate an ENCODE load widget for any genome assembly with data hosted by the ENCODE project. Currently this includes assemblies for human (hg19, GRCh38), mouse (mm10), worm (ce11), and fly (dm6). { "label": "ENCODE", "type": "ENCODE", "description": "<a href=hhttps://www.encodeproject.org/ target=_blank>Encylopedia of Genomic Elements</a>", "genomeID": "hg19" }

I hacked the local source code to load our own table keeping the ENCODE schema but this is a very short term approach. I would like to have a more general approach for including custom datasources leveraging what is alredy there inside igv-web, igv-widgets, data-modal, and maybe other libraries. My goal is to add new buttons to the Track menu, where each button loads data ala ENCODE.

If you have comments, suggestions or ideas just let me know.

[rjpbonnal]

I did a hack modifing the igv-webapp (https://github.com/helios/igv-webapp/commit/415eb12abc54b14b4928c12b77d1c29a62b43662) and igv-widgets (https://github.com/helios/igv-widgets/commit/9fc981c351345e144ea13762607e317c5ae40bbc) Then I modded trackRegistry.json

  "hg38": [
    "resources/tracks/hg38_annotations.json",
    "resources/tracks/hg38_encode.json",
    "resources/tracks/hg38_1KG_lowCoverageCrams.json",
      "resources/tracks/hg38_custom.json",
      "resources/tracks/hg38_custom2.json"
  ],

and the customs files are something like :

{
    "label": "MAP-organoids",
    "type": "CUSTOM",
    "description": "<a href=hhttps://www.encodeproject.org/ target=_blank>Encylopedia of Genomic Elements</a>",
    "genomeID": "GRCh38",
    "suffix": ".tracks-organoids.txt",
    "dataSetPathPrefix": "http://localhost:8080/files/hg38/MAP/",
    "urlPrefix": "http://localhost:8080/files/hg38/MAP/",
    "url" : "http://localhost:8080/files/hg38/MAP/GRCh38.tracks-organoids.txt"
}

and

{
    "label": "MAP-fight-cancer",
    "type": "CUSTOM",
    "description": "<a href=hhttps://www.encodeproject.org/ target=_blank>Encylopedia of Genomic Elements</a>",
    "genomeID": "GRCh38",
    "suffix": ".tracks-fight-cancer.txt",
    "dataSetPathPrefix": "http://localhost:8080/files/hg38/MAP/",
    "urlPrefix": "http://localhost:8080/files/hg38/MAP/",
    "url" : "http://localhost:8080/files/hg38/MAP/GRCh38.tracks-fight-cancer.txt"
}

[turner]

This commit adds custom track types to the app and adds their appearance in the track menu 28966535

[tweep]

Hi @turner, can you point us to some docs on how to use the custom track types, maybe with an example ?

[turner]

Hi tweep. We will have that in the web app README soon.

[helgathorv]

@helios The Readme has now been updated. Let us know if you have any questions. Note that what is described in the Readme is supported by the code in master, but we don't yet have a pre-built package with this version at https://igv.org/app-archive (that will come soon).

[rjpbonnal]

@helgathorv sorry for this huge delay. Just read the docs here and in the data-modal, this sounds very cool. I will test this new feature this week. Having multiple ways (json, url, ..) for feeding the modal opens a lot of new possibilities.

[tweep]

Thanks everyone, we'll start test-driving this - it looks really promising. The REAMDE can be found here : https://github.com/igvteam/igv-webapp ( search for "my platinum genomes" )

[rjpbonnal]

@turner and @helgathorv I'am testing the new feature. Using the JSON data structures as in the example n.2 for the track registry it works as expected (great!). Instead, how can I use a text file formatted in CSV as input? In the mean time I can concert my CSV to JSON and load the data, just to let you know that my "usual" number of line in a table is more than 500 rows ( tracks)

[rjpbonnal]

errata corrige, I am not able to run the example n.2. For reproducibility I am running it inside a Docker, this is my environment:

FROM openresty/openresty:1.17.8.1-centos
#https://computingforgeeks.com/install-node-js-14-on-centos-rhel-linux/

RUN yum install -y curl &&\
    curl -sL https://rpm.nodesource.com/setup_14.x | bash - &&\
    yum install -y nodejs
RUN yum install -y git

WORKDIR /opt/
RUN git clone https://github.com/igvteam/igv-webapp.git
WORKDIR /opt/igv-webapp

RUN git fetch --all &&\
    rm -rf node_modules &&\
    git checkout 9631842 &&\
    npm install &&\
    npm run build

RUN mv dist/* /usr/local/openresty/nginx/html
RUN mv node_modules /usr/local/openresty/nginx

RUN yum remove -y diffutils patch git nodejs
RUN yum clean all

and this is the docker-compose.yml

version: '3.8'

services:
  igv-webapp-dev:
    build: ./
    image: igv-webapp-dev:9631842
    ports:
      - 7777:80
    networks:
      - development
    volumes:
      - "./resources/genomes.json:/usr/local/openresty/nginx/html/resources/genomes.json"
      - "./resources/tracks:/usr/local/openresty/nginx/html/resources/tracks"
      - "../files:/usr/local/openresty/nginx/html/files"
networks:
  development:

@turner @tweep @helgathorv the container and compose above are meant for using mostly the default configuration from git. You need to have a directory resources. If you have a better deployment strategy to perform tests I would more than happy to use it so we are aligned.

From the above dockerfile i did not included the installation of the master igv.js which is reported below:

WORKDIR /opt/
RUN git clone https://github.com/igvteam/igv.js.git
WORKDIR /opt/igv.js
RUN git fetch --all &&\
    git checkout 95f99d5 &&\
    npm install &&\
    npm run build
# TODO no mini js is generated just the esm
#FIX RUN mv dist/igv.esm.js /usr/local/openresty/nginx/html/

#FIX WORKDIR /usr/local/openresty/nginx/html
# Change the reference in the index to using the newly compiled/generated igv.js
# I doubt that esm is working ( i need to investigate more because I don't know the process very well)
#FIX RUN sed -i 's#https://cdn.jsdelivr.net/npm/igv@2.7.4/dist/igv.min.js#igv.esm.js#g' index.html

[helgathorv]

@helios Regarding CSV input - that's not currently supported, but were planning on adding it, and I've bumped that up a bit in priority. I created a separate issue for that if you want to follow it.

[jrobinso]

@helios Sorry we don't use Docker. To align with us you would just pull from master and build with NPM. If that doesn't work out of the box something is wrong with our build.

[rjpbonnal]

errata corrige, I am not able to run the example n.2. @jrobinso it took me a while to figure our the problem. I re did the whole installation, it was a detail in my reverse proxy, I can confirm that the solution n.2 works.

In any case the docker configuration can be useful to others to setup an isolated environment for IGV for testing NPM installation and not cluttering the local environment. Eventually we may add in a wiki sub section so that if others want try.

The custom-data-modal is a game changer.