Dealing with huge amount of data

algarsi3 commented 4 years ago

Hello Eric,

first of all, thank you so much for this wonderful tool. I've been using it for more than two years. Currently, we are trying to show around 4 million variations in the ideogram. This means a 116 MB JSON file. It seems that the library is not able to show such data. Do you have any advice on how to deal with this? variations are stored with format: [snp,14587459,1,1]

Thanks.

eweitz commented 4 years ago

I've been using it for more than two years.

Glad to hear, Alberto! Could you tell me where and how you use Ideogram.js? Knowing such things helps steer my work.

This means a 116 MB JSON file. It seems that the library is not able to show such data. Do you have any advice on how to deal with this?

I can't think of anything off-hand that would preclude that. Two things would help me debug:

Could you post the data somewhere, and link me to it?
Could you provide your Ideogram.js configuration options here (e.g. config from new Ideogram(config))?

algarsi3 commented 4 years ago

Glad to hear, Alberto! Could you tell me where and how you use Ideogram.js? Knowing such things helps steer my work.

We are comparing full genome sequences of fruits to show their differences at a genome-wide level.

Could you post the data somewhere, and link me to it?

Sent you an email :)

Could you provide your Ideogram.js configuration options here (e.g. config from new Ideogram(config))?

{
      annotationsColor: '#ff5717',
      annotationsLayout: 'histogram',
      annotationsPath: '/assets/data/as6fb2k.json',
      barWidth: 1,
      chrHeight: 600,
      chrMargin: 80,
      container: '#ideo-container',
      dataDir: '/assets/data/',
      histogramScaling: 'absolute',
      isBrush_p: false,
      organism: 'human', // replaced homo-sapiens.js with our own data
      rotatable: false,
      showBandLabels: false,
}

homo-sapiens.js:

{
  "chrBands": [
    "1 p 1 0 14651197 0 14651197 gneg",
    "1 q 1 14651197 29302395 14651197 29302395 acen",
    "2 p 1 0 18415412 0 18415412 gneg",
    "2 q 1 18415412 36830824 18415412 36830824 acen",
    "3 p 1 0 25844203 0 25844203 gneg",
    "3 q 1 25844203 51688407 25844203 51688407 acen",
    "4 p 1 0 12985073 0 12985073 gneg",
    "4 q 1 12985073 25970147 12985073 25970147 acen",
    "5 p 1 0 21920875 0 21920875 gneg",
    "5 q 1 21920875 43841751 21920875 43841751 acen",
    "6 p 1 0 12966972 0 12966972 gneg",
    "6 q 1 12966972 25933944 12966972 25933944 acen",
    "7 p 1 0 10698868 0 10698868 gneg",
    "7 q 1 10698868 21397736 10698868 21397736 acen",
    "8 p 1 0 12713096 0 12713096 gneg",
    "8 q 1 12713096 25426192 12713096 25426192 acen",
    "9 p 1 0 15901768 0 15901768 gneg",
    "9 q 1 15901768 31803537 15901768 31803537 acen"

  ]
}

eweitz commented 4 years ago

That context helps, Alberto. At what organization or institution do you use Ideogram?

I confirmed Ideogram displays the 113 MB as6fb2k.json you sent me. Perhaps check that your dataDir and annotationsPath are properly set, e.g. try moving the files from your working local "Annotations, histogram" reference example into the location specified by annotationsPath and verify that works.

What do you see in your case? What do Console and Network tabs in DevTools report?

For reference, below is my HTML, and what I see. I put huge-data.html and as6fb2k.json in /examples/vanilla. It takes about 20 seconds to load and render the 5.7 million(!) variants.

<!DOCTYPE html>
<html>
<head>
  <script type="text/javascript" src="../../dist/js/ideogram.min.js"></script>
</head>
<body>
  <h1>5.7 million variants (issue #232) | Ideogram</h1>

  <div id="ideo-container" style="height:800px;"></div>

  <script type="text/javascript">

    var ideogram = new Ideogram({
      annotationsColor: '#ff5717',
      annotationsLayout: 'histogram',
      annotationsPath: '/examples/vanilla/as6fb2k.json',
      dataDir: '/dist/data/bands/native/',
      barWidth: 1,
      chrHeight: 600,
      chrMargin: 80,
      container: '#ideo-container',
      histogramScaling: 'absolute',
      isBrush_p: false,
      organism: 'human', // replaced homo-sapiens.js with our own data
      rotatable: false,
      showBandLabels: false,
    });
  </script>
</body>
</html>

eweitz commented 4 years ago

Hi @algarsi3, I hope this works for you! If not, let me know and we'll revisit.

algarsi3 commented 4 years ago

Hello Eric,

That context helps, Alberto. At what organization or institution do you use Ideogram?

At polytechnic university of valencia.

Thank you for your time, it is working. There was an issue storing the generated file.

eweitz / ideogram

Dealing with huge amount of data #232