local data sources - Githubissues

ajo2995 commented 8 years ago

I would like to embed a browser configured with data sources from local javascript objects. Is something like this possible?

var features = [
  {chr: '1', start: 123, end: 234, label: 'foo'},
  {chr: '1', start: 333, end: 444, label: 'bar'},
  {chr: '3', start: 111, end: 222, label: 'baz'}
];
var styles = [
{
    type: 'default', 
    style: {
       glyph: 'PLIMSOLL', 
       HEIGHT: '12', 
       STROKECOLOR: 'black',
       FGCOLOR: 'red'
    }
},
{
    type: 'nasty',
    style: {
        glyph: 'CROSS',
        FGCOLOR: 'green'
    }
}
];
browser.addTier({
    name: 'MyData',
    desc: 'I have these features in memory', 
    localData: features,
    localStyleSheet: styles
});

ajo2995 commented 8 years ago

After reading some of the code, I decided to modify MemStoreFeatureSource to add an option to bypass the parsing. It will be a less ugly hack if the features can be checked for validity with the source.payload parser. Does this make sense to you? I'll submit a pull request when I get it working.

max-l commented 7 years ago

@ajo2995 is "localData" supported in the current release ?

I tried it and it doesn't seem to work,

it's definitely a worthwhile feature...

ajo2995 commented 7 years ago

It has not been merged in to the dasmoth/dalliance AFAIK.

I haven't used it yet but the idea was to embed a protein browser in the search.gramene.org results list and display domain annotations as tracks. I already had the positions of the domains on the protein sequence, but I needed to modify things to allow for a protein sequence source adaptor (via ensembl REST) and massage the domain annotations into an acceptable format. I recall getting a small test instance working, but the protein browser feature was postponed.

dasmoth commented 7 years ago

If you're using the trunk version of Biodalliance, then for small numbers of features, you can use the "test-source" store, as per:

 https://github.com/dasmoth/dalliance/blob/master/test/render-test.html#L146-L151

For a larger number of features, something closer to what @ajo2995 suggests is probably a better bet. Although I'd be curious to know more about the use case -- in general, I've found that making a bigBed file or similar is usually the easiest solution.

max-l commented 7 years ago

@dasmoth the use case I have is that I need to show a small number of trancsripts, proteins, peptides (never more than a dozen at a time), but the data feeds from a database with hundreds of thousands of them, and the small selected subset is deternined by a search result from a database.

Currently I produce a set of huge bed files, and move the view so a specific set of segments can be seen (transcripts, peptides). The problem is that I only want to show a tiny subset of the segmets, and if I load the browser with my bedfiles, everyhing gets displayed (not only the small subset).

If I could "turn off" (erase visually) all the segments in a bed file except for the few desired ones, I wondered at one point if I could achieve this with a rendering callback, it seems not.

One thing I have tried is to create small bed files with only the tiny set I need to show, but this tiny set is the result of a database search (driven by a UI), so it's not so practical to pre generate all of them them (there would be millions), it would be better if I could hand them to the browser in the form of JSON rows from the client in javascript.

A third alternative Is to serve dynamically generated bed files, I have tried that, but it seems the browser doesn't like uncompressed bed files https://github.com/dasmoth/dalliance/issues/206 I could convert them to bigbed, but for such small files it's unfortunate if I have to call bedToBigBed...

moritzschaefer commented 6 years ago

@max-l I have a similar use case. On top it can be useful to compute some features on the fly in the web application and directly show them in a then dalliance sequence browser. @dasmoth Is there something wrong with the pull request? If so maybe we can edit it so it can get merged.

dasmoth commented 6 years ago

Sorry, I rather lost track of this one :blush:

I'll hopefully have time this weekend to take a look. Moritz -- if you get a chance, splitting the "pre-parsed features" stuff out of the larger pull request would make it that bit easier to consider in isolation.

moritzschaefer commented 6 years ago

I'll try to work on it during the weekend.

dasmoth commented 6 years ago

Just wanted to point out one other alternative:

    {name:      'Inline BED test',
     tier_type: 'memstore',
     payload:   'bed',
     uri:        URL.createObjectURL(
                     new Blob([
                         ['22 30000000 30010000',
                          '22 30050000 30060000'].join('\n')
                     ])
                 )
     }

(It should be possible to use a data: URL instead of a blob: if that's easier for specific cases...)

This may seem a little more complex, but it does have a couple of advantages:

Clear separation between "external" data model's and Biodalliance's internal "feature" type
It preserves the possibility of identifying source configs by the URI of the data they point to (this isn't an absolutely hard requirement, but does avoid needing special case code in a few places...)

TransGirlCodes commented 6 years ago

Hi, is it possible to inline something like a BigWig file in this way?

I tried by doing something like this:


        new window.Browser(

{"viewStart":1,"chr":"1","pageName":"39db3c5f-4572-4b5c-954d-5b83c1479504","viewEnd":100000,"sources":[{"bwgURI":URL.createObjectURL(
    new Blob([38,252,143,136,3,0,4,0,200,0,0,0,0,0,0,0,248,0,0,0,0,0,0,0,247,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,160,0,0,0,0,0,0,0,0,64,0,0,0,0,0,0,0,0,0,0,64,0,0,0,0,0,0,0,75,4,0,0,0,0,0,0,58,13,0,0,0,0,0,0,0,1,0,0,0,0,0,0,142,13,0,0,0,0,0,0,169,16,0,0,0,0,0,0,0,4,0,0,0,0,0,0,253,16,0,0,0,0,0,0,234,17,0,0,0,0,0,0,0,16,0,0,0,0,0,0,62,18,0,0,0,0,0,0,144,18,0,0,0,0,0,0,16,39,0,0,0,0,0,0,0,0,0,64,95,232,160,63,0,0,0,96,106,193,239,63,0,0,0,32,164,76,181,64,0,0,0,96,33,228,172,64,145,140,202,120,1,0,0,0,4,0,0,0,8,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,1,0,99,104,114,49,0,0,0,0,16,39,0,0,1,0,0,0,0,0,0,0,120,156,61,212,109,72,214,87,28,198,241,171,82,179,238,202,199,212,150,149,101,217,35,98,117,71,82,247,255,252,204,41,40,86,203,205,178,135,27,138,208,117,87,46,163,230,114,180,150,133,65,13,153,173,133,211,205,160,192,182,32,217,12,170,249,194,162,145,140,96,35,238,162,23,22,70,65,110,19,74,10,42,52,87,217,117,21,116,224,251,255,188,59,231,127,14,135,3,188,27,177,51,241,126,12,67,217,91,245,253,174,191,215,201,107,172,242,219,72,147,243,134,1,125,45,125,78,30,101,251,35,22,154,124,198,46,248,31,57,185,118,56,112,124,176,208,100,7,43,217,123,198,201,180,17,192,224,226,42,147,181,44,51,228,60,217,203,202,63,235,117,114,121,4,144,95,243,218,147,109,108,119,77,156,147,137,145,192,177,101,191,122,178,154,237,8,239,114,178,155,253,123,241,186,147,57,81,64,125,184,217,201,22,118,123,81,145,201,232,145,192,39,190,175,77,86,176,35,249,93,78,134,217,245,184,118,39,253,209,192,178,169,89,38,27,217,239,129,59,78,190,100,3,7,190,48,185,105,20,240,127,213,40,147,157,172,234,82,208,228,172,209,220,207,151,255,56,89,199,134,18,154,60,249,132,53,108,29,107,178,196,7,252,212,176,196,100,59,219,21,179,215,100,234,24,160,245,210,58,147,53,236,191,246,67,78,246,176,194,79,203,77,22,140,5,238,23,23,153,108,101,253,87,103,120,50,118,28,231,133,223,228,231,172,35,242,27,39,187,88,209,221,225,38,3,49,60,19,31,76,158,100,115,134,54,152,140,136,229,89,108,46,55,25,98,73,63,223,241,228,223,236,214,160,223,100,86,28,176,101,230,95,158,60,206,6,238,254,226,222,202,158,215,79,49,25,140,7,210,123,146,77,94,97,147,138,47,59,57,61,1,248,51,144,96,242,48,91,121,122,141,201,71,108,107,242,31,78,174,74,228,255,79,107,115,242,60,75,109,250,216,100,202,120,174,151,249,145,201,175,152,175,116,188,201,251,204,59,183,192,100,94,18,240,192,95,104,242,12,155,29,222,237,228,152,100,32,183,62,206,228,78,150,152,189,218,228,45,54,180,237,132,147,217,41,192,252,123,48,217,204,126,204,141,50,137,9,192,247,17,27,77,150,177,156,135,158,201,107,108,114,203,99,39,231,125,0,100,60,244,153,60,202,178,87,31,243,228,51,54,152,59,210,147,107,39,2,81,129,60,147,29,172,237,212,13,79,166,165,2,167,75,107,157,172,101,241,161,124,147,189,236,172,183,198,228,242,73,188,255,157,75,77,182,177,248,210,113,38,19,39,3,77,121,15,156,172,102,87,19,206,57,217,205,214,151,101,120,50,103,10,176,47,101,187,201,22,246,234,183,128,201,232,52,96,221,193,185,38,43,88,168,127,132,201,48,11,102,151,155,244,79,5,138,183,103,152,108,100,231,251,110,59,249,146,69,103,109,49,185,105,26,112,51,152,110,178,147,173,218,184,194,228,172,116,96,113,245,82,147,117,44,233,195,37,38,159,176,230,31,54,155,44,153,14,44,218,22,48,217,206,130,251,187,156,76,157,193,187,88,176,207,100,13,171,124,58,224,100,15,171,235,223,227,100,65,6,247,83,241,194,147,173,236,72,76,165,147,122,183,66,37,153,246,6,151,130,18,234,224,172,104,36,64,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,16,39,0,0,247,3,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,16,39,0,0,0,1,0,0,0,0,0,0,247,2,0,0,0,0,0,0,157,0,0,0,120,156,77,216,123,116,79,87,22,7,240,131,68,164,241,74,60,18,84,144,16,163,49,94,9,34,242,187,231,119,246,208,241,174,72,48,17,197,32,146,122,20,177,48,168,184,53,177,198,107,36,69,169,16,109,103,121,140,25,212,35,245,168,182,110,4,107,97,22,65,25,209,210,26,207,241,156,97,76,194,132,204,217,103,239,174,28,107,125,255,250,220,155,239,239,236,115,239,253,221,31,33,232,159,159,243,81,249,61,135,163,182,213,72,85,63,155,107,249,244,85,129,242,230,221,114,213,179,203,76,227,104,30,59,26,7,10,71,173,48,142,38,106,144,63,218,242,200,156,255,32,229,223,42,252,250,4,227,198,44,95,20,16,39,155,253,41,24,74,194,10,168,95,155,203,142,198,129,53,99,94,82,191,54,143,253,203,248,135,230,252,91,169,129,208,124,88,62,245,107,19,53,171,125,237,203,254,50,227,187,64,24,48,104,45,245,163,177,163,113,32,59,189,21,152,126,109,46,123,234,130,63,155,243,23,79,20,224,196,210,231,67,243,44,127,217,115,182,220,92,240,181,10,216,21,69,253,218,68,45,114,52,14,116,189,53,217,252,125,99,236,157,178,28,31,186,147,39,160,116,230,3,90,191,54,215,114,142,255,220,230,122,142,233,215,230,89,158,49,237,158,51,118,194,247,106,238,139,190,212,175,77,4,144,247,117,95,27,159,212,122,135,106,113,33,152,214,143,102,57,199,127,170,230,66,105,250,181,185,150,207,114,67,157,137,141,235,171,163,205,55,27,71,243,216,87,171,221,198,151,222,171,169,78,255,184,220,56,154,8,172,118,142,63,51,241,145,249,252,198,44,127,191,52,219,89,209,1,84,86,65,123,191,233,215,230,178,163,221,57,112,214,169,23,150,171,58,44,92,101,28,205,99,71,227,168,107,247,99,104,253,218,68,109,242,188,210,77,198,135,111,221,167,226,35,95,155,243,141,89,206,81,123,90,122,212,175,205,101,47,235,62,80,114,96,230,144,54,102,255,208,60,203,83,66,114,164,51,235,183,80,20,212,223,56,154,8,34,71,227,192,188,230,31,208,254,163,177,47,235,123,197,65,223,117,180,82,189,122,190,138,246,95,155,203,126,54,244,144,131,199,140,233,126,86,29,174,160,245,161,121,150,115,212,134,180,0,90,191,54,81,167,218,85,155,46,50,182,69,11,56,247,206,73,218,127,52,246,131,73,87,141,31,200,143,134,204,244,203,212,175,205,181,156,163,230,132,36,82,191,54,207,242,138,15,231,200,194,74,128,171,103,218,209,250,181,137,96,242,255,205,14,150,232,9,123,167,67,84,187,9,180,126,52,203,57,240,114,243,26,234,215,230,90,62,251,155,209,242,80,179,56,184,20,22,64,243,215,230,177,163,113,224,151,35,28,234,215,38,222,32,207,157,119,219,65,223,120,248,169,170,49,109,6,173,31,141,189,170,209,6,31,30,115,225,68,161,42,43,170,73,253,218,92,203,57,254,243,161,7,233,254,211,230,89,190,238,189,122,50,183,226,161,154,242,52,135,230,175,77,132,144,163,109,92,151,40,255,94,20,1,23,98,15,83,63,26,59,26,7,186,85,5,209,243,71,155,107,121,118,131,5,114,247,63,39,193,177,19,41,180,126,109,30,251,206,111,210,140,39,222,158,3,199,215,189,79,235,215,38,234,86,59,7,154,199,245,162,249,163,177,223,61,180,196,65,175,188,30,0,143,247,126,73,235,215,230,90,206,81,3,247,175,48,247,7,154,199,222,127,82,134,228,64,251,252,193,212,175,77,212,35,255,41,121,160,241,244,87,105,240,211,219,137,212,143,102,57,7,158,205,167,251,11,205,101,47,47,105,231,67,63,251,32,88,13,153,188,133,250,181,121,150,167,138,120,25,145,179,80,21,167,183,162,249,107,19,245,249,249,172,141,3,15,91,191,162,249,163,177,127,29,184,220,65,159,155,91,172,154,141,235,65,235,215,230,90,62,240,90,77,185,181,231,231,234,111,125,174,81,191,54,143,29,141,3,85,167,151,80,191,54,209,128,159,47,33,194,120,143,230,53,96,77,226,98,234,71,179,252,173,170,116,121,234,116,32,44,58,152,71,253,218,92,118,52,14,228,215,247,209,254,107,243,44,159,58,62,67,158,217,51,14,238,12,236,75,243,215,38,26,146,163,113,160,247,50,218,31,99,236,77,183,93,245,161,207,185,220,74,53,28,50,207,172,15,205,181,252,187,151,241,50,236,206,14,85,163,199,175,233,254,215,230,177,163,113,96,220,205,42,90,191,54,17,74,158,217,254,140,57,191,77,159,81,42,118,64,169,249,251,198,44,175,184,182,221,57,214,178,181,58,54,184,46,245,107,115,217,209,56,170,40,230,7,154,191,54,207,242,231,121,173,228,246,71,65,112,240,233,6,234,215,38,194,200,163,111,133,75,244,249,5,145,240,98,245,113,154,63,154,229,28,40,243,21,209,252,181,185,236,45,147,191,117,208,253,207,159,168,152,65,211,104,253,218,60,203,57,234,195,15,234,81,191,54,209,136,252,100,82,35,201,129,162,119,119,80,63,154,229,67,182,142,144,139,191,234,2,25,223,86,82,191,54,151,29,141,3,167,42,226,104,255,181,121,236,239,133,23,59,232,165,133,215,213,47,166,166,81,191,54,209,152,175,255,168,61,14,30,115,225,244,17,117,176,138,191,255,209,44,231,168,25,99,158,210,252,181,185,150,191,185,97,152,108,93,30,7,155,23,7,83,191,54,143,125,109,167,119,36,122,244,243,100,248,79,92,71,186,254,180,137,38,213,206,129,209,59,219,211,245,135,198,30,50,178,137,241,235,65,177,48,108,251,125,90,191,54,215,114,223,222,110,114,119,199,112,152,249,175,125,52,127,109,30,59,26,7,238,206,126,65,235,111,130,23,110,181,223,140,239,47,27,62,243,193,146,233,245,168,31,141,29,141,3,225,109,35,233,249,163,205,101,239,80,58,203,65,239,60,99,133,42,189,127,132,174,63,109,158,229,144,23,42,115,70,31,86,219,167,181,166,126,109,34,156,28,141,3,221,254,176,149,230,143,102,121,227,132,225,114,234,210,126,80,118,150,223,63,181,185,236,85,147,11,29,244,233,235,227,224,143,190,16,154,191,54,207,114,142,186,248,73,177,249,124,104,34,162,218,187,254,40,100,202,196,127,168,163,219,135,82,63,26,59,90,1,212,150,155,87,212,128,119,163,126,79,243,215,230,178,163,113,224,227,139,43,105,254,218,60,203,215,4,140,149,7,58,39,64,106,120,93,218,127,109,162,25,57,26,7,42,250,1,205,31,141,221,255,192,39,57,208,253,54,157,143,230,178,71,110,121,226,160,39,250,91,65,252,218,115,212,175,205,179,156,163,202,82,198,209,252,181,137,230,213,30,243,32,68,214,202,10,134,223,148,209,251,133,49,246,132,225,171,125,232,149,231,207,168,101,79,82,104,253,218,92,203,57,254,207,6,108,167,239,127,109,30,251,75,8,50,126,101,83,39,127,132,12,54,142,38,90,84,123,237,164,62,242,211,109,157,212,7,217,55,205,254,24,99,71,227,64,249,148,166,180,126,109,46,251,158,207,207,155,243,251,61,222,175,242,71,100,209,250,181,121,150,111,29,153,235,228,20,222,246,39,167,101,208,251,183,54,241,38,57,26,71,137,75,75,169,31,205,242,176,172,190,242,233,169,90,240,223,20,122,62,161,185,236,104,28,24,209,43,130,174,63,109,158,229,127,245,141,144,201,93,134,67,72,65,87,218,127,109,162,37,121,223,227,189,141,183,27,58,8,34,138,162,105,255,209,44,231,64,202,33,122,190,160,185,236,97,35,235,27,95,249,125,83,56,233,167,231,51,154,199,190,161,207,77,7,143,185,209,33,8,50,159,208,247,39,154,136,172,118,142,250,234,88,127,218,127,52,246,146,70,123,141,215,46,241,84,42,52,162,245,107,115,217,71,77,140,241,225,49,63,44,47,84,71,78,92,54,243,67,243,44,231,248,63,25,116,197,103,250,181,137,86,213,190,48,98,138,204,61,93,7,210,212,51,234,71,99,71,227,64,197,237,145,180,126,109,46,251,171,47,146,36,7,158,135,241,253,175,205,99,79,91,28,107,188,118,199,238,112,103,145,160,249,107,19,173,171,157,3,103,206,222,166,126,52,246,172,242,90,198,79,222,171,3,35,79,242,251,183,54,215,242,209,9,25,178,40,61,10,114,74,175,209,252,181,121,236,104,28,184,225,227,247,63,109,162,13,121,242,148,24,227,107,35,123,64,101,54,189,191,27,99,47,122,84,230,224,49,11,182,182,128,73,93,139,169,95,155,107,57,71,93,201,237,69,253,218,60,203,235,116,201,148,235,71,245,130,183,111,52,162,126,109,34,138,252,194,232,104,137,126,233,233,120,232,42,7,80,63,154,229,28,248,75,78,41,245,107,115,45,31,58,118,176,12,29,147,0,249,110,109,154,191,54,143,29,141,3,167,139,219,82,191,54,17,77,222,243,119,189,141,239,88,151,4,89,130,246,207,24,123,211,95,37,74,60,102,64,64,111,40,252,180,14,237,191,54,215,114,14,172,90,66,191,15,208,60,203,55,173,31,47,75,130,210,33,108,76,18,245,107,19,109,201,187,79,78,50,14,117,211,32,44,156,223,255,209,44,231,192,91,59,223,160,126,109,46,251,232,69,87,28,244,138,65,129,240,209,198,143,105,254,218,60,203,3,250,45,148,51,139,43,213,235,163,244,251,21,77,180,35,71,227,192,212,245,243,169,31,141,125,250,179,10,115,254,139,139,241,176,127,87,3,234,215,230,90,206,81,121,17,217,212,175,205,99,95,89,62,215,225,168,207,58,207,240,155,126,109,34,134,124,203,212,23,62,244,90,33,160,250,125,209,198,184,49,203,57,254,37,1,179,232,255,63,180,185,150,47,107,48,221,25,127,124,156,202,124,60,192,156,143,230,177,163,101,165,118,146,17,81,71,85,66,9,253,62,70,19,237,249,254,209,198,129,125,211,30,209,253,135,214,222,188,134,219,174,180,251,255,15,171,153,181,186,224,172,104,36,64,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,64,39,0,0,58,13,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,64,39,0,0,79,4,0,0,0,0,0,0,235,8,0,0,0,0,0,0,39,0,0,0,120,156,69,210,105,108,140,81,20,6,224,15,221,40,186,153,182,150,106,171,106,82,101,180,73,169,14,223,61,51,247,86,168,218,74,205,88,6,9,29,26,42,182,148,84,83,91,74,144,210,160,150,137,109,136,45,72,137,84,53,177,38,150,254,104,98,233,31,148,80,90,75,162,182,8,83,147,216,206,188,19,189,201,205,253,241,228,61,39,57,247,104,154,255,116,194,221,229,121,47,150,237,12,164,176,124,143,156,240,117,153,252,111,157,241,94,206,104,19,85,222,28,106,255,22,161,62,169,106,56,91,23,120,126,233,25,225,205,44,166,216,18,163,74,212,52,229,115,182,0,184,169,80,232,236,141,179,79,203,162,172,241,200,179,5,194,119,91,171,245,85,27,34,68,91,245,55,107,182,115,133,197,231,108,65,29,254,182,246,190,184,237,118,201,61,149,13,86,159,179,5,195,43,31,30,18,211,66,215,209,144,177,66,25,14,199,163,63,91,8,252,126,68,157,207,39,76,254,36,215,174,41,70,127,182,174,240,43,163,155,68,251,198,213,100,126,156,166,10,206,132,32,207,214,13,94,94,242,70,20,95,119,80,217,217,129,170,57,165,21,121,182,80,248,159,40,151,126,112,159,153,172,7,235,228,139,165,2,206,214,29,206,182,50,172,148,26,182,58,84,77,139,5,245,217,122,192,223,213,109,22,57,11,157,116,183,62,81,101,68,121,145,103,235,9,247,220,78,214,155,243,114,201,190,239,139,108,204,244,207,159,45,12,126,45,112,187,200,215,50,168,254,108,147,92,172,38,194,217,194,253,243,9,213,168,104,190,147,114,61,57,234,249,231,100,244,103,139,128,47,50,54,232,236,86,115,141,180,205,153,141,60,91,100,135,127,175,140,167,177,244,72,110,175,152,6,103,139,130,199,229,221,16,73,173,49,20,180,75,83,237,55,183,192,217,122,249,247,99,192,69,49,233,164,141,66,146,13,106,173,235,46,156,205,208,225,253,92,83,73,203,77,85,115,31,255,130,179,69,195,67,237,6,170,50,77,166,17,181,67,148,169,180,13,206,22,3,79,121,184,74,180,100,228,144,190,247,183,52,91,14,192,217,98,253,255,179,248,176,232,53,114,58,189,234,20,166,234,106,174,194,217,122,195,211,95,106,180,39,96,30,185,111,25,213,241,161,31,224,108,125,224,253,79,124,22,150,15,58,45,112,197,168,146,25,254,60,91,95,184,87,6,235,65,163,179,169,79,189,73,118,47,63,133,253,100,235,7,191,120,236,145,207,109,219,110,72,119,186,19,121,182,56,248,73,123,185,56,167,219,232,200,121,179,154,83,17,142,255,97,235,15,119,101,183,136,49,119,70,209,159,166,72,85,145,112,1,121,182,120,248,172,130,65,122,89,236,18,122,145,122,76,38,212,102,193,217,18,224,51,55,165,18,123,109,156,84,61,15,24,80,159,45,17,94,232,233,66,142,145,78,122,118,42,93,69,30,237,12,103,27,0,175,249,248,84,132,164,45,162,229,230,20,149,182,227,7,234,179,37,193,27,29,73,52,101,222,68,202,44,202,82,91,220,193,200,179,13,132,71,43,51,29,218,63,159,154,108,227,85,213,194,68,56,91,50,220,177,254,137,8,24,87,70,94,251,48,101,127,30,8,103,27,4,63,81,244,83,103,31,252,238,129,188,151,231,159,31,155,177,195,11,243,77,20,126,252,146,116,191,30,14,255,103,22,35,214,156,205,127,173,151,150,126,180,252,5,243,184,9,13,224,172,104,36,64,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,64,39,0,0,169,16,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,64,39,0,0,146,13,0,0,0,0,0,0,23,3,0,0,0,0,0,0,9,0,0,0,120,156,99,96,128,2,22,8,214,203,176,179,253,101,94,106,239,96,200,228,146,114,109,145,51,92,142,3,66,79,114,92,103,27,204,93,103,223,25,117,193,185,202,168,10,34,15,146,227,129,200,255,23,158,105,91,196,95,109,207,57,77,217,197,97,222,87,136,60,72,78,0,34,255,253,176,170,173,103,90,170,253,139,112,17,23,217,205,215,32,242,32,57,17,136,124,186,250,105,219,156,164,84,251,76,203,55,206,51,62,119,64,228,65,114,18,16,121,205,11,197,118,34,22,161,246,185,147,37,93,30,76,60,15,145,7,201,201,64,228,127,57,177,219,78,102,137,183,183,210,120,234,252,103,253,4,136,60,72,78,1,34,31,149,162,102,91,43,145,109,127,129,79,194,69,86,16,106,63,72,78,5,34,191,229,237,77,59,14,131,116,123,137,149,230,46,134,187,57,93,192,242,64,57,7,117,160,51,153,25,24,150,228,252,180,101,241,168,181,127,127,96,175,243,61,145,18,103,0,146,176,66,33,224,172,104,36,64,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,64,39,0,0,234,17,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,64,39,0,0,1,17,0,0,0,0,0,0,233,0,0,0,0,0,0,0,2,0,0,0,120,156,99,96,128,2,1,8,254,126,88,213,54,152,187,206,94,118,50,135,171,189,192,110,23,184,156,2,132,254,229,196,110,91,43,145,109,127,100,10,155,235,146,139,235,33,242,64,57,7,117,160,52,59,3,195,146,156,159,182,44,30,181,246,39,92,186,93,50,14,184,185,0,0,9,201,20,38,224,172,104,36,64,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,64,39,0,0,144,18,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,64,39,0,0,66,18,0,0,0,0,0,0,78,0,0,0,0,0,0,0])
)
,"name":"User Data"},{"name":"Genome","twoBitURI":"http://www.biodalliance.org/datasets/hg38.2bit","tier_type":"sequence"}],"coordSystem":{"taxon":9606,"ucscName":"hg38","speciesName":"Human","auth":"GRCh","version":"38"}}
        );

Where the array is an inline BigWig file, but it tells me the magic number is incorrect (which I've checked and in the above array the magic number is actually correct (0x888FFC26) according to my hexdump:

N82106:GenomeBrowsers bward$ hexdump mybigwig.bw
0000000 26 fc 8f 88

Is it possible to inline BigBed and BigWig in a similar way to the Inline BED test as above?

dasmoth commented 6 years ago

The reason you're getting a "bad magic" error is that the Blob constructor doesn't treat arrays of ints as bytes in the way you might hope -- It's actually converting them to strings!

The solution to this is to do:

          new Blob([new Uint8Array([38, 252, 143, 136, ...])])

There are potentially a couple of other issues here. The first is that Biodalliance needs random access to the file, which won't necessarily work with blob: URLs. There's actually an easy way around this:

           sources: [
                {
                      name: 'User Data',
                      bwgBlob: new Blob(...)
                }
          ]

This was originally an internal mechanism which is used when you create a track from a local file (files appear inside the browser as Blobs), but does work with arbitrary Blobs, so I'll add this to the docs next time I'm in the area.

This got your example mostly working for me. I did run into a problem when browsing beyond the end of the data. I need to look into this more closely -- it might imply a truncated bigWig file, but it's not impossible that it could be a Biodalliance error. I'll investigate further...

dasmoth commented 6 years ago

Okay, there was a boundary-condition bug which could affect a few very small bigwig files (including yours). Fixed now.

TransGirlCodes commented 6 years ago

Wow that's brilliant, thank you.

The reason I find the ability to inline some of these things desirable, is because I'm building a package https://github.com/BioJulia/GenomeBrowsers.jl/tree/updates, so our users can see their results in Dalliance in Jupyter.

By default a local https server gets spun up in the background, keeping the data, like the array above in memory, and serves it to the Dalliance instance in the notebook. This is fine until you want to look at the notebook in a viewer, or if the notebook is converted to reveal.js slides.

Then you need to either, recreate the server that got spun up (my expeirence with fellow biologists with no interest in computers is they won't bother), or have dalliance read data from local files, or inline all the data.

dasmoth commented 6 years ago

Thanks Ben, sounds like a great project, and some extra encouragement for me to give Julia a try one of these days.

If there's anything we can do Biodalliance-side to better support this sort of use case, please do shout. I certainly agree that avoiding server restarts and the like is very worthwhile. For more general discussion about this kind of thing, the mailing list (biodalliance-dev@googlegroups.com) might be a better format.

TransGirlCodes commented 6 years ago

Thanks @dasmoth, one of the things I'm curious of is support for custom genomes. I work at an institute that does genomic sequencing and research of some interesting organisms.

Say I have a FASTA file of some reference genome, and then some BAMs, some SAMs and so on, and I want to visualise them, I know I can enter the BAM, SAM and other track datasets.

I know that Dalliance will accept a TwoBit URI or Blob for the actual sequence itself, but I was wondering about the other parameters like: "coordSystem":{"taxon":9606,"ucscName":"hg38","speciesName":"Human","auth":"GRCh","version":"38"}. Sometimes genomes we sequence and species we work on might not nessecerily have a version, a ucscName or taxon code if it's not a model organism that has been worked on a lot or isn't on the DAS registry and other hubs and databases and so on. Would this be an issue for someone who just wanted to load their FASTA reference sequence and some alignment / annotation files?

dasmoth / dalliance

local data sources #174