JBrowse2 tracks deconvolution

nekrut commented 6 days ago

VEuPathDB had JBrowse instances set up so that each config and tracks are done dynamically (not typical); REST service sits in front of database to write config files and extract data dynamically to display in JBrowse
For sequence and genes, .gff and .fasta; not accessed
There are many many tracks for plasmodium genome
PlasmoDB flat files are massive
Can see github repo where we could extract information from other data tracks and database features, can reverse engineer from here

nekrut commented 6 days ago

Since VEuPathDb is accessible now, can we do this via some kind of crawling @scottcain ?

nekrut commented 6 days ago

Contingent on our ability to query the db

maximilianh commented 6 days ago

Can't this be done via the jbrowser API endpoint, the one that jbrowse uses? Veupathdb staff told us how to do this...

On Thu, Oct 10, 2024 at 6:14 PM Anton Nekrutenko @.***> wrote:

Contingent on our ability to query the db

— Reply to this email directly, view it on GitHub https://github.com/galaxyproject/brc-analytics/issues/129#issuecomment-2405535100, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACL4TP2SNIMJXYHEJGCHSTZ22RWDAVCNFSM6AAAAABPXATNM6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMBVGUZTKMJQGA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

nekrut commented 6 days ago

Can't this be done via the jbrowser API endpoint, the one that jbrowse uses? Veupathdb staff told us how to do this... … On Thu, Oct 10, 2024 at 6:14 PM Anton Nekrutenko @.> wrote: Contingent on our ability to query the db — Reply to this email directly, view it on GitHub <#129 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACL4TP2SNIMJXYHEJGCHSTZ22RWDAVCNFSM6AAAAABPXATNM6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMBVGUZTKMJQGA . You are receiving this because you are subscribed to this thread.Message ID: @.>

We just discussed this with @scottcain

maximilianh commented 6 days ago

Was the conclusion that it's possible using the API? (sorry I had a meeting at exactly the same time and duration as the BRC meeting today)

scottcain commented 6 days ago

@maximilianh it depends on what we want to do. Since these tracks' data are now static from VEuPathDB (that is, they aren't being actively curated, which would be a reason you might want to run the tracks from a database query), using the database to serve up JBrowse data is (in my opinion) a little obnoxious, when the alternative is something like tabix indexed GFF, BigBed or BigWig (along with associated metadata in all cases). Shifting to static files sitting in a bucket or web server somewhere would make implementing JBrowse 2 a lot easier (and presumably other genome browsers 😉), and is better for the community, since it makes getting a whole dataset for a given track a lot easier.

The next question is, could we write a spider that crawled all of the JBrowse instances at VEuPathDB and extracts all of the data? Well, yes, I suppose we could, but it would be kind of an unfriendly thing to do. Rather, I think a reasonable approach is wait for a new hire in Sergei's group who knows this database pretty well and can (presumably) help us extract the track data and metadata from our instance of the database.

maximilianh commented 6 days ago

Yes, I meant a spider. I don't know how unfriendly that would be, depends on how fast the transfer is... but OK, if someone starts with Sergei, then I guess there is an easier way. And yes, I convert everything to bigBed.

On Thu, Oct 10, 2024 at 9:32 PM Scott Cain @.***> wrote:

@maximilianh https://github.com/maximilianh it depends on what we want to do. Since these tracks' data are now static from VEuPathDB (that is, they aren't being actively curated, which would be a reason you might want to run the tracks from a database query), using the database to serve up JBrowse data is (in my opinion) a little obnoxious, when the alternative is something like tabix indexed GFF, BigBed or BigWig (along with associated metadata in all cases). Shifting to static files sitting in a bucket or web server somewhere would make implementing JBrowse 2 a lot easier (and presumably other genome browsers 😉), and is better for the community, since it makes getting a whole dataset for a given track a lot easier.

The next question is, could we write a spider that crawled all of the JBrowse instances at VEuPathDB and extracts all of the data? Well, yes, I suppose we could, but it would be kind of an unfriendly thing to do. Rather, I think a reasonable approach is wait for a new hire in Sergei's group who knows this database pretty well and can (presumably) help us extract the track data and metadata from our instance of the database.

— Reply to this email directly, view it on GitHub https://github.com/galaxyproject/brc-analytics/issues/129#issuecomment-2405885835, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACL4TNJL7UK6ECFBJWBVH3Z23I47AVCNFSM6AAAAABPXATNM6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMBVHA4DKOBTGU . You are receiving this because you were mentioned.Message ID: @.***>

maximilianh commented 6 days ago

Thanks for the quick reply!

On Thu, Oct 10, 2024 at 10:30 PM Maximilian Haeussler @.***> wrote:

Yes, I meant a spider. I don't know how unfriendly that would be, depends on how fast the transfer is... but OK, if someone starts with Sergei, then I guess there is an easier way. And yes, I convert everything to bigBed.

On Thu, Oct 10, 2024 at 9:32 PM Scott Cain @.***> wrote:

@maximilianh https://github.com/maximilianh it depends on what we want to do. Since these tracks' data are now static from VEuPathDB (that is, they aren't being actively curated, which would be a reason you might want to run the tracks from a database query), using the database to serve up JBrowse data is (in my opinion) a little obnoxious, when the alternative is something like tabix indexed GFF, BigBed or BigWig (along with associated metadata in all cases). Shifting to static files sitting in a bucket or web server somewhere would make implementing JBrowse 2 a lot easier (and presumably other genome browsers 😉), and is better for the community, since it makes getting a whole dataset for a given track a lot easier.

The next question is, could we write a spider that crawled all of the JBrowse instances at VEuPathDB and extracts all of the data? Well, yes, I suppose we could, but it would be kind of an unfriendly thing to do. Rather, I think a reasonable approach is wait for a new hire in Sergei's group who knows this database pretty well and can (presumably) help us extract the track data and metadata from our instance of the database.

— Reply to this email directly, view it on GitHub https://github.com/galaxyproject/brc-analytics/issues/129#issuecomment-2405885835, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACL4TNJL7UK6ECFBJWBVH3Z23I47AVCNFSM6AAAAABPXATNM6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMBVHA4DKOBTGU . You are receiving this because you were mentioned.Message ID: @.***>

galaxyproject / brc-analytics

JBrowse2 tracks deconvolution #129