search

borenstein-lab / burrito

A visualization tool for exploratory data analysis of metagenomic data
https://elbo-spice.gs.washington.edu/shiny/burrito/
GNU General Public License v3.0
35 stars 9 forks source link

Processing time #4

Open ghost opened 5 years ago

ghost commented 5 years ago

Hi, does it takes longer time for the Burrito analysis to complete ? I waited for about an hour but still doesnt show any progress image

cnoecker commented 5 years ago

Hello, BURRITO can take a long time to load if you have a very large dataset (either many samples or many taxa). It can occasionally time out and freeze, particularly if you have a weak internet connection. I would suggest trying again with a smaller dataset - either choose a subset of samples or filter out low-abundance OTUs (which should have relatively little impact on the visualization).

najouamghazli commented 4 years ago

Hello,

I do have the same problem and even worse :)

I generated a contrib table using PICRUSt2 (#5 ) that contains 16,418,850 lines and uploaded it to BURRITO ... it's been 5 days since it's running with no improvement (still at Step2: Calcuating attributions) Is this normal ?

Thank you,

Najoua

engal commented 4 years ago

Hi Najoua,

It looks like your connection to the server dropped at some point, as nothing is currently running. I'm trying out a fix, so if you could retry running BURRITO with your full data and let me know the results, that would be great!

Quick note, you may need to restart your browser.

najouamghazli commented 4 years ago

Okay, I just restarted my browser and rerun BURRITO. I'll keep you up to date.

najouamghazli commented 4 years ago

Still at the 2nd step ... From time to time it goes back to step 1 for less than 1 second and then come back to step 2.

I precise that my contrib file has 23,829,462 lines !

engal commented 4 years ago

Sorry for the delayed response, I missed your update.

The two most likely possibilities are either:

A) We have noticed a recent increase in BURRITO usage that may be resulting in too high a server load and the subsequent failure of BURRITO for some users. However, we are currently in the process of migrating BURRITO to a server with enough resources to handle this increased load, so this issue should be resolved soon.

B) Your connection may have been interrupted, causing BURRITO to freeze. As was mentioned earlier in this thread, this could be due to a weak connection. If you are not already, I would recommend using a wired connection rather than WiFi.

I'll update this thread once we've finished our server upgrade.

najouamghazli commented 4 years ago

I think that the first possibilty is more probable, because I am using a wired connection.

Just a suggestion: couldn't it be possible to use BURRITO as a linux command or as an R package ?

engal commented 4 years ago

Hi,

Sorry for the long delay, but we've finally completed our server migration. The new address for BURRITO is http://elbo-spice.cs.tau.ac.il/shiny/burrito/. Please give this a try.

Regarding your second question, BURRITO is technically available as this GitHub project, and you can clone the repository. Theoretically, you could run it as a Shiny app in RStudio, though we have not done in-depth testing for this use case, so we can't make any guarantees. Additionally, you will be unable to perform the automatic contribution calculation without supplying your own underlying genomic content database (we do not include the database in the GitHub project). BURRITO was not originally designed to serve as a stand-alone package, though we may consider this for future develoment.

Best, Alex

Qepanna commented 4 years ago

Hi! I'm also having an issue regarding processing time: it ran for 3 days and crashed. It's was still at the 2nd step (going back to first step for less than a second every now and then). I'm also using a wired connection and I'm using the new server...

I used PICRUSt2 and Qiime2 with SILVA to obtain my results. I wonder if I'm using the right files. I attached the files I obtained in this message, except for the functional attributions table because I can't attach a tsv file but you can see it here: https://drive.google.com/file/d/1TU56CxUCbYuud1Cv2jpDr9zmHFXukwBH/view?usp=sharing

metadata_file_16S.txt ESV_table_abundance.txt ESV_table_taxonomy.txt

Thank you!

engal commented 4 years ago

Hi,

Depending on your location relative to the server, there can still be significant delays and potential connection problems when uploading very large datasets. I would recommend first testing that your files work with BURRITO by using a subset of your data. For example, generate test versions of your files by filtering the abundance table down to 10-20 ESVs and then filtering your metagenome contribution table to only include contributions from those ESVs.

If that test works, then the issue is likely due to the size of your data. If this is the case, then you might still be able to use BURRITO but it would require you to filter out some ESVs to reduce the size of the data you are uploading. This could be done by removing ESVs that appear at very low abundance across samples, filtering out genes that are at consistently low abundance, or only visualizing a subset of your samples at any given time.

I hope that helps!

Qepanna commented 4 years ago

Hi,

I filtered my sequences and reran BURRITO with only 17 ESVs and it worked! I will filter my data the best I can to use this wonderfull tool, thank you very much!

P.S. It would be much easier to understand wich files are needed if it was more clearly specified which are optional and which are mendatory. Additionally, it would be great if it was specified which extentions are valid (.txt, .biom, .tsv, .csv, etc.) and if a compressed file can be used.

engal commented 4 years ago

Glad you got things working!

Thanks for the feedback, I'll look into adding those clarifications to the submission form.