UCSF-DSCOLAB / data_processing_pipelines

A repository to store the existing pipelines to process the various CoLabs datasets
0 stars 1 forks source link

Bulk rnaseq dynamic resource allocation #16

Closed AlaaALatif closed 11 months ago

AlaaALatif commented 1 year ago
erflynn commented 1 year ago

This looks great! very exciting to be making the switch to more dynamic resource allocation. Will test it out more when I am back week of 10/16.

erflynn commented 11 months ago

@AlaaALatif -- update here, tested on files 0.75-18.5GB and this completed. I think fine to merge, but you may want to fine tune some steps more for less resources if not needed.

AlaaALatif commented 11 months ago

Tested branch pipeline against a dataset with 2 samples and another dataset with 1 sample only, without any errors.

@erflynn thank you for providing resource usage information from your dataset, and fixing issues with dynamic resource allocation @amadeovezz thank you for writing a bare-bones example of dynamic resource allocation using Nextflow, and testing the pipeline @lzemelko thank you for fixing a bug to handle datasets consisting of one sample only, and testing the pipeline

Looking good to me! Going to merge this and delete the branch shortly after.