search

pjgreer / ukb-rap-tools

Scripts and workflows for use analyzing UK Biobank data from the DNANexus Research Analysis Platform
45 stars 9 forks source link

Is there a way to determine the type of instance to use? #4

Closed dianacornejo closed 1 year ago

dianacornejo commented 1 year ago

@pjgreer I wonder if you have find a way to see a report from the peak memory/storage usage to select more efficiently the type of instance to use? thank you

pjgreer commented 1 year ago

Unfortunately it is a bit of trial and error. You can look through the log of a job that fails due to memory issues. That is how I know that certain chromosomes require more disk space during the pre-processing phases. For GWAS, RAM and disk space is pretty easy to estimate. For rare variant analysis, the wall time is based on the number of genes in a test panel, the number of variants in those genes, and the number of subjects in the analysis.