Closed liamgd closed 1 year ago
Combinations of gene expression chunking and methylation chunking:
Gene expression chunking, methylation chunking: save files with name '{methylation chunk}-{gene expression chunk}.csv' during mlr
Gene expression chunking, no methylation chunking: save files with name '1-{gene expression chunk}.csv' during mlr
No gene expression chunking, methylation chunking: ~yield results in generator in mlr; save results as they yield in command as '{methylation chunk}.csv'~ save files with name '{methylation chunk}-1.csv' during mlr
No gene expression chunking, no methylation chunking: return results in mlr; save results in command as 'out.csv'
From these tests, we can see that tecpg run mlr
and tecpg run mlr --trans
both failed due to an out of memory error. The methylation chunking avoided this error. However, if we compare tecpg run mlr --cis
(which succeeded) and tecpg run mlr -m 10000 --cis
, we see a dramatic performance decrease. Gene expression loci chunking results in a mild performance decrease in some cases but far less than methylation chunking.
Only use methylation chunking if it is absolutely necessary to avoid an out of memory error and still maintain a decent gene expression loci per chunk. In general, from these tests, gene expression loci chunking is far less negatively impactful on performance than methylation chunking. However, low gene expression loci per chunk values could also result in massive performance decreases. Minimize the use of methylation chunking, but also ensure that gene expression chunks are sufficiently large. Saving to the disk is also an expensive operation. The number of saves is equal to the number of methylation chunks multiplied by the number of gene expression chunks. Avoid chunking unless it is needed, and use gene expression chunking over methylation chunking.
Added in ea6d9f9.
Currently, MLR allows for chunking to combat memory limitations. This splits the gene expression matrix into chunks and clears memory after saving each chunk. However, the algorithm allocates constants before the main loop with a memory size proportional to the number of methylation loci. Therefore, given enough methylation loci and samples, the constants could try to allocate more memory than there is available, rendering the algorithm unusable for that input. If the constants memory exceeds the available memory, even the lowest chunk size of one gene locus per chunk would run into a memory error.
To get around this, the methylation matrix also has to be split into chunks. Each of these chunks can then be fed into the previous MLR function. This way, constants memory would be reduced, allowing for more gene expression loci per chunk. This decrease in memory sacrifices parallelization, as fewer values would be computed in parallel. The name "2D chunking" refers to chunking along two dimensions: methylation and gene expression. This should get around memory limits for large methylation inputs.