liamgd
commented
1 year ago Combinations of gene expression chunking and methylation chunking:
-
Gene expression chunking, methylation chunking: save files with name '{methylation chunk}-{gene expression chunk}.csv' during mlr
-
Gene expression chunking, no methylation chunking: save files with name '1-{gene expression chunk}.csv' during mlr
-
No gene expression chunking, methylation chunking: ~yield results in generator in mlr; save results as they yield in command as '{methylation chunk}.csv'~ save files with name '{methylation chunk}-1.csv' during mlr
-
No gene expression chunking, no methylation chunking: return results in mlr; save results in command as 'out.csv'
Currently, MLR allows for chunking to combat memory limitations. This splits the gene expression matrix into chunks and clears memory after saving each chunk. However, the algorithm allocates constants before the main loop with a memory size proportional to the number of methylation loci. Therefore, given enough methylation loci and samples, the constants could try to allocate more memory than there is available, rendering the algorithm unusable for that input. If the constants memory exceeds the available memory, even the lowest chunk size of one gene locus per chunk would run into a memory error.
To get around this, the methylation matrix also has to be split into chunks. Each of these chunks can then be fed into the previous MLR function. This way, constants memory would be reduced, allowing for more gene expression loci per chunk. This decrease in memory sacrifices parallelization, as fewer values would be computed in parallel. The name "2D chunking" refers to chunking along two dimensions: methylation and gene expression. This should get around memory limits for large methylation inputs.