rcavalcante
commented
4 years ago Hi Stephanie,
Thanks for downloading the updated methylSig and giving it a spin. I tried methylSig on one of the larger datasets I have lying around (WGBS with 17M CpGs after filtering) and I downsampled it to 6M to be similar in size to yours.
I got similarly fast results with 10K and 100K downsampling. The 1M downsampling was also fast (would have taken longer to make some coffee). It's looking like the 6M finishes in a reasonable amount of time as well (<20m). Note, I am not using local information.
Using local information does inherently take longer because of the operations needed to figure out which sites are nearby. I did some benchmarking using system.time():
1K CpGs
> system.time(diff_methylsig(
+ bs = bs[1:1000],
+ group_column = 'tissue',
+ comparison_groups = c('case' = 'liver', 'control' = 'blood'),
+ disp_groups = c('case' = TRUE, 'control' = TRUE),
+ local_window_size = 0,
+ t_approx = TRUE,
+ n_cores = 10))
user system elapsed
1.012 4.329 2.498
>
> system.time(diff_methylsig(
+ bs = bs[1:1000],
+ group_column = 'tissue',
+ comparison_groups = c('case' = 'liver', 'control' = 'blood'),
+ disp_groups = c('case' = TRUE, 'control' = TRUE),
+ local_window_size = 200,
+ t_approx = TRUE,
+ n_cores = 10))
user system elapsed
5.906 7.090 3.149 10K CpGs
> system.time(diff_methylsig(
+ bs = bs[1:10000],
+ group_column = 'tissue',
+ comparison_groups = c('case' = 'liver', 'control' = 'blood'),
+ disp_groups = c('case' = TRUE, 'control' = TRUE),
+ local_window_size = 0,
+ t_approx = TRUE,
+ n_cores = 10))
user system elapsed
9.884 7.263 3.681
>
> system.time(diff_methylsig(
+ bs = bs[1:10000],
+ group_column = 'tissue',
+ comparison_groups = c('case' = 'liver', 'control' = 'blood'),
+ disp_groups = c('case' = TRUE, 'control' = TRUE),
+ local_window_size = 200,
+ t_approx = TRUE,
+ n_cores = 10))
user system elapsed
61.449 13.705 10.876 100K CpGs
> system.time(diff_methylsig(
+ bs = bs[1:100000],
+ group_column = 'tissue',
+ comparison_groups = c('case' = 'liver', 'control' = 'blood'),
+ disp_groups = c('case' = TRUE, 'control' = TRUE),
+ local_window_size = 0,
+ t_approx = TRUE,
+ n_cores = 10))
user system elapsed
119.481 18.577 16.162
>
> system.time(diff_methylsig(
+ bs = bs[1:100000],
+ group_column = 'tissue',
+ comparison_groups = c('case' = 'liver', 'control' = 'blood'),
+ disp_groups = c('case' = TRUE, 'control' = TRUE),
+ local_window_size = 200,
+ t_approx = TRUE,
+ n_cores = 10))
user system elapsed
816.593 62.267 92.880 6M CpGs
> system.time(diff_methylsig(
+ bs = bs[1:6000000],
+ group_column = 'tissue',
+ comparison_groups = c('case' = 'liver', 'control' = 'blood'),
+ disp_groups = c('case' = TRUE, 'control' = TRUE),
+ local_window_size = 0,
+ t_approx = TRUE,
+ n_cores = 10))
user system elapsed
7017.760 374.039 863.185 Notice that the increase is non-linear for the same analysis modulo using local information. Given that it takes 6M CpGs without local information 863.185s / 60s = ~14.38m, I'm sure the local information test will be in the hours. (I'm still waiting on mine, but figured I should respond before it's done.)
Since I'm seeing a "reasonable" run time without local information for data about your size, do you mind trying it with local_window_size = 0 and letting me know if you get about the same?
It might be worthwhile to investigate how many CpGs fall within 200bp of another one in your data. I know it's ERRBS, so it'll likely be higher than WGBS, but maybe it's less than you think and so ditching the local information is fine. Also, if the coverage tends to be quite high perhaps that's another reason to skip local information.
One last thought. I'm noticing, when looking at memory usage, that each child process uses 10-15GB of RAM, and spread over 10 child processes and the parent process, that's 110-165GB total. This is when I only have the bsseq object loaded and nothing else. If you have other large things loaded in your R environment, you may have even more memory usage. You helpfully mentioned that your server has 192GB RAM, and I wonder if perhaps you're running out of memory? Either by this job alone, or with other people running things on your server simultaneously. To remedy this, you'll likely have to use fewer cores.
Thanks, Raymond
srialle
shawpa
Dear Raymond,
I am trying to run methylSig on RRBS data and I have a problem with diff_methylsig() function. I have a BSobj object with 5,791,812 cytosines, which I give as input to diff_methylsig(). When I extract the 10,000 first positions the function runs well (in 8 seconds), but when I try to run on the whole object, it runs for hours (more than 6 hours now) and seems to never end. I am working on a linux machine with 192G RAM and 48 available threads.
Here is the code I launched:
bs = filter_loci_by_group_coverage(bs = bs,group_column = 'Condition',min_samples_per_group)bspData(bs)diff_gr = diff_methylsig( bs = bs, group_column = 'Condition', comparison_groups = comp, disp_groups = c('case' = TRUE, 'control' = TRUE), local_window_size = 200, t_approx = TRUE, n_cores = 10)Do you have an idea while the code doesn't seem to finish?
Thank you,
Regards,
Stephanie
sessionInfo()