bentsherman
commented
4 years ago @spficklin I took a quick look at the corrpower analytic and I think the poor performance has to do with how the analytic is iterating through the input data. It is iterating through every pairwise index the same way as similarity, and trying to read a gene pair for each index. This results in a lot of unnecessary work because the correlation matrix is very sparse.
If you look at other analytics such as export correlation matrix, extract, or conditional test you will see that these analytics iterate directly through the gene pairs in the input data, instead of testing every possible pairwise index. Of course this makes it harder to split up the work into uniform chunks but it seems you figured out how to do that with conditional test so I would refer to that analytic.
spficklin
The corrpower filter, which removes edges that don't have sufficient power, is too slow. Its much much slower than conditional filtering. Perhaps a way to speed it up is to keep a lookup table of the parameters and when the exact combination has already been performed just use the lookup table rather than recompute.