Is there a fast way of doing pairwise distance correlation (dcor.distance_correlation)

[amjass12]

Hi,

I am trying to do a pairwise distance correlation for every column in a pandas dataframe of shape (1000, 10000) - i want to do a pairwise correlation of all columns (so 10k pairwise correlations, each column by every other column.)

if i run the following code:

dist_corr = lambda column1, column2: dcor.distance_correlation(column1, column2)
d_corr = df.apply(lambda col1: df.apply(lambda col2: dist_corr(col1, col2)))

this takes far too long, many many hours and in some cases doesn't finish. Is there an implementation that is more optimised? any advice would be much appreciated.

thank you

[vnmabus]

You can try using rowwise (https://dcor.readthedocs.io/en/latest/functions/dcor.rowwise.html#dcor.rowwise), probably with "parallel" compile mode, as in the last section of https://dcor.readthedocs.io/en/latest/performance.html. Tell me if that works for you.

[amjass12]

Hi @vnmabus ,

I am afraid this doesn't work for me - I tried doing pairwise distance corr from columns on a np.array as follows:

d_cor = np.apply_along_axis(lambda col1: np.apply_along_axis(lambda col2: dcor.rowwise(dcor.distance_correlation, [col1], [col2], compile_mode=dcor.CompileMode.COMPILE_PARALLEL), axis = 0, arr=df), axis =0, arr=df)

Running all night and still not completed in the morning - can you see anything wrong in the code that might be causing it not to work optimally (or another implementation code wise that may make it run faster)? thank you

[vnmabus]

rowwise can only truly parallelize if you give it several columns. You want to pass to it all column combinations. If you for example had 3 columns, you would have to pass: [col1, col1, col1, col2, col2, col2, col3, col3, col3], [col1, col2, col3, col1, col2, col3, col1, col2, col3].

As the correlation is symmetric you can also avoid computing almost half of the combinations. There is still a memory problem, as the same column is copied several times. As dcor uses the CPUs for parallelization, you can always pass chunks with the same size as the total number of CPUs in your system, in order to keep memory usage low.

But at the end, computing a measure pairwise between 10000 columns is going to take time. We are talking about 50.005.000 operations here. In a machine with 32 cores and perfect parallelization that would be 1.562.657 parallel runs. Even if each parallel run takes 0.1 seconds you will have to run that for almost two days (please repeat the math yourself as I did that quickly).

[amjass12]

thank you for the quick response and explanation!

[vnmabus]

Did that work for you?

[amjass12]

Hi @vnmabus ,

thank you for following up,

I am afraid not - I will run it as per my code above on a HPC to see if this can be left alone and run for a few days!

[vnmabus]

I recommend you to try to estimate the time that it will take, based on the number of operations and the time spend in one operation. Then if the estimated time is too high, try implementing the improvements I mentioned before. Otherwise you could potentially wait for weeks without knowing it.

[amjass12]

yes, i think that time you calculated is accurate - I will try especially the running in chunks as you suggested this is sensible