Regression full is majorly bottlenecked by the conversion from p-value GPU tensor to p-value CPU dataframe. This could be reduced by filtering the p-values prior to converting from tensor to dataframe. For this, not only does the P tensor need to be filtered such that P > p_thresh, but the indices that remain must also be stored when indexing the resultant dataframe.
liamgd
commented
1 year ago
Regression full is majorly bottlenecked by the conversion from p-value GPU tensor to p-value CPU dataframe. This could be reduced by filtering the p-values prior to converting from tensor to dataframe. For this, not only does the P tensor need to be filtered such that P > p_thresh, but the indices that remain must also be stored when indexing the resultant dataframe.