search

SUwonglab / scABC

19 stars 5 forks source link

moving to sparse matrix format #7

Open timydaley opened 5 years ago

timydaley commented 5 years ago

@linzx06 I've run into difficulties applying scABC to data coming from the new 10X machines. It's too big! I'm trying to move scABC to use sparse matrices to save space and memory. I've done some tweaks, specifically to reading in the data and computing the spearman correlation (see https://github.com/timydaley/SparseSpearmanCorrelation for my solution for computing the spearman correlation for sparse matrices). Can you check the computation of the p-values? I think the only place you use the matrix is in computing the crossprod, and the Matrix package has a built-in overloaded crossprod function. I think the only thing we need to do is import that function. Can you check this over when you get a chance? Thanks.

linzx06 commented 5 years ago

Hi Tim,

The matrix mu is the same dimension as the data matrix, and it is not sparse. That one can cause some trouble too. Is it an issue of memory?

Best, Z

On Thu, Feb 7, 2019 at 8:59 AM Timothy Daley notifications@github.com wrote:

@linzx06 https://github.com/linzx06 I've run into difficulties applying scABC to data coming from the new 10X machines. It's too big! I'm trying to move scABC to use sparse matrices to save space and memory. I've done some tweaks, specifically to reading in the data and computing the spearman correlation (see https://github.com/timydaley/SparseSpearmanCorrelation for my solution for computing the spearman correlation for sparse matrices). Can you check the computation of the p-values? I think the only place you use the matrix is in computing the crossprod, and the Matrix package has a built-in overloaded crossprod function. I think the only thing we need to do is import that function. Can you check this over when you get a chance? Thanks.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/SUwonglab/scABC/issues/7, or mute the thread https://github.com/notifications/unsubscribe-auth/ALQrSRRHxnrLuFzo1U3fUF-_79lEkoR1ks5vK3prgaJpZM4amami .

timydaley commented 5 years ago

Yes it's an issue of memory. But mu should not be sparse, right?

timydaley commented 5 years ago

I'll test out all the parts and make sure everything works. But this is good knowledge for what I need to look for. Thanks Z.

linzx06 commented 5 years ago

Sure Tim. Let me know if you need anything else.

Best, Z

On Feb 10, 2019, at 1:03 AM, Timothy Daley notifications@github.com wrote:

I'll test out all the parts and make sure everything works. But this is good knowledge for what I need to look for. Thanks Z.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

MahdiZ11 commented 5 years ago

Hi Tim, I think we moved to sparse matrix format. Right?

timydaley commented 5 years ago

Not finished yet. Had to work on other stuff. I created a stable version before I messed around with the sparse matrix format. Look under the release tab.

MahdiZ11 commented 5 years ago

Okay. Thanks!