search

alexandrelevada / PCA-KL

PCA-KL: a relative entropy dimensionality reduction method for unsupervised metric learning
3 stars 0 forks source link

Regarding the problem of insufficient memory when using PCA-KL #1

Open thereblue opened 2 years ago

thereblue commented 2 years ago

I used the PCA-KL method proposed in your paper. This method has insufficient memory when performing KNN calculations. I used 600000 pieces of data, and the data generated a 600000*6000000 matrix. This matrix requires 2TiB of memory space. Is there a solution for this?

alexandrelevada commented 2 years ago

At the moment, the code is not optimized to deal with large datasets. In this version the program pre-allocates the memory for all the computations. We are currently working in a solution to this issue in order to release a new version.

thereblue commented 2 years ago

Today, I made some modifications to the code locally on my computer. Since esparsa.toarry() in Pycharm will be matrixed, and then memory will be allocated, resulting in memory overflow. Therefore, I discarded this step in advance, and then used esparsa[i,:].toarry() in the corresponding vizinhos=A[i:] to reduce the direct memory allocation and obtain the dimensionality reduction result. I wonder if this method is feasible? Thank you for your reply and help, and I wish you a happy life and all the best.

------------------ 原始邮件 ------------------ 发件人: "alexandrelevada/PCA-KL" @.>; 发送时间: 2021年10月31日(星期天) 晚上7:21 @.>; @.**@.>; 主题: Re: [alexandrelevada/PCA-KL] Regarding the problem of insufficient memory when using PCA-KL (Issue #1)

At the moment, the code is not optimized to deal with large datasets. In this version the program pre-allocates the memory for all the computations. We are currently working in a solution to this issue in order to release a new version.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

alexandrelevada commented 2 years ago

I think that's a good idea, feel free to contribute with the code. I has been almost 2 years since I've made this code, so I really need to remember exactly how the code works. Anyway, feel free to improve it and release better versions. Thanks, have a happy life and all the best.

thereblue commented 2 years ago

Thanks, I am honored to be able to contribute to your open source code.

------------------ Original ------------------ From: Alexandre L. M. Levada @.> Date: Mon,Nov 1,2021 0:06 AM To: alexandrelevada/PCA-KL @.> Cc: thereblue @.>, Author @.> Subject: Re: [alexandrelevada/PCA-KL] Regarding the problem of insufficientmemory when using PCA-KL (Issue #1)