When looking for the most similar friend of every user, I load the JSON object of every (61,800) user and each of their friend (between 0 and 5000) in memory.
Therefore, the memory consumption is too large and I have to find a solution to solve this problem. Two ideas:
process the user one by one instead of in bulk (ie. load one user and his/her friends, find the most similar friend of this user, do the same with the next user);
When looking for the most similar friend of every user, I load the JSON object of every (61,800) user and each of their friend (between 0 and 5000) in memory. Therefore, the memory consumption is too large and I have to find a solution to solve this problem. Two ideas: