search

uclasystem / dorylus

Dorylus: Affordable, Scalable, and Accurate GNN Training
77 stars 12 forks source link

Question about the loading of Friendster #5

Open TracyRixner opened 2 years ago

TracyRixner commented 2 years ago

@ivanium @kevalvora @josehu07 Hi authors, Good work! I use the c5n.4xlarge instance. However, I find the command to prepare dataset

./prepare <PathToRawGraph> <Undirected? (0/1)> <NumVertices> <NumPartitions> <PathToRawFeatures> <DimFeatures> <PathToRawLabels> <LabelKinds>

There is an Out of Memory error.(42GB is not enough) To run friendster, do I need to use a large memory node to make the partition first? And then distribute those partition to each node, right? How large should the RAM be?

Thanks for the help!

ivanium commented 2 years ago

Hi Tracy,

Thank you for your interest in Dorylus.

Yes, you are right that partitioning large graphs can take a huge amount of memory. I didn't have concrete memory consumption numbers right now, but I would suggest to use AWS r-series instances, specifically I recommend at least 256GB memory for friendster. For example, r6i.8xlarge is a good choice.

Hope this helps, and good luck!