search

CrawlScript / tf_geometric

Efficient and Friendly Graph Neural Network Library for TensorFlow 1.x and 2.x
GNU General Public License v3.0
491 stars 91 forks source link

[Issue] Distributed training OOM with large datasets #47

Open cymarechal-devoteam opened 1 year ago

cymarechal-devoteam commented 1 year ago

Hi,

I'm using tf_geometric in a distributed fashion for a node classification problem. My BatchGraph contains thousands of Graph's and spans over several hundreds of gigabytes. When using the GCN architecture, all my nodes run out of memory, despite the fact that I use "experimental_distribute_datasets_from_function" as provided in the sample codes. Isn't it supposed to distribute the dataset over all my nodes? I see a new parameter "num_or_size_splits" added to the API. Is this anything that could help me? How does it differentiate from the distributed setting?

Thank you,

hujunxianligong commented 1 year ago

Sorry for the late reply. I think tfg cannot address your problem. DGL may be a proper solution for you.