ogbn_papers100M load too much data into Ram such that zsh kills the process

snap-stanford / ogb

Benchmark datasets, data loaders, and evaluators for graph machine learning

https://ogb.stanford.edu

MIT License

1.89k stars 397 forks source link

ogbn_papers100M load too much data into Ram such that zsh kills the process #306

Closed zhuyuecai closed 2 years ago

zhuyuecai commented 2 years ago

Hi , I trying to load the ogbn_papers100M dataset in my macbook pro with 32G ram. After downloading the data, and trying to convert the raw data into ogb data format, Zsh kill the process with the following error message: 'miniconda3/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown'

Is there any fix or workaround?

weihua916 commented 2 years ago

Hi! You will need to use a larger RAM machine to handle such a large graph.