alibaba / GraphScope

🔨 🍇 💻 🚀 GraphScope: A One-Stop Large-Scale Graph Computing System from Alibaba | 一站式图计算系统
https://graphscope.io
Apache License 2.0
3.17k stars 424 forks source link

feat(interactive): add `use-mmap-vector` and `build-csr-in-mem` option for bulk loading #3957

Closed liulx20 closed 1 week ago

liulx20 commented 1 week ago

When loading the edges of a graph, there are two stages involved:

During the first stage, the edges are stored in the form of triplets, which can lead to a certain amount of memory expansion, so the use-mmap-vectoroption is provided, mmap_vector utilizes mmap to map files, supporting runtime swapping memory to disk.

Constructing the CSR involves random reads and writes, we offer the build-csr-in-mem option, which allows CSR to be built in-memory to avoid extensive disk random read and write operations.