Current Version whether support Caching Neighbors of Important Vertices

[YijianLiu]

Hello, 1 year ago I see it doesn't support of caching neighbors. Whether current version support this function, if not, what are the main difficulties? Urgently need your answer, thanks a lot!

[Seventeen17]

We implement batch cache in this version, by prefetching neighbors and features for several batches and caching them in a TapeStore; For computation, the sampling and feature extraction operators are organized as DAG, and the computation is executed in parallel inter- and intra- DAGs, which substantially improves the sampling performance.

The batch cache achieves the desired performance in most of our sampling scenarios. If the capacity of the cache is sufficient and sampling is still a performance bottleneck situation, adding hotspot cache can also orthogonally improve the sampling performance.

[YijianLiu]

We implement batch cache in this version, by prefetching neighbors and features for several batches and caching them in a TapeStore; For computation, the sampling and feature extraction operators are organized as DAG, and the computation is executed in parallel inter- and intra- DAGs, which substantially improves the sampling performance.

The batch cache achieves the desired performance in most of our sampling scenarios. If the capacity of the cache is sufficient and sampling is still a performance bottleneck situation, adding hotspot cache can also orthogonally improve the sampling performance.

According to my understanding, it means that sampling and training are in parallel? When the first batch is training, it begins to prefetch next batch of neighbors and features?

[Seventeen17]

Yeah

[YijianLiu]

Yeah

Hello, I have a question that why not use pytorch DataLoader workers or tensorflow tf.data.Dataset.prefetch, I think both of them can also achieve prefetch for batches, and why you achieve again by c++

[LiSu]

As graphlearn only uses Pytorch for training, it needs its own dataloader to transform the sampling results into the formats that the trainer requires.