A Question about the Code

[shuoyinn]

Hello. I appreciate that your work is really good and influenced me a lot. But excuse me, could you tell me in pyg_autoscale/torch_geometric_autoscale/history.py, within the member of the class History, push(), the branch else, why is it self.emb[dst_o:dst_o + c] = x[src_o:src_o + c]? Where is the indices vector n_id? I thought self.emb, also "history" as mentioned in your paper, recorded node v's most recent embedding in its v-th block or self.emb[v]. So in History's push(), within the last elif, self.emb is assigned values from x as the batch feature vectors only for x in n_id. But why in else, it has nothing to do with n_id? Or I want to know, if it should be self.emb[n_id[dst_o:dst_o + c]] = x[src_o:src_o + c], or the offset and count are not exactly like what you've described in pyg_autoscale/torch_geometric_autoscale/models/base.py, within ScalableGNN: __call__() ? Since I can't find the source code of torch.ops.torch_sparse.partition (trust me, I have tried a lot )

[rusty1s]

The n_id case is handled in the branch before:

elif offset is None or count is None:
    self.emb[n_id] = x.to(self.emb.device)

The offset and count arguments are used in case we push embeddings to a contiguous region in memory. For example, if n_id is described by [0, 1, 2, 6, 7, 8, 9], then offset and count will be given as [0, 6] and [3, 4], respectively, and the memory transfer happens via:

self.emb[0:3] = x[0:3]
self.emb[6:10] = x[3:7]

This can be actually way faster than writing via a node index vector.

[shuoyinn]

Thank you very much for your response, it's very kind of you to help and answer me so soon. And I am really grateful for this! As for the code, with your help, now I can understand that the core is taking offset and count as an efficient version of n_id thus the latter is replaced by these two vectors, i.e., not used in this else branch.

But what about the comment in the function __call__() of class ScalableGNN in the module pyg_autoscale/torch_geometric_autoscale/models/base.py? There, it writes n_id = [0, 1, 5, 6, 7, 3, 4] and batch_size = 5, whereas offset = [0, 2, 5] and count = [2, 3]. I want to know, according to what you've said, whether the offset should be [0, 5] instead of [0, 2, 5] here, since it is actually used in the function push() within the branch else mentioned above. In fact, it is this comment that confuses me a lot.

I will introduce and present this fantastic paper and your idea of AutoSacle GNN in the seminar of my laboratory, so I would appreciate it very much if you could answer me.

[rusty1s]

Note that we only push the node embeddings of in-mini-batch nodes, i.e. the first 5 nodes of [0, 1, 5, 6, 7, 3, 4] which are grouped contiguously in memory as given by:

offset = [0, 2, 5]
count = [2, 3]

The remaining nodes are used to pull from histories, which are given by n_id[batch_size:].

[shuoyinn]

Thanks a lot for your response, again. Amm, well, I still cannot understand and I am sorry for that. Let me clarify and conclude my questions.

Q1.

(1) Case 1 Here, you gave me an example days ago, as follows: If

n_id := [0, 1, 2, 6, 7, 8, 9]
offset := [0, 6]
count := [3, 4]

Then, while storing embeddings into History,

self.emb[0:3] = x[0:3]
self.emb[6:10] = x[3:7]

(2) Case 2 But in that comment, if n_id := [0, 1, 5, 6, 7, 3, 4], offset = [0, 2, 5] and count = [2, 3]. While storing in-mini-batch node embeddings, it will be:

self.emb[0:2] = x[0:2]
self.emb[2:5] = x[2:5]   # Because `offset[1] == 2`.

In Case 2, I don't think self.emb[2] should be updated, and I think self.emb[5] should be assigned x[2] according to the n_id and principle of Case 1, am I right? So, like the former case, I think offset should be [0, 5] in Case 2.

Q2. A tiny question. I want to know, if in the first epoch, in the first GNN layer and for the first mini-batch, out-of-mini-batch is [2, 3], then what are the values of their historical embeddings or self.emb[2] and self.emb[3]? Empty i.e. 0s, or something else? Can we use these initial histories?

Q3. Another tiny one. Will historical embeddings be shared between different epochs? Or, for the histories computed and stored in epoch 1, will they be retained in epoch 2, or be emptied before epoch 2?

[rusty1s]

1) Yes, you are right. The comment is wrong and it should be offset=[0, 5]. I fixed it in master. 2) We use a single inference forward pass to initialize historical embeddings to their exact values before training starts. If you do not do this, they will get initialized to zero. 3) Yes, the values of histories refer to the embeddings acquired in the previous epoch.

[shuoyinn]

I really appreciate your help. Now I clearly understand and grasp the mechanism of GAS. Thank you, again!