otiliastr
commented
3 years ago Hi Vijay,
Thanks for discovering this issue! I have made a pull request with a fix for the edge_iterator. See pull request #82. Given this change, we need to tune again the hyperparameters, so we are reruning some experiments and will post updates on the GAM repository.
However, I am not sure I understand the issue with "pair_iterator". You are right that it returns the true labels for unlabeled nodes, but these are not used. If you check the function "_construct_feed_dict" line 610 in "trainer_classification_gcn.py", we do not use the labels of the targets of LU edges (which are the unlabeled ones). Based on this, GAM* results should not be affected.
aheydon-google
kvis-google
Hi GAM Authors,
I have noticed a massive label leakage bug in your implementation of GAM/GAM*. Both edge_iterator and pair_iterator are using true labels instead of predicted labels. Below are more details on label leakage bug in each of these iterators.
I have bold the line of concern.
In line: 692, LU_edges and UL_edges are concatenated. Note that unlabeled edges are added as source nodes in the bolded line. We also see this by printing the edges variable in line::710.
While iterating through edges in Line::718, true labels are assigned to unlabeled indices. I have pasted the lines below and have highlighted the line of concern.
for edge in iterator: indices_src = edge[:, 0] indices_tgt = edge[:, 1] features_src = data.get_features(indices_src) features_tgt = data.get_features(indices_tgt) labels_src = data.get_labels(indices_src) labels_tgt = data.get_labels(indices_tgt) yield (indices_src, indices_tgt, features_src, features_tgt, labels_src, labels_tgt)
data.get_edges() returns true labels for unlabeled indices, thus showing massive improvements as reported in the paper.
Post fixing this label-leakage bug (either by remove UL edges and reversing UL edges), we can only observe marginal improvements over baselines.
The way pair-iterator is defined in trainer_classification_gcn.py::633 printing variables labels_src, labels_tgt in line 668, 669, we can see that true label instead of predicted labels are assigned for LU and UU pair iterators. the _select_from_pool() method invoked by pair_iterator assigns labels by using data.get_labels(indices_batch). This call returns true labeles for unlabeled indices instead of predicted labels.