A bug in Metattack & MetaApprox

[ocrinz]

I encountered a bug when I'm trying Mettack.

In deeprobust/graph/global_attack/mettack.py, line 346 to 361:

            adj_meta_score = torch.tensor(0.0).to(self.device)
            feature_meta_score = torch.tensor(0.0).to(self.device)

            if self.attack_structure:
                adj_meta_score = self.get_adj_score(self.adj_grad_sum, modified_adj, ori_adj, ll_constraint, ll_cutoff)
            if self.attack_features:
                feature_meta_score = self.get_feature_score(self.feature_grad_sum, modified_features)

            if adj_meta_score.max() >= feature_meta_score.max():
                adj_meta_argmax = torch.argmax(adj_meta_score)
                row_idx, col_idx = utils.unravel_index(adj_meta_argmax, ori_adj.shape)
                self.adj_changes.data[row_idx][col_idx] += (-2 * modified_adj[row_idx][col_idx] + 1)
                self.adj_changes.data[col_idx][row_idx] += (-2 * modified_adj[row_idx][col_idx] + 1)
            else:
                feature_meta_argmax = torch.argmax(feature_meta_score)
                row_idx, col_idx = utils.unravel_index(feature_meta_argmax, ori_features.shape)
                self.feature_changes.data[row_idx][col_idx] += (-2 * modified_features[row_idx][col_idx] + 1)

The default values of adj_meta_score and feature_meta_score are 0. This seems OK in most cases but can cause errors in some extreme cases (as I encountered).

The problem is, if self.attack_adj is True, self.attack_features is False and adj_meta_score.max() < 0. then it will go in the "else" branch, and setting self.feature_changes (line 361) will cause an ModuleAttributeError error (self.feature_changes is undefined since self.attack_features is False).

I think the default values should be -inf, so that it will never go into the branch when self.attack_features is False.

This bug exists in MetaApprox, too.

[ChandlerBang]

Hi, thanks for pointing this out! But if we take a look at the code https://github.com/DSE-MSU/DeepRobust/blob/2a52969fb8b881ac5325a8d0a26a6880aa8b6a9b/deeprobust/graph/global_attack/mettack.py#L118-L119

we will find that the value of adj_meta_score.max() should be at least 0. Could you provide more details on the bug you met?

[ocrinz]

I managed to print those values, and it turned out that adj_meta_score was full of NaNs. So, the cause of the error was not as my initial guess, but the NaNs.

Then I printed adj_grad, and it had a few NaNs, too. I also checked adj, modified_adj, self.adj_changes and attack_loss, and they had no NaNs. This seemed strange, as attack_loss and self.adj_changes had no NaNs while adj_grad had NaNs. I'm not sure how this happened.

https://github.com/DSE-MSU/DeepRobust/blob/2a52969fb8b881ac5325a8d0a26a6880aa8b6a9b/deeprobust/graph/global_attack/mettack.py#L294-L295

Below I provide some relevant information about my code. Hope this helps.

---

Dataset: a directed graph, which has no NaNs in its adj and features.

Relevant code (adapted from Pro-GNN):

surrogate = GCN(nfeat=features.shape[1], nclass=labels.max().item()+1, nhid=16, dropout=0.5, with_relu=False, with_bias=True, weight_decay=5e-4, device=device)
surrogate = surrogate.to(device)
surrogate.fit(features, adj, labels, idx_train)
lambda_ = 0 # Meta-Self
model = Metattack(model=surrogate, nnodes=adj.shape[0], feature_shape=features.shape,
    attack_structure=True, attack_features=False, device=device, lambda_=lambda_)
model = model.to(device)
model.attack(features, adj, labels, idx_train, idx_unlabeled, perturbations, ll_constraint=False)

Output & error:

Perturbing graph:   0%|          | 0/38 [00:00<?, ?it/s]
GCN loss on unlabled data: 1.3923810720443726
GCN acc on unlabled data: 0.5547945205479452
attack loss: 1.311768889427185
Perturbing graph:   3%|▎         | 1/38 [00:00<00:26,  1.41it/s]
GCN loss on unlabled data: 1.4063081741333008
GCN acc on unlabled data: 0.5547945205479452
attack loss: 1.3405990600585938
Perturbing graph:   5%|▌         | 2/38 [00:01<00:22,  1.57it/s]
GCN loss on unlabled data: 1.4276279211044312
GCN acc on unlabled data: 0.5547945205479452
attack loss: 1.3694512844085693
Perturbing graph:   8%|▊         | 3/38 [00:01<00:21,  1.61it/s]
GCN loss on unlabled data: 1.4045186042785645
GCN acc on unlabled data: 0.5547945205479452
attack loss: 1.3383311033248901
Perturbing graph:  11%|█         | 4/38 [00:02<00:20,  1.66it/s]
GCN loss on unlabled data: 1.4385275840759277
GCN acc on unlabled data: 0.5547945205479452
attack loss: 1.387487769126892
Perturbing graph:  13%|█▎        | 5/38 [00:03<00:19,  1.69it/s]
GCN loss on unlabled data: 1.4007790088653564
GCN acc on unlabled data: 0.5547945205479452
attack loss: 1.3371026515960693
Perturbing graph:  16%|█▌        | 6/38 [00:03<00:18,  1.70it/s]
GCN loss on unlabled data: 1.4176795482635498
GCN acc on unlabled data: 0.5547945205479452
attack loss: 1.364121675491333
Perturbing graph:  18%|█▊        | 7/38 [00:04<00:18,  1.71it/s]
GCN loss on unlabled data: 1.448235034942627
GCN acc on unlabled data: 0.5547945205479452
attack loss: 1.4047572612762451
Perturbing graph:  21%|██        | 8/38 [00:04<00:17,  1.73it/s]
GCN loss on unlabled data: 1.401132583618164
GCN acc on unlabled data: 0.5547945205479452
attack loss: 1.3446844816207886
Perturbing graph:  24%|██▎       | 9/38 [00:05<00:16,  1.73it/s]
GCN loss on unlabled data: 1.4135085344314575
GCN acc on unlabled data: 0.5547945205479452
attack loss: 1.3655924797058105
Perturbing graph:  26%|██▋       | 10/38 [00:05<00:16,  1.74it/s]
GCN loss on unlabled data: 1.4655441045761108
GCN acc on unlabled data: 0.547945205479452
attack loss: 1.4301691055297852
---------------------------------------------------------------------------
ModuleAttributeError                      Traceback (most recent call last)
<ipython-input-10-095b9bd03317> in <module>
----> 1 model.attack(features, adj, labels, idx_train, idx_unlabeled, perturbations, ll_constraint=False)
      2 model.save_adj(root='.', name='{}_meta_adj_{}_{}'.format(args.dataset, args.ptb_rate, args.seed))

/opt/conda/lib/python3.7/site-packages/deeprobust/graph/global_attack/mettack.py in attack(self, ori_features, ori_adj, labels, idx_train, idx_unlabeled, n_perturbations, ll_constraint, ll_cutoff)
    359                 feature_meta_argmax = torch.argmax(feature_meta_score)
    360                 row_idx, col_idx = utils.unravel_index(feature_meta_argmax, ori_features.shape)
--> 361                 self.feature_changes.data[row_idx][col_idx] += (-2 * modified_features[row_idx][col_idx] + 1)
    362 
    363         if self.attack_structure:

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in __getattr__(self, name)
    777                 return modules[name]
    778         raise ModuleAttributeError("'{}' object has no attribute '{}'".format(
--> 779             type(self).__name__, name))
    780 
    781     def __setattr__(self, name: str, value: Union[Tensor, 'Module']) -> None:

ModuleAttributeError: 'Metattack' object has no attribute 'feature_changes'

[ChandlerBang]

Hi, it is a bit hard for me to debug since I cannot reproduce the bug currently. You may check what results in the NaN value in the gradients.

I want to comment that the code of metattack is designed for attacking undirected attack. As you can see in https://github.com/DSE-MSU/DeepRobust/blob/2a52969fb8b881ac5325a8d0a26a6880aa8b6a9b/deeprobust/graph/global_attack/mettack.py#L69-L74

we will symmetrize adj_changes and perturb the graph. It might cause some problem if you don't modify this part.

[ocrinz]

Hi, after I modified this function for directed graphs, the error disappeared. I guess this is the cause,

    def get_modified_adj(self, ori_adj, undirected):
        adj_changes_square = self.adj_changes - torch.diag(torch.diag(self.adj_changes, 0))
        #ind = np.diag_indices(self.adj_changes.shape[0]) # this line seems useless
        if undirected:
            adj_changes_square = torch.clamp(adj_changes_square + torch.transpose(adj_changes_square, 1, 0), -1, 1)
        modified_adj = adj_changes_square + ori_adj
        return modified_adj

By the way, I suggest adding an argument undirected to BaseMeta, Metattack and MetaApprox so that they can work on directed graphs, too.

I created a gist of this adaptation (but I have not run my code to check its correctness). It's just a minor modification to the current code. Modified lines are: 38-39, 44, 55, 72-75, 165-167, 392-394.

P.S. Sorry but I don't know how to quote lines of a gist on GitHub. You may use the diff command on Linux/Unix or fc on Windows or something like that to find out the modified lines.

---

Upd: The modification worked fine on a few graphs, but the error happened again on another graph. Unfortunately, it is difficult for me to rerun my code to print relevant information because this graph is large and thus each run would take quite a few GPU hours before triggering the error.

I suspect that it is also because the adj is asymetric. I wonder whether there is other code in Mettack that depends on the symmetry of adj.

[ChandlerBang]

Hi,

(1) To update the code, you can try to pull a request. See details here. (2) I believe the following code should also be changed https://github.com/DSE-MSU/DeepRobust/blob/2a52969fb8b881ac5325a8d0a26a6880aa8b6a9b/deeprobust/graph/global_attack/mettack.py#L89-L90

[ocrinz]

Hi,

Besides function filter_potential_singletons, I found another code to revise:

https://github.com/DSE-MSU/DeepRobust/blob/2a52969fb8b881ac5325a8d0a26a6880aa8b6a9b/deeprobust/graph/global_attack/mettack.py#L356-L357

I changed it to:

                self.adj_changes.data[row_idx][col_idx] += (-2 * modified_adj[row_idx][col_idx] + 1)
                if self.undirected:
                    self.adj_changes.data[col_idx][row_idx] += (-2 * modified_adj[row_idx][col_idx] + 1)

Besides, I think function log_likelihood_constraint also need revision (although I didn't use ll_constraint):

https://github.com/DSE-MSU/DeepRobust/blob/2a52969fb8b881ac5325a8d0a26a6880aa8b6a9b/deeprobust/graph/global_attack/mettack.py#L109-L113

I changed t_possible_edges to:

if self.undirected:
    t_possible_edges = np.array(np.triu(np.ones((self.nnodes, self.nnodes)), k=1).nonzero()).T
else:
    t_possible_edges = np.array((np.ones((self.nnodes, self.nnodes)) - np.eye(self.nnodes)).nonzero()).T

And, in function utils.likelihood_ratio_filter:

https://github.com/DSE-MSU/DeepRobust/blob/2a52969fb8b881ac5325a8d0a26a6880aa8b6a9b/deeprobust/graph/utils.py#L598

I changed it to:

if undirected:
    allowed_mask += allowed_mask.t()

Now it worked without error on my datasets. I've created a pull request about my changes.

P.S. Since I'm not very familiar with Mettack, I'm not sure about theoretical correctness of my code. Please double check my pull request. Thx.