Invest the structural information in job-level anomaly

[cshjin]

• [x] Compare the MLP with GNN (GCN, GAT, GraphSAGE) w/ and w/o position matrix
• [x] Add early stopping to avoid overfitting
• [x] HPS on the base models to check the best performance that can be achieved for each model
• [x] Remove the noisy data, and train again using HPS to check how much we can gain using clean data only (both train and test)

[cshjin]

Model comparison

• Montage workflow

Model	Acc.	F1	Prec.	Recall	ROC-AUC	Conf Mat
MLP(X)	0.8003	0.7375	0.7577	0.8003	0.5354	[[26986, 490], [ 6397, 623]]
MLP(X+P)	0.8017	0.7412	0.7620	0.8017	0.5401	[[26961, 515], [ 6325, 695]]
GCN(X, A)	0.8077	0.7480	0.7831	0.8077	0.5479	[[27092, 384], [ 6248, 772]]
GCN(X+P, A)	0.7954	0.7644	0.7591	0.7954	0.5845	[[25832, 1644], [ 5413, 1607]]
GAT(X, A)	0.7964	0.7062	0.6344	0.7964	0.5	[[27476, 0], [ 7020, 0]]
GAT(X+P, A)	0.7964	0.7062	0.6344	0.7964	0.5	[[27476, 0], [ 7020, 0]]
GraphSAGE(X, A)	0.8134	0.7704	0.7872	0.8134	0.5812	[[26731, 745], [ 5689, 1331]]
GraphSAGE(X+P, A)	0.8089	0.7672	0.7774	0.8089	0.5784	[[26576, 900], [ 5689, 1331]]

check the commit: 89eca0b

[cshjin]

• check the performance (test accuracy) in Cora (citation network, standard benchmark) and our 1000Genome B
• base models: GCN and MLP Results:
• with the additional position matrix, there is no significant improvement from both datasets and both models
• in Cora: there is big gap between GCN and MLP models, however, in 1000Genome B, the gap is quite small (MLP even slightly better than GCN) Potential reasons:
• Cora: a fundamental assumption that two papers have the same labels if one cites to another
• 1000Genome B: the anomaly labels DO NOT have such kind of assumption.

[cshjin]

HPS

• base models: MLP and GraphSAGE
• hyperparameters:

  problem.add_hyperparameter([16, 32, 64, 128, 256], "hidden_channels", default_value=128)
  problem.add_hyperparameter((1e-5, 1e-1, "log-uniform"), "lr", default_value=1e-3)
  problem.add_hyperparameter((1e-5, 1e-4, "uniform"), "weight_decay", default_value=1e-5)
  problem.add_hyperparameter([1, 2, 3, 4, 5, 6], "num_layers", default_value=3)
  problem.add_hyperparameter([0., 0.1, 0.2, 0.3, 0.4, 0.5], "dropout", default_value=0.5)

• single objective: accuracy
• multiple objective: accuracy, ROC-AUC score
• data: without position matrix, but

	w/o clean data	w/ clean data
MLP	0.8076	0.9610
GraphSAGE	0.8487	0.9668