get stuck in evaluation of genn_astar

[TriplePool]

Hi, thanks for the excellent work. I'm trying to re-implement the result of genn_astar on AIDS700nef and LINUX. However, I get stuck in running the evaluation script. I have downloaded the released weight file, but the evaluation makes no progress like in an endless loop. The outputs are as followed:

(genn) wbl@SA:~/workspace/codes/GED/GENN-Astar$ python main.py --test --dataset AIDS700nef --enable-astar --astar-use-net --batch-size 1                                                                                                              
+---------------------+------------+
|      Parameter      |   Value    |
+=====================+============+
| Astar beamwidth     | 0          |
+---------------------+------------+
| Astar nopred        | 0          |
+---------------------+------------+
| Astar trustfact     | 1          |
+---------------------+------------+
| Astar use net       | 1          |
+---------------------+------------+
| Batch size          | 1          |
+---------------------+------------+
| Bins                | 16         |
+---------------------+------------+
| Bottle neck neurons | 16         |
+---------------------+------------+
| Cuda                | 0          |
+---------------------+------------+
| Dataset             | AIDS700nef |
+---------------------+------------+
| Diffpool            | 0          |
+---------------------+------------+
| Dropout             | 0          |
+---------------------+------------+
| Enable astar        | 1          |
+---------------------+------------+
| Epochs              | 350        |
+---------------------+------------+
| Filters 1           | 64         |
+---------------------+------------+
| Filters 2           | 32         |
+---------------------+------------+
| Filters 3           | 16         |
+---------------------+------------+
| Gnn operator        | gcn        |
+---------------------+------------+
| Histogram           | 0          |
+---------------------+------------+
| Learning rate       | 0.001      |
+---------------------+------------+
| Measure time        | 0          |
+---------------------+------------+
| Notify              | 0          |
+---------------------+------------+
| Plot                | 0          |
+---------------------+------------+
| Synth               | 0          |
+---------------------+------------+
| Tensor neurons      | 16         |
+---------------------+------------+
| Test                | 1          |
+---------------------+------------+
| Val                 | 0          |
+---------------------+------------+
| Weight decay        | 0.001      |
+---------------------+------------+

Preparing dataset.

Processing...
Done!

Model evaluation.

  0%|                                                                                            | 0/58800 [00:00<?, ?it/s]

Moreover, the evaluation on LINUX seems to take about 60+ hours. Is it running properly? Could you please help me with the above issues? Many thanks!

[rogerwwww]

Hi there,

Thanks for your interest in our paper. Perhaps it is just because the first test instance of the AIDS dataset is too difficult and it may take a longer time than you expected to solve. If everything works fine, it will eventually output a result.

About the testing time, yes it is time-consuming because our implementation is based on the A* algorithm which is even more time-consuming than ours. And we follow the testing protocol by existing GED learning papers that all pairs of graphs are compared. So if there are N graphs in the test dataset, N(N-1) GED problems need to be solved.

If you find the current efficiency unacceptable for your case, I have some suggestions: 1) Set a beam search width for A* so that it will predict a result more efficiently (at the cost of decreased accuracy); 2) Sample a subset of pairs of graphs during evaluation (and please note that the result is no longer a fair comparison with existing GED learning papers); 3) Use other deep learning regression GED methods which are super fast, but cannot predict the edit path;

BTW, there is another paper recently accepted where the GED problem on AIDS is also studied. If you are interested, you can refer to the preprint https://arxiv.org/abs/2106.04927

[TriplePool]

Got it! Many thanks :)