search

yuanqidu / GraphGT

59 stars 7 forks source link

Reproducibility of benchmark results for generation and transformation tasks? #3

Closed jwkirchenbauer closed 1 year ago

jwkirchenbauer commented 2 years ago

Hello!

Thank you for the tutorial notebooks showing how to download and instantiate the various datasets.

I was wondering whether you had any code or documentation as to how to reproduce the benchmark results reported in Table 2 and especially Table 3 in the full paper? I'm interested in using some of the graph transformation data and want to have a reference point for the baseline results you reported.

It seems like the modeling and evaluation code for the two transformation benchmark/baselines is located at IN/model.py and NEC-DGT/main.py, but is not necessarily "plug and play" or configured for automated setup. For example, IN/model.py#L148 seems to require that the MolOpt data is downloaded and named appropriately before the Interaction Network benchmarking code will run.

  1. Is the MolOpt data in the GraphGT release? If so what is the name (in {'qm9', 'zinc', 'moses', 'chembl', 'profold' ... }) and
  2. What are the naming and pathing conventions for the transformation datasets to get the IN/model.py and the NEC-DGT/main.py codes both running?

Overall I just want to confirm that this is where the work representing how to try and reproduce your benchmark results is located, and any extra guidance you can provide on how to set it up. If it is mostly just for historical parsing, and not really workable at this point (I see some 2018 dates on certain files), that would be good to know too.

Thanks!

yuanqidu commented 2 years ago

Thanks for your interest. I will let you about the details soon with the person who conducted the experiment.

yuanqidu commented 2 years ago

We have made all the datasets downloadable through the link provided on our website https://graphgt.github.io/. The code and pip-installed datasets would be ready soon.

yuanqidu commented 2 years ago

Sorry for the late reply, @shi-yu-wang will help you with this issue about code for graph transformation.