Config For Dataset Fold

[LYZ02]

Hello, I'd like to know whether I can get the configuration file for training the fold dataset?

[Oxer11]

Hi! I'm currently re-organizing the codebase for the fold dataset and configs and it may take some time. I'll notify you when the codes are ready (probably before the notification of ICLR).

[LYZ02]

You mean Nov 05 22 03:00 AM CAT Or Jan 21 '23 02:00 AM UTC? Right now I think I only need the configuration (especially the hyper-parameters) to validate the model(GearNet-Edge) on Fold dataset. Cause I can't reproduce the similar result based on the EC and GO datasets settings. The accuracy I test on the family fold can only achieve about 0.7 by GearNet-Edge.

[Oxer11]

I mean by Jan 21 '23 04:00 AM CAT.

For the GearNet-Edge model on Fold, the configs should be the same with that on EC and GO except the optimizer and scheduler. We train the model with SGD optimizer for 300 epochs and use StepLR to adjust the learning rate. You can find the detailed hyperparameters in the paper.

Could you share the details about the environment, command and results for running the model on EC and GO? Maybe I could help you find out the reasons why the model cannot achieve good performance.

[LYZ02]

Thank you for your kindness!!!

Here is my EC training log. I use two RTX-3090 to reproduce your work. Accordingly I change the batch size from 2/GPU two 4/GPU, and the rest parameters are the same as your config file.

I trained the EC task for 150 Epochs(In order to show the result timely, the model will show valid and test result once finished one training epoch). However, I can't get the $F{1{max}}$ (81.0) in my experiment. My $F{1{max}}$ is about 0.74. May I ask you for the two-GPU training parameters? (Sorry that I can't use 4 GPU at the same time, and I am curious about the model performance)

Besides, I have a question: Can I add the batch size? Under this circumstance, could you recommend another hyper-parameter combination? Training GeatNet-edge with batch size 2(if use 4 GPUs, the actual batch size is 8) is two slow, I spend more than 1 day by two RTX3090 only finishing 150 EPs.

EC_GearNet-Edge.log

[LYZ02]

I have some questions

1. In your paper, you wrote the GNN dropout is 0.1 for EC and GO, and for Fold and reaction the dropout is 0.2. But I didn't see the dropout setting for the EC dataset according to the existing config file.
2. Did you add the ReduceLROnPlateau scheduler for the EC and GO dataset?
3. For the Fold dataset, the task is a multi-class classification, and I am curious about the loss function you used to train GearNet-Edge.(Maybe you use the cross-entropy instead of the binary cross-entropy? cause the EC and GO is a multi-label task which is different from the fold dataset).
4. Actually the accuracy function implemented in TorchDrug is the micro-averaged. And I think the paper result is the micro-accuracy? ( In the paper, you said that "For fold and reaction classification, the performance is measured with the mean accuracy")

[Oxer11]

Thanks for providing your log! I'll try to figure out the reason. I'm now working on the ICLR rebuttal, so maybe I'll reply to you later.

To answer your questions, 1 & 2. I've simplified the codebase and removed the parts that are not neccessary. Ideally, you can achieve the performance reported in the paper with the current version. So you can refer to this repo, where the dropout and scheduler are discarded. I'll update the paper later.

3. Yes. For fold classification, you need to use the tasks.PropertyPrediction with cross entropy loss instead of tasks.MultipleBinaryClassification in TorchProtein.
4. Yes. The reported accuracy is micro-accuracy, which is commonly used in this task (see this).

[LYZ02]

Thank you for your reply! I will try more to reproduce your work. Good luck!

[LYZ02]

Hey, Recently I re-run the script for reproducing the EC tasks. This time I use 4 RTX-3090 to reproduce your work. However, I still can't achieve the f1 max (0.82). I run about 90 epochs and after 20 epochs, the f1 max is still 0.6~0.7. Here is my training log.

full_ec.log

[Oxer11]

I can share my log file with you. It's a little outdated with different names of API, but should be fine for reference. It seems that your optimization is much slower than mine. Also, here are the versions of the packages on my cluster.

python=3.8
cuda=11.1
pytorch=1.8.0
pytorch-cluster=1.5.9
pytorch-scatter=2.0.8
rdkit=2021.09.2

Have you ever modified the training or model codes?

gearnet_edge_ec_50epoch.txt

[LYZ02]

It was kind and generous of you to share the config and log files with me， I will re-run the experiment based on your python environment. Best Wishes!

[Oxer11]

Hi! Sorry for the late response!

I re-run the code with the current codebase and get similar results as yours, too. Then, I compare it with my previous codebase and find that the ReduceLROnPlateau scheduler is very important for optimization. Therefore, I add it back in 437333f and now can reproduce the results in the paper. I attach the log file here. gearnet_ec_50epoch_4gpu.txt

I'm really sorry for the inconvenience!

[Oxer11]

Hi!

The configs for Fold3D and GearNet_Edge_IEConv have been released.