problem of replicating the outcomes of 'ofa-ind-st'

[noah-yxk]

Hello，I tried to reproduce the results of ofa, but did not achieve the effect in the article. And this is my setting python run_cdm.py task_names cora_node num_epochs 100 num_layers 6 dropout 0.15 Could you advise me on replicating the outcomes of 'ofa-ind-st' as mentioned in the paper? What specific command line parameters should I employ to run 'cdm.py'?

[LechengKong]

Hi @noah-yxk , thank you for your interest in our work!

I ran your command and it gives me ~ 0.66 test accuracy, can you also share your replication results? Thanks.

It appears to me that the resulting model is not well-trained. The following command

python run_cdm.py task_names cora_node num_epochs 20 d_multiple 10.0 d_min_ratio 10.0 lr 0.0001 JK none batch_size 64

gives a test accuracy of 0.745. d_multiple 10.0 means that we sample 10*len(dataset) data points for training in one epoch. If you still can't get similar results, please let us know. I think what happened in the command you provided is that we have a learning_rate scheduler that decreases learning rate for every 15 epochs, but this causes performance decrease for cora-node dataset. In the command I provided, the learning rate only decreases once at epoch 15.

For the individual experiment, especially for small datasets like cora-node, extra care is needed to make sure the model is well-trained and not overfitted, we have learning rate and batch_size set at 0.0001 and 64 to reduce the likelihood of overfitting. We will hopefully add hyperparameter settings for individual experiments in our next revision.

Cheers

[noah-yxk]

I'm sorry for not replying promptly.

First of all, thank you for your answer. My previous reproduction result was indeed around 0.66. After using your command, reasonable experimental results were obtained.

Thank you for replying so much and teaching me the correct experimental setup.

Have a nice day.

[zhongjian-zhang]

Hi @noah-yxk , thank you for your interest in our work!

I ran your command and it gives me ~ 0.66 test accuracy, can you also share your replication results? Thanks.

It appears to me that the resulting model is not well-trained. The following command

python run_cdm.py task_names cora_node num_epochs 20 d_multiple 10.0 d_min_ratio 10.0 lr 0.0001 JK none batch_size 64

gives a test accuracy of 0.745. d_multiple 10.0 means that we sample 10*len(dataset) data points for training in one epoch. If you still can't get similar results, please let us know. I think what happened in the command you provided is that we have a learning_rate scheduler that decreases learning rate for every 15 epochs, but this causes performance decrease for cora-node dataset. In the command I provided, the learning rate only decreases once at epoch 15.

For the individual experiment, especially for small datasets like cora-node, extra care is needed to make sure the model is well-trained and not overfitted, we have learning rate and batch_size set at 0.0001 and 64 to reduce the likelihood of overfitting. We will hopefully add hyperparameter settings for individual experiments in our next revision.

Cheers

Hi, @LechengKong , when I use the llama2-7b or llama2-13b as the llm encoder, the acc is around 0.68. Could you please provice the run command about lama2-7b and llama2-13b?