Could you please provice the run command about lama2-7b and llama2-13b?

[zhongjian-zhang]

          > Hi @noah-yxk , thank you for your interest in our work!

I ran your command and it gives me ~ 0.66 test accuracy, can you also share your replication results? Thanks.

It appears to me that the resulting model is not well-trained. The following command

python run_cdm.py task_names cora_node num_epochs 20 d_multiple 10.0 d_min_ratio 10.0 lr 0.0001 JK none batch_size 64

gives a test accuracy of 0.745. d_multiple 10.0 means that we sample 10*len(dataset) data points for training in one epoch. If you still can't get similar results, please let us know. I think what happened in the command you provided is that we have a learning_rate scheduler that decreases learning rate for every 15 epochs, but this causes performance decrease for cora-node dataset. In the command I provided, the learning rate only decreases once at epoch 15.

For the individual experiment, especially for small datasets like cora-node, extra care is needed to make sure the model is well-trained and not overfitted, we have learning rate and batch_size set at 0.0001 and 64 to reduce the likelihood of overfitting. We will hopefully add hyperparameter settings for individual experiments in our next revision.

Cheers

Hi, @LechengKong , when I use the llama2-7b or llama2-13b as the llm encoder, the acc is around 0.68. Could you please provice the run command about lama2-7b and llama2-13b?

Originally posted by @1957787636 in https://github.com/LechengKong/OneForAll/issues/6#issuecomment-2081955490

[zhongjian-zhang]

If you have a free moment, could you please look at this? Thanks.

[LechengKong]

Hi @1957787636 , do you mean individual experiment or joint experiment? Can you share your command? Thanks.

Can you also check if you are on the latest version? We committed several major changes recently.

[zhongjian-zhang]

Hi @1957787636 , do you mean individual experiment or joint experiment? Can you share your command? Thanks.

Can you also check if you are on the latest version? We committed several major changes recently.

Hi @LechengKong, thank for your reply. I want to reproduce the results of cora_node and pubmed_node in Table 3 using the llama2-7b and llama2-13b. The command I use is as follows: python run_cdm.py task_names cora_node num_epochs 20 d_multiple 10.0 d_min_ratio 10.0 lr 0.0001 JK none batch_size 64 the accuracy is 0.54493.

Lastly, I'm sure I'm using the latest version. I would be grateful if you could provide me with the command to run cora_node and cora_pubmed using using the llama2-7b and llama2-13b in Table3. Looking forward to your reply!

[LechengKong]

Hi @1957787636 , note that the llama2 results in Table 3 are NOT individual dataset experiments, it is obtained by jointly training all datasets together and evaluating on different datasets. You should not get the same results as in Table 3 if you only train on cora or PubMed as your command suggested.

Meanwhile, we just tried individual experiments using your command and got the 0.68 results. This makes sense because llama2 provides much more fine-grained text embedding which easily leads to overfitting (all train results reach 1.00 accuracy). We tried varying dropout and l2 but they didn't seem to help with the test results, and I think the results you obtained are valid.

[zhongjian-zhang]

Hi, @LechengKong , I want to reproduce the result of cora_node under OFA-ind-st, but the ACC I get is only 70%, which is quite different from the 75.61±0.87 reported in Table 3. I use the command you recommended in issue6: python run_cdm.py task_names cora_node num_epochs 20 d_multiple 10.0 d_min_ratio 10.0 lr 0.0001 JK none batch_size 64

[LechengKong]

Hi @1957787636 can you try the following command:

python run_cdm.py task_names cora_node num_epochs 20 d_multiple 10.0 d_min_ratio 10.0 lr 0.0001 JK none batch_size 64 dropout 0.15

we removed the default 0.15 dropout in one of our commits.

[zhongjian-zhang]

Hi @1957787636 can you try the following command:

python run_cdm.py task_names cora_node num_epochs 20 d_multiple 10.0 d_min_ratio 10.0 lr 0.0001 JK none batch_size 64 dropout 0.15

we removed the default 0.15 dropout in one of our commits.

thank your for your reply. I use your commend, the reslut is reasonable. Furthermore, I want to reproduce the result of cora_node under OFA-llama2-7b(jointly), I used the recommended commands, but couldn't get the results in Table 3. The commend as follows: python run_cdm.py --override e2e_all_config.yaml num_layers 7 batch_size 512 dropout 0.15 JK none