Closed zhongjian-zhang closed 3 months ago
If you have a free moment, could you please look at this? Thanks.
Hi @1957787636 , do you mean individual experiment or joint experiment? Can you share your command? Thanks.
Can you also check if you are on the latest version? We committed several major changes recently.
Hi @1957787636 , do you mean individual experiment or joint experiment? Can you share your command? Thanks.
Can you also check if you are on the latest version? We committed several major changes recently.
Hi @LechengKong, thank for your reply. I want to reproduce the results of cora_node and pubmed_node in Table 3 using the llama2-7b and llama2-13b. The command I use is as follows:
python run_cdm.py task_names cora_node num_epochs 20 d_multiple 10.0 d_min_ratio 10.0 lr 0.0001 JK none batch_size 64
the accuracy is 0.54493.
Lastly, I'm sure I'm using the latest version. I would be grateful if you could provide me with the command to run cora_node and cora_pubmed using using the llama2-7b and llama2-13b in Table3. Looking forward to your reply!
Hi @1957787636 , note that the llama2 results in Table 3 are NOT individual dataset experiments, it is obtained by jointly training all datasets together and evaluating on different datasets. You should not get the same results as in Table 3 if you only train on cora or PubMed as your command suggested.
Meanwhile, we just tried individual experiments using your command and got the 0.68 results. This makes sense because llama2 provides much more fine-grained text embedding which easily leads to overfitting (all train results reach 1.00 accuracy). We tried varying dropout and l2 but they didn't seem to help with the test results, and I think the results you obtained are valid.
Hi, @LechengKong , I want to reproduce the result of cora_node under OFA-ind-st, but the ACC I get is only 70%, which is quite different from the 75.61±0.87 reported in Table 3. I use the command you recommended in issue6: python run_cdm.py task_names cora_node num_epochs 20 d_multiple 10.0 d_min_ratio 10.0 lr 0.0001 JK none batch_size 64
Hi @1957787636 can you try the following command:
python run_cdm.py task_names cora_node num_epochs 20 d_multiple 10.0 d_min_ratio 10.0 lr 0.0001 JK none batch_size 64 dropout 0.15
we removed the default 0.15 dropout in one of our commits.
Hi @1957787636 can you try the following command:
python run_cdm.py task_names cora_node num_epochs 20 d_multiple 10.0 d_min_ratio 10.0 lr 0.0001 JK none batch_size 64 dropout 0.15
we removed the default 0.15 dropout in one of our commits.
thank your for your reply. I use your commend, the reslut is reasonable. Furthermore, I want to reproduce the result of cora_node under OFA-llama2-7b(jointly), I used the recommended commands, but couldn't get the results in Table 3. The commend as follows: python run_cdm.py --override e2e_all_config.yaml num_layers 7 batch_size 512 dropout 0.15 JK none
Hi, @LechengKong , when I use the llama2-7b or llama2-13b as the llm encoder, the acc is around 0.68. Could you please provice the run command about lama2-7b and llama2-13b?
Originally posted by @1957787636 in https://github.com/LechengKong/OneForAll/issues/6#issuecomment-2081955490