Why each training result is different, how to ensure that each training result is the same

YoumiMa commented 1 year ago

@yxy0102 Thank you for the interest in this project! We do not expect each training result to be the same because we are using CUDA >= 10.2, where a handful of CUDA operations are non-deterministic (see this pytorch documentation for more details). If you do want to ensure each training result is the same, please use the same random seed and try the techniques described in the documentation above (i.e., setting the environment variable CUBLAS_WORKSPACE_CONFIG=:4096:8 or CUBLAS_WORKSPACE_CONFIG=:16:8).

YoumiMa commented 1 year ago

Hi, do you have any further question on this? If not, I’d be really grateful if you could close the issue for me.

yxy0102 commented 1 year ago

Training command: bash script/run_bash.sh bert 0.1 66 This is the saved file of my training results. I compressed the result.json file and uploaded it to the official website of the competition. The test showed 0, and when I used best.ckpt to train the remote data, the generated att file also showed 0. I don’t understand what happened. problem, please advise,thanks

I will delete the question, please communicate by email

? @.***

------------------ 原始邮件 ------------------ 发件人: "YoumiMa/dreeam" @.>; 发送时间: 2023年4月1日(星期六) 下午4:10 @.>; @.**@.>; 主题: Re: [YoumiMa/dreeam] Why each training result is different, how to ensure that each training result is the same (Issue #5)

Hi, do you have any further question on this? If not, I’d be really grateful if you could close the issue for me.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

YoumiMa commented 1 year ago

Thank you for the message.

The training script predicts on the development set, not the test set. To obtain predictions on the test set that can be evaluated on the official competition website, please perform evaluation on test set, following instructions in the evaluation section of README.md.

Yours, Youmi

2023年4月1日(土) 17:17 颜新月 @.***>:

Training command: bash script/run_bash.sh bert 0.1 66 This is the saved file of my training results. I compressed the result.json file and uploaded it to the official website of the competition. The test showed 0, and when I used best.ckpt to train the remote data, the generated att file also showed 0. I don’t understand what happened. problem, please advise,thanks

I will delete the question, please communicate by email

? @.***

------------------ 原始邮件 ------------------ 发件人: "YoumiMa/dreeam" @.>; 发送时间: 2023年4月1日(星期六) 下午4:10 @.>; @.**@.>; 主题: Re: [YoumiMa/dreeam] Why each training result is different, how to ensure that each training result is the same (Issue #5)

Hi, do you have any further question on this? If not, I’d be really grateful if you could close the issue for me.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/YoumiMa/dreeam/issues/5#issuecomment-1492875428, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANFUSWZV4ENUPA4JKD2RSOLW67QBLANCNFSM6AAAAAAWBCGMVQ . You are receiving this because you commented.Message ID: @.***>

-- Best Regards マヨウミ馬尤咪 Youmi Ma, Department of Computer Science, School of Computing Tokyo Institute of Technology E-mail: @.; @.

yxy0102 commented 1 year ago

Thank you very much for your patient answer. I'm just beginning to get started, and I'm very disturbed. The author is great.

? @.***

------------------ 原始邮件 ------------------ 发件人: "YoumiMa/dreeam" @.>; 发送时间: 2023年4月1日(星期六) 晚上10:00 @.>; @.**@.>; 主题: Re: [YoumiMa/dreeam] Why each training result is different, how to ensure that each training result is the same (Issue #5)

Thank you for the message.

The training script predicts on the development set, not the test set. To obtain predictions on the test set that can be evaluated on the official competition website, please perform evaluation on test set, following instructions in the evaluation section of README.md.

Yours, Youmi

2023年4月1日(土) 17:17 颜新月 @.***>:

> Training command: bash script/run_bash.sh bert 0.1 66 > This is the saved file of my training results. I compressed the > result.json file and uploaded it to the official website of the > competition. The test showed 0, and when I used best.ckpt to train the > remote data, the generated att file also showed 0. I don’t understand what > happened. problem, please advise,thanks > > I will delete the question, please communicate by email > > > > > > ? > @. > > > >   > > > > > ------------------ 原始邮件 ------------------ > 发件人: "YoumiMa/dreeam" @.>; > 发送时间: 2023年4月1日(星期六) 下午4:10 > @.>; > @*.**@*.>; > 主题: Re: [YoumiMa/dreeam] Why each training result is different, how > to ensure that each training result is the same (Issue #5) > > > > > > > Hi, do you have any further question on this? If not, I’d be really > grateful if you could close the issue for me. > > — > Reply to this email directly, view it on GitHub, or unsubscribe. > You are receiving this because you were mentioned.Message ID: > @.> > > — > Reply to this email directly, view it on GitHub > <https://github.com/YoumiMa/dreeam/issues/5#issuecomment-1492875428>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/ANFUSWZV4ENUPA4JKD2RSOLW67QBLANCNFSM6AAAAAAWBCGMVQ> > . > You are receiving this because you commented.Message ID: > @.> > -- Best Regards マヨウミ馬尤咪 Youmi Ma, Department of Computer Science, School of Computing Tokyo Institute of Technology E-mail: @.; @.***

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

yxy0102 commented 1 year ago

Excuse me again, I encountered this problem while verifying the dev data. What is the reason,We look forward to hearing from youbash scripts/isf_bert.sh ${name} ${model_dir} dev # for BERT

? @.***

------------------ 原始邮件 ------------------ 发件人: "YoumiMa/dreeam" @.>; 发送时间: 2023年4月1日(星期六) 晚上10:00 @.>; @.**@.>; 主题: Re: [YoumiMa/dreeam] Why each training result is different, how to ensure that each training result is the same (Issue #5)

Thank you for the message.

The training script predicts on the development set, not the test set. To obtain predictions on the test set that can be evaluated on the official competition website, please perform evaluation on test set, following instructions in the evaluation section of README.md.

Yours, Youmi

2023年4月1日(土) 17:17 颜新月 @.***>:

> Training command: bash script/run_bash.sh bert 0.1 66 > This is the saved file of my training results. I compressed the > result.json file and uploaded it to the official website of the > competition. The test showed 0, and when I used best.ckpt to train the > remote data, the generated att file also showed 0. I don’t understand what > happened. problem, please advise,thanks > > I will delete the question, please communicate by email > > > > > > ? > @. > > > >   > > > > > ------------------ 原始邮件 ------------------ > 发件人: "YoumiMa/dreeam" @.>; > 发送时间: 2023年4月1日(星期六) 下午4:10 > @.>; > @*.**@*.>; > 主题: Re: [YoumiMa/dreeam] Why each training result is different, how > to ensure that each training result is the same (Issue #5) > > > > > > > Hi, do you have any further question on this? If not, I’d be really > grateful if you could close the issue for me. > > — > Reply to this email directly, view it on GitHub, or unsubscribe. > You are receiving this because you were mentioned.Message ID: > @.> > > — > Reply to this email directly, view it on GitHub > <https://github.com/YoumiMa/dreeam/issues/5#issuecomment-1492875428>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/ANFUSWZV4ENUPA4JKD2RSOLW67QBLANCNFSM6AAAAAAWBCGMVQ> > . > You are receiving this because you commented.Message ID: > @.> > -- Best Regards マヨウミ馬尤咪 Youmi Ma, Department of Computer Science, School of Computing Tokyo Institute of Technology E-mail: @.; @.***

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

yxy0102 commented 1 year ago

In inference fusion verification, the prediction of evidence is always 0. What is the reason

? @.***

------------------ 原始邮件 ------------------ 发件人: "YoumiMa/dreeam" @.>; 发送时间: 2023年4月1日(星期六) 晚上10:00 @.>; @.**@.>; 主题: Re: [YoumiMa/dreeam] Why each training result is different, how to ensure that each training result is the same (Issue #5)

Closed #5 as completed.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

YoumiMa / dreeam

Why each training result is different, how to ensure that each training result is the same #5