search

ZhangYuanhan-AI / NOAH

[TPAMI] Searching prompt modules for parameter-efficient transfer learning.
MIT License
225 stars 11 forks source link

Table results from the last model or the best model? #6

Closed fanq15 closed 2 years ago

fanq15 commented 2 years ago

Hi, thanks for your great work! I wonder the model performance in your paper tables. Is it from the last trained model? Or from the model with the best testing performance?

ZhangYuanhan-AI commented 2 years ago

Best model

fanq15 commented 2 years ago

Thanks!

RaptorMai commented 1 year ago

Thanks for your answer. When you said the best model, does it mean the model with the best testing performance? Shouldn't we use the validation set to find the best model? Thanks in advance and look forward to hearing back from you.

ZhangYuanhan-AI commented 1 year ago

Thanks for your answer. When you said the best model, does it mean the model with the best testing performance? Shouldn't we use the validation set to find the best model? Thanks in advance and look forward to hearing back from you.

Greetings,

My apologies for any confusion caused earlier.

The NOAH approach comprises two stages. (1) find the optimal prompt structure. (2) the model, which is implied by the optimal prompt structure. is retrained using this optimal prompt structure.

In the first stage, the optimal prompt structure is determined based on the validation set. Subsequently, during the second stage, the model's performance is evaluated and reported using the test set.

The reported result for the retrained model typically does not correspond to the final training step, but rather reflects the highest performance attained on the test set.

In summary, your assertion is correct, and we indeed use the validation set to identify the most effective model.