Open tanayshah23 opened 10 months ago
Hello @tanayshah23,
thanks for using the TabLLM project and please excuse the delayed response.
We do not use a separate validation set, since we fix the numbers epochs and no additional parameter tuning is done (following the original t-few setup). Instead, we treated our test set as the validation (=dev) set in the code because this gave us the test performance (named dev_scores) immediately during training without another test inference run.
This is definitely a bit confusing, so thanks for pointing this out. We also added this to our revised readme. As a result, you should find the test performance of each run in dev_scores.json
. We also added the script for evaluation to the repository and the readme instructions now.
To obtain the prediction results themselves you probably have to change the t-few code to output the data. I don't think there exists a routine for this yet.
I hope that helps!
Hi,
I am trying to train a model on the heart dataset. The training works, but I am unable to find the following files in the exp_out folder:
Here's the screenshot for your reference:
My main concern is now that we have the trained model checkpoint, how can we use it to get results on the dev/test set and get the results?
Thanks!