search

TsinghuaC3I / SoRA

The source code of the EMNLP 2023 main conference paper: Sparse Low-rank Adaptation of Pre-trained Language Models.
62 stars 8 forks source link

May I ask if the results in the paper are from the eval set or the test set? Please let me know #6

Open marcocst opened 5 months ago

marcocst commented 5 months ago

May I ask if the results in the paper are from the eval set or the test set? Please let me know

telxt commented 5 months ago

Thank you for your interest in our work! The results in the paper are from the test set.

marcocst commented 5 months ago

Thank you！

sborse3 commented 4 months ago

@marcocst were you able to run inference on the test set?

telxt commented 4 months ago

@sborse3 @marcocst Sorry for not making it clear earlier. The results in the paper are from the test set. But this test set differs from the test part of the original dataset from Huggingface. We partition the dataset as follows:

For small datasets (n_samples < 10K), we divide validation set to half, use one half as test set and one half as validation set. For larger datasets (n_samples > 10K), we divide training set into 1K as validation and the rest as training set, keeping the original validation set as the test set. You can find the specific implementation in the get function within the SoRA/src/processor.py file (Lines 87-106).

Please don't hesitate to contact us if you have further questions or need more assistance!