Closed lippman1125 closed 8 months ago
@wangfei-2019 has encountered the same problem. I was helping debug with it but didn't identify the issue yet. Some users have reproduced the experiments without meeting this issue though, so I am also very confused. Will keep you updated once we have more findings about it!
I reran the experiments with my data, and got a curve as this: https://wandb.ai/xmzzzzz/pruning/runs/u7c5y8t5/overview So I think what you are seeing should mostly be a data problem, though I am not exactly sure why that happens... I will upload my data as soon as I can!
@xiamengzhou Thanks for your reply. I found that the wiki is encoded in UTF-8. Should I decode the wiki to text before feeding it into the tokenizer?
@lippman1125 We have successfully resolved this problem by splitting the Wiki dataset into smaller subfiles and recalculating the reference loss. It is possible that the issue stemmed from inadequate randomness in the data sampling process. Additionally, inconsistencies between the reference loss and the test dataset could also play a role in contributing to this problem.
@WangFei-2019 I split the Wiki dataset into 100 subfiles. Is it enough? Another question: How many sentences did you select for evaluation? The paper said that 500 sentences are enough. Could you share your wiki processing code with me? If it's possible, I would really appreciate it.
@WangFei-2019 Ok. I will have a try. It looks like recalculating the reference loss is a key solution.
the script of pruning as follow: