Closed sajastu closed 3 years ago
Hi,
following up on this @patrickvonplaten and @patil-suraj
Hey @sajastu,
It's pretty difficult for us to debug the script - from a first look, the hyperparameter settings look good to me. An effective batch size of 8 (4 * 2) seems rather small to me, but you can see from your loss curves whether 8 is enough I guess.
Also note that the x-sum dataset has some rather special distribution which is not really the same as reddit data IMO. X-sum is extreme summarization and has very dense sentences as summaries. Not sure if this works well with reddit.
He @patrickvonplaten,
Thanks for your response!
The problem that I'm facing is that: when I'm running the generation phase of facebook/bart-large-xsum
(i.e., without fine-tuning), I'm getting comparably high scores (22.43/ 7.21 / 17.65); however and interestingly, when I finetune it for a few training steps (let's say 10 training steps), and then run the fine-tuned model on the same test set, the scores get much much lower (15.32 / 2.35 / 9.78). This in fact doesn't make sense to me. Theoretically, I expect the scores to stay near to the main model, if not surpassing it, especially when it has been trained for a very few steps...
Do you have any thoughts on this? is this behaviour expectable?
Also, do you think that the model is overfitting, or get stuck in a local minimum that it's producing the same one output regardless of the input that it gets?
I struggle at the same point - the output of the generate-method in a fine-tuned BART seems to be independent of the input.
Interestingly, this holds only for the generate method. If I call the fine-tuned model directly, as with
tokenizer.batch_decode(torch.argmax(model(input_ids = input_ids)[0], axis=-1))
the output is perfectly related to the input, hence, it differs from input to input. Therefore, I assume there is a bug in the BART.generate()-method, or to be more precise with my assumption, in the specific modeling_tf_bart.prepare_inputs_for_generation()
. I tried to verify my assumption ( I guess fine-tuning freezes somehow the past-/-cache-value which disconnects the output from the input), but I don't find the point which triggers this special generate-method-behavouir.
Hi @phhei,
I think the code is probably correct. Or if any flaw, it must exist in the tokenization module, since I'm not getting this "fixed" output on other datasets that I've been using to fine-tune BART. For my special case here, I changed the dataset (i.e., reddit_tifu), ran the same code, and finally able to get it working.
@patrickvonplaten might be of some help here.
Hi @sajastu,
thanks for your reply. However, if the tokenization-module would cause this behavior, then tokenizer.batch_decode(torch.argmax(model(input_ids = input_ids)[0], axis=-1))
(in which input_ids is generated by the tokenizer.encode-method - the same variable I use for the BART.generate(input_ids)-method) would also output always the same. I already investigated the raw tensor output of both approaches, and there is the same: the generate(input_ids)-method always produces the same tensor, torch.argmax(model(input_ids = input_ids)[0], axis=-1)
depended on the input_ids.
I'm asking myself why changing the dataset (without anything else) would solve this issue. In my case, I have a non-huggingface-dataset, preprocessed by tokenizer-calls, so a bug in a huggingface-dataset is therefore not the point, too.
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
I'm getting stuck on fine-tuning BART model on reddit-tifu dataset. When I use a pre-trained model of BART, for example,
bart-large-xsum
without finetuning, it works fine and produces sort of sensible output for each input, but as I start finetuning it with BART, it starts to predict irrelevant text for each given input; as if it has been overfit to training data. Although, overfitting doesn't seem rational to me as the reddit has over 30k training samples. I'm wondering if there's any problem with my bash script or in the fine-tuning scripts? Since I've been using the instructions on https://github.com/huggingface/transformers/tree/master/examples/pytorch/summarization to this end. Following is my bash script for fine-tuningbart-large-xsum
model.I have used these hyperparams to match with the performance of https://arxiv.org/pdf/2008.03156v1.pdf
Outputs, after 1 training epoch:
input:
fine-tuned bart prediction:
input:
fine-tuned bart prediction:
Environment info
transformers
version: 4.7.0 dev0@patrickvonplaten, @patil-suraj, @sgugger
Information
Model I am using (Bert, XLNet ...): BART
The problem arises when using:
The tasks I am working on is:
To reproduce
Steps to reproduce the behavior:
Expected behavior
After fine-tuning for a few steps/epochs, I expect the model learn to generate at least different outputs for varying input texts.
@patrickvonplaten @patil-suraj