search

google-research / pegasus

Apache License 2.0
1.61k stars 316 forks source link

How to load Fine-tuned checkpoint files on custom data and use for prediction #160

Open SagarPalyal opened 3 years ago

SagarPalyal commented 3 years ago

Hi Team,

I have fine tuned existing pegasus model on custom domain specific dataset and end up with below checkpoint files

  1. checkpoint
  2. model.ckpt.data
  3. model.ckpt.index
  4. model.ckpt.meta

But Pegasus default model_dir having below patter:

  1. config.json
  2. pytorch_model.bin
  3. special_token_maps.json
  4. spiece.model
  5. tokenizer_config.json

And Code implementation which uses Pegasus default model dir for generating summary is below:

from transformers import PegasusForConditionalGeneration, PegasusTokenizer import torch src_text = [ """ Text to summarize.

""" ] model_path = 'google/pegasus-wikihow' vice = 'cuda' if torch.cuda.is_available() else 'cpu' tokenizer = PegasusTokenizer.from_pretrained(model_path)

model = PegasusForConditionalGeneration.from_pretrained(model_path).to(torch_device)

batch = tokenizer.prepare_seq2seq_batch(src_text, truncation=True, padding='longest', return_tensors="pt").to(torch_device) translated = model.generate(**batch) tgt_text = tokenizer.batch_decode(translated, skip_special_tokens=True) print(tgt_text[0])

Now problem is that after fine-tuning with custome dataset, I end up with new model directory having some checkpoint files and I am unable to load and use this new model directory for prediction.

Please let me know what are the changes required in the code now so that I can generate predictions using fine-tuned model checkpoint files directory. I have done alot research on it but not able to resolve it.

PS: Also tried "from_tf=True" parameter but not working.

JingqingZ commented 3 years ago

Hi, I don't think the checkpoints of tensorflow implementation (this repo) can be compatible with the pytorch implementation by Huggingface.

SagarPalyal commented 3 years ago

okay...is there any work around because I want to use huggingface implementation with fine-tuned models.

karimfayed commented 3 years ago

okay...is there any work around because I want to use huggingface implementation with fine-tuned models.

Hi there. I have used the hugging face code to use already fine-tuned models then I wanted to fine-tune the model using a new dataset so I used this https://gist.github.com/jiahao87/50cec29725824da7ff6dd9314b53c4b3. This is a script to fine-tune on PEGASUS and produce the following files

karimfayed commented 3 years ago

okay...is there any work around because I want to use huggingface implementation with fine-tuned models.

Hi there. I have used the hugging face code to use already fine-tuned models then I wanted to fine-tune the model using a new dataset so I used this https://gist.github.com/jiahao87/50cec29725824da7ff6dd9314b53c4b3. This is a script to fine-tune on PEGASUS and produce the following files

  • config.json
  • pytorch_model.bin
  • optimizer.pt
  • scheduler.pt
  • trainer_state.json
  • training_args.bin As you might notice unfortunately there isn't any tokenizer files and this is where I am stuck right now. Hope this helps and if you found a way to produce the tokenizer files please inform me.

Hi @SagarPalyal, The author of the fine-tuning script updated it and you can use it easily and it will produce tokenizer files with each checkpoint

NamraRehman commented 1 year ago

Hi, @SagarPalyal Did you find any solution? I am facing the same issue.