search

wasiahmad / PLBART

Official code of our work, Unified Pre-training for Program Understanding and Generation [NAACL 2021].
https://arxiv.org/abs/2103.06333
MIT License
184 stars 36 forks source link

How to calculate loss in PLBART? #36

Closed bear96 closed 2 years ago

bear96 commented 2 years ago

Hi! I'm relatively new to this. I am trying to fine-tune PLBART on an SQL-natural language dataset for code synthesis task. The downside is I am using Google Colab to do this. I am downloading the pretrained PLBART model from huggingface using AutoModelForSeq2SeqLM.from_pretrained("uclanlp/plbart-base"). I am tokenizing the codes and the natural language using the tokenizer provided in the documentation for huggingface and providing the code tokens as input to the model. However, the output of the model has two outputs: logits and a large tuple which I think are the hidden state values. I feel like I should use the logits to calculate loss against the natural language tokens, but the logits are in decimals and some are negative, while the natural language tokens are probably indices that correspond to some internal vocabulary. Can you advise what to do? I apologize if this sounds like a very basic query.

wasiahmad commented 2 years ago

I think you should look at some hugginface examples, this (https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization) should help you.