search

yizhongw / Tk-Instruct

Tk-Instruct is a Transformer model that is tuned to solve many NLP tasks by following instructions.
https://arxiv.org/abs/2204.07705
MIT License
177 stars 27 forks source link

Tokenizer & Model info #26

Open sbmaruf opened 1 year ago

sbmaruf commented 1 year ago

From the given line, it appears that the data has been transformed using a gpt2 tokenizer.

https://github.com/yizhongw/Tk-Instruct/blob/7c0e4000901e7c148fbb8964042ebfc433f8d98e/src/convert_data_to_s2s.py#L35

However, the model trained is T5.v.1.-lm-adapted checkpoint. Can you confirm that the model was trained with a T5 tokenizer?

Also all the model cards are same in huggingface,

allenai/tk-instruct-11b-def-pos-neg-expl, allenai/tk-instruct-11b-def-pos, allenai/tk-instruct-11b-def.

Can you confirm that the task encoding for these models are following,

  1. allenai/tk-instruct-11b-def-pos-neg-expl : {"add_task_name": False, "add_task_definition": True, "num_pos_examples": 1, "num_neg_examples": 1, "add_explanation": True},
  2. allenai/tk-instruct-11b-def-pos : {"add_task_name": False, "add_task_definition": True, "num_pos_examples": 1, "num_neg_examples": 0, "add_explanation": False},
  3. allenai/tk-instruct-11b-def : {"add_task_name": False, "add_task_definition": True, "num_pos_examples": False, "num_neg_examples": 0, "add_explanation": False},