search

DAMO-NLP-MT / PolyLM

76 stars 7 forks source link

Model architecture is not same in variant models #3

Open dat-browny opened 1 year ago

dat-browny commented 1 year ago

In my experiments, i realize that the model architecture of polylm-13b is difference from polylm-multialpaca-13b version, but in model cards, it was said that multialpaca is a SFT version of polylm-13b. This is so confusing that the model type is not matching that in config of base model polylm the LMhead is PolyLMHead, and in SFT version is GPT2LMHead, can you explain clearly about this mismatching?

And if i can using the GPT2LMHead replace for the base model PolyLMHead, and if i can, how can i convert the tensor shape of model to this type of LMHead? (my aims to deploy model with vLLM, supported GPT2LMHead)

This is config of two version, polylm-13b:

{
  "activation_function": "gelu_fast",
  "architectures": [
    "PolyLMHeadModel"
  ],
  "auto_map": {
    "AutoModelForCausalLM": "modeling_polylm.PolyLMHeadModel"
  },
  "attn_pdrop": 0.0,
  "embd_pdrop": 0.0,
  "eos_token_id": 2,
  "initializer_range": 0.02,
  "layer_norm_epsilon": 1e-05,
  "model_type": "gpt2",
  "n_ctx": 2048,
  "n_embd": 5120,
  "n_head": 40,
  "n_inner": 20480,
  "n_layer": 40,
  "n_positions": 2048,
  "reorder_and_upcast_attn": false,
  "resid_pdrop": 0.0,
  "scale_attn_by_inverse_layer_idx": false,
  "scale_attn_weights": true,
  "summary_activation": null,
  "summary_first_dropout": 0.0,
  "summary_proj_to_labels": true,
  "summary_type": "cls_index",
  "summary_use_proj": true,
  "tokenizer_class": "AutoTokenizer",
  "transformers_version": "4.31.0",
  "use_cache": true,
  "vocab_size": 256000
}

and poly-multialpaca-13b:

{
  "activation_function": "gelu_fast",
  "architectures": [
    "GPT2LMHeadModel"
  ],
  "attn_pdrop": 0.0,
  "bos_token_id": 255999,
  "embd_pdrop": 0.0,
  "eos_token_id": 255999,
  "initializer_range": 0.02,
  "layer_norm_epsilon": 1e-05,
  "model_type": "gpt2",
  "n_embd": 5120,
  "n_head": 40,
  "n_inner": 20480,
  "n_layer": 40,
  "n_positions": 2048,
  "reorder_and_upcast_attn": false,
  "resid_pdrop": 0.0,
  "scale_attn_by_inverse_layer_idx": false,
  "scale_attn_weights": true,
  "summary_activation": null,
  "summary_first_dropout": 0.0,
  "summary_proj_to_labels": true,
  "summary_type": "cls_index",
  "summary_use_proj": true,
  "tokenizer_class": "AutoTokenizer",
  "transformers_version": "4.29.2",
  "use_cache": true,
  "vocab_size": 256000
}