search

NVIDIA / FasterTransformer

Transformer related optimization, including BERT, GPT
Apache License 2.0
5.77k stars 882 forks source link

gpt-neox have no model.wpe.bin #293

Closed cdj0311 closed 2 years ago

cdj0311 commented 2 years ago

hi, I use examples/pytorch/gptneox/utils/eleutherai_gpt_neox_convert.py convert GPT-NeoX model， but there have no model.wpe.bin. However, it must need a model.wpe.bin file when inference with gpt_example.py, such as follows code in gpt.py:

wpe = torch.from_numpy(np.fromfile(ckpt_path + "/model.wpe.bin", dtype=self.weights_data_type)).reshape(-1, self.global_hidden_units) assert self.max_seq_len <= wpe.size(0), ( f"max_seq_len ({self.max_seq_len} must not exceed " f"the value of maximum sequence length during training ({wpe.size(0)})." ) w.append(wpe)

byshiue commented 2 years ago

gpt_example.py is only used for GPT model. You cannot run GPT-NEOX on it. You can use gptneox_example.cc to test.

GPT-NEOX does not have position embedding table (model.wpe.bin).

cdj0311 commented 2 years ago

gpt_example.py is only used for GPT model. You cannot run GPT-NEOX on it. You can use gptneox_example.cc to test.

GPT-NEOX does not have position embedding table (model.wpe.bin).

Do you have any plans to support GPT-NeoX example with Python scripts?

byshiue commented 2 years ago

We will consider, but not have such plan now.