Closed snoop2head closed 2 years ago
yes that model can't be parallelized because the number of head is not divisible by 2.
for tensor model parallelism, the number of head must be divisible by tensor model parallel size (num_gpus
in parallelformers). it's limitation of tensor model parallelism.
when gpt2 was published, model parallelism techniques were not developed as now. so they didn't consider this aspect. please use gpt-neo-1.3b model for 1~2b size causal modeling.
thank you!
I see! Thank you for the detailed reply and model recommendation 🤗
Thank you for the great project!
How to reproduce
https://github.com/snoop2head/Language_Model_Memorization/blob/2c5db6f9bdd0206cba87d13b158d8c27ce0e55a7/parallel_inference.py#L39-L82
gpt2
,gpt2-medium
,gpt2-large