Closed yongzx closed 2 years ago
https://github.com/bigscience-workshop/multilingual-modeling/blob/1a383288d89dafa16e462558a209e33a020ebe85/scripts/madx_exp/madx_lngembft_clm.py#L457
@vnikouliNLE It seems like we are not fine-tuning the positional embedding layers transformer.wpe.weight. I believe we should also train it too?
transformer.wpe.weight
Yes we should train it.
https://github.com/bigscience-workshop/multilingual-modeling/blob/1a383288d89dafa16e462558a209e33a020ebe85/scripts/madx_exp/madx_lngembft_clm.py#L457
@vnikouliNLE It seems like we are not fine-tuning the positional embedding layers
transformer.wpe.weight
. I believe we should also train it too?