Hi,
Executing train.sh, I encountered a problem below.
Which torch version needs to be installed? thanks
File "KERPLE/megatron/mpu/layers.py", line 238, in get_stats
name+'_max': obj.max().detach().cpu(),
RuntimeError: "max_all_cuda" not implemented for 'BFloat16'
Hi, Executing train.sh, I encountered a problem below. Which torch version needs to be installed? thanks
File "KERPLE/megatron/mpu/layers.py", line 238, in get_stats name+'_max': obj.max().detach().cpu(), RuntimeError: "max_all_cuda" not implemented for 'BFloat16'
def stats(self): def get_stats(name, obj): return {name+'_mean': obj.mean().detach().cpu(), name+'_std': obj.std().detach().cpu(), name+'_max': obj.max().detach().cpu(), name+'_min': obj.min().detach().cpu()} dd = {} self.bias_a.data = self.bias_a.data.clamp(min=self.eps) dd.update(get_stats('bias_a', self.bias_a)) self.bias_p.data = self.bias_p.data.clamp(min=self.eps) dd.update(get_stats('bias_p', self.bias_p)) return dd