search

chijames / KERPLE

Apache License 2.0
16 stars 1 forks source link

RuntimeError: "max_all_cuda" not implemented for 'BFloat16' #5

Closed vangogh0318 closed 9 months ago

vangogh0318 commented 9 months ago

Hi, Executing train.sh, I encountered a problem below. Which torch version needs to be installed? thanks

File "KERPLE/megatron/mpu/layers.py", line 238, in get_stats name+'_max': obj.max().detach().cpu(), RuntimeError: "max_all_cuda" not implemented for 'BFloat16'

def stats(self): def get_stats(name, obj): return {name+'_mean': obj.mean().detach().cpu(), name+'_std': obj.std().detach().cpu(), name+'_max': obj.max().detach().cpu(), name+'_min': obj.min().detach().cpu()} dd = {} self.bias_a.data = self.bias_a.data.clamp(min=self.eps) dd.update(get_stats('bias_a', self.bias_a)) self.bias_p.data = self.bias_p.data.clamp(min=self.eps) dd.update(get_stats('bias_p', self.bias_p)) return dd

chijames commented 9 months ago

Hi,

Try any version that is newer than 1.8 and see if it works.

Thanks.

vangogh0318 commented 9 months ago

thanks, I will try

vangogh0318 commented 9 months ago

Hi, After installed torch==1.12, the problem solved. thanks