search

Jannoshh / simple-sam

Sharpness-Aware Minimization for Efficiently Improving Generalization
MIT License
41 stars 9 forks source link

TypeError: unsupported operand type(s) for *: 'IndexedSlices' and 'float' #8

Open 353xiong opened 2 years ago

353xiong commented 2 years ago

e_w = gradients[i] * self.rho / (grad_norm + 1e-12)

TypeError: unsupported operand type(s) for *: 'IndexedSlices' and 'float'

How to deal with it?

Jannoshh commented 2 years ago

It seems like the variable 'gradients' is not a list (which it should be). If you are trying to minimize a single tensor using a GradientTape like the following:

image

try index_slices = tape.gradient(r, [v]) instead of index_slices = tape.gradient(r, v)

If you are trying something different, could you give me some more details?

dennism-tulcolabs commented 2 years ago

I ran into the same error (TF 2.5), and replacing 

e_w = gradients[i] * rho / (grad_norm + eps)

with

e_w = tf.math.scalar_mul(rho, gradients[i]) / (grad_norm + eps)

appears to fix it.