ClipLoss - RuntimeError: cannot register a hook on a tensor that doesn't require gradient

[Miriam2040]

Hi, thanks for the nice work and great repo! I changed config to train_with_clip=1 to include ClipLoss.

Then, I am getting the following error in the eval step:

File "/app/CLIP_/clip/model.py", line 347, in encode_image return self.visual(image.type(self.dtype)) File "/home/miniconda/envs/habitat/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _callimpl result = self.forward(*input, **kwargs) File "/app/CLIP/clip/model.py", line 238, in forward x = self.transformer(x) File "/home/miniconda/envs/habitat/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _callimpl result = self.forward(*input, **kwargs) File "/app/CLIP/clip/model.py", line 209, in forward return self.resblocks(x) File "/home/miniconda/envs/habitat/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, kwargs) File "/home/miniconda/envs/habitat/lib/python3.7/site-packages/torch/nn/modules/container.py", line 117, in forward input = module(input) File "/home/miniconda/envs/habitat/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _callimpl result = self.forward(*input, **kwargs) File "/app/CLIP/clip/model.py", line 196, in forward x = x + self.attention(self.ln1(x)) File "/app/CLIP/clip/model.py", line 193, in attention attention_probs_backwards_hook=self.set_attn_grad)[0] File "/home/miniconda/envs/habitat/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, kwargs) File "/app/CLIP_/clip/auxilary.py", line 422, in forward attention_probs_backwards_hook=attention_probs_backwardshook) File "/app/CLIP/clip/auxilary.py", line 250, in multi_head_attention_forward attn_output_weights.register_hook(attention_probs_backwards_hook) File "/home/miniconda/envs/habitat/lib/python3.7/site-packages/torch/tensor.py", line 257, in register_hook raise RuntimeError("cannot register a hook on a tensor that " RuntimeError: cannot register a hook on a tensor that doesn't require gradient

Am I missing something? Thanks!

[huo852yan]

I meet the the same problem, do you adress this question?

[hucorz]

I meet the the same problem.

[davidfreire]

Hello everyone,

Fantastic work on this repo! Big thanks to @yael-vinker for the amazing repo—I'm truly inspired by your research on sketches.

To address the issue at hand, I suggest modifying the auxiliary.py file located at CLIP_/clip/auxiliary.py. Specifically, you should add a conditional check to ensure that the hook is only registered if attn_output_weights.requires_grad is True.

Currently, lines 247-250 look like this:

# use hooks for the attention weights if necessary
if attention_probs_forward_hook is not None and attention_probs_backwards_hook is not None:
    attention_probs_forward_hook(attn_output_weights)
    attn_output_weights.register_hook(attention_probs_backwards_hook)

Please update it to the following:

# Use hooks for the attention weights if necessary
if attention_probs_forward_hook is not None and attention_probs_backwards_hook is not None:
    attention_probs_forward_hook(attn_output_weights)
    # Only register the backward hook if gradients are required
    if attn_output_weights.requires_grad:
        attn_output_weights.register_hook(attention_probs_backwards_hook)

Also, remember to assign some weight to self.clip_weight; otherwise, it will always be zero.

This should resolve the error.

Best regards,
David