Closed TideDra closed 4 months ago
Hi @TideDra, thanks for opening this issue!
You first have to enable gradient checkpointing for the model:
In [1]: from transformers import CLIPVisionModel
...: vision_tower = CLIPVisionModel.from_pretrained("openai/clip-vit-large-patch14-336")
...: vision_tower.gradient_checkpointing_enable({"use_reentrant": True})
...: vision_tower.vision_model.encoder._gradient_checkpointing_func
Out[1]: functools.partial(<function checkpoint at 0x15accd240>, use_reentrant=True)
Thanks for your help!
System Info
transformers
version: 4.39.1Who can help?
@amyeroberts
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
When using gradient_checkpointing,
CLIPEncoder
calls theself._gradient_checkpointing_func
https://github.com/huggingface/transformers/blob/51bcadc10a569847b93a30dbe3a077037ae63bad/src/transformers/models/clip/modeling_clip.py#L628 However, it seems that this attribute is defined inPretrainedModel
, whileCLIPEncoder
is not a subclass ofPretrainedModel
. This bug can be reproduced using the following codewhich reports
'CLIPEncoder' object has no attribute '_gradient_checkpointing_func'
Expected behavior
Currently, I replace the
self.__gradient_checkpointing_func
directly withtorch.utils.checkpoint.checkpoint
, and it works when training with gradient checkpointing. However, I think this is not a perfect solution, since we need__gradient_checkpointing_func
to handle the gradient_checkpointing_kwargs.