search

sdc17 / UPop

[ICML 2023] UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformers.
https://dachuanshi.com/UPop-Project/
BSD 3-Clause "New" or "Revised" License
96 stars 5 forks source link

Problem with CLIP implementation #26

Closed jiseshen closed 1 year ago

jiseshen commented 1 year ago

There's a bug concerning CLIP implementation, since nn.MultiheadAttention does not have a parameter called "search", which is customized in your implementation of other modules to give the model a learnable pruning parameter. However, the command runs without error. I will appreciate it if you would help me with that.

sdc17 commented 1 year ago

Hi, @Jiseshen this is not a bug. When running the code, clip/mock.py dynamically replaces some functions of nn.MultiheadAttention with our own implementations and inserts new parameters UPop needs. For example, this line https://github.com/sdc17/UPop/blob/4ddd4bb85cfd3e0f4bbdb4a6f3c7dc0fbee5b399/clip/mock.py#L248 inserts the "search" into nn.MultiheadAttention. Some related explanations are also mentioned in #20.