Open yangzijia opened 2 months ago
Hello, in general those are both open research questions, but I can get you started with some simple strategies.
Though like I said, both of those questions are open points of research. Feel free to use what I suggested as your baseline and let me know if you can find anything that works better!
Hello Author,
Thank you for your contributions. I am currently looking to optimize the ImageEncoderViT method from “Segment Anything” using your token merging method, but I have encountered two issues:
- I noticed that the Block in the ImageEncoderViT uses windowed attention, and the shape of the tokens is (B, W, H, C), such as (1, 64, 64, 1280) for vit_h. This dimensionality cannot be processed by bipartite soft matching. I am considering whether merging W and H directly for computation would work. Do you have a better suggestion?
- The implementation of ToMe can reduce the number of tokens by about 98%, which changes the final feature shape. In the Image Encoder ViT, there are two Conv2d operations at the end, and after the token shape is changed, it cannot undergo convolution operations. I am wondering if adding a shape-expanding operation at this point would be feasible?
Thank you for your help.
Can I ask if you have made any progress?I'd like to try this but don't know how to go about it?
Hello Author,
Thank you for your contributions. I am currently looking to optimize the ImageEncoderViT method from “Segment Anything” using your token merging method, but I have encountered two issues:
Thank you for your help.