search

wangzy22 / P2P

[NeurIPS 2022 Spotlight] P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Prompting
https://p2p.ivg-research.xyz
MIT License
126 stars 9 forks source link

About colouring #5

Closed ZhangYuanhan-AI closed 1 year ago

ZhangYuanhan-AI commented 1 year ago

Hi,

Thanks for this exciting and excellent work!

From Table.3, we can indicate that coloring is essential for tuning the model.

I want to ask how to ensure the learned "lightweight 2D neural network gΦ" can predict RGB values. The predicted value of this model is might not in [0,255].

Or, can we refer to this model as an "Adapter" (an inserted module in the image-model for tuning tiny-sized parameters).

Looking forward to your insights！

wangzy22 commented 1 year ago

Hi, thanks for your interest in our work!

About the coloring module, the Sigmoid function in this line of models/layers/encoder.py ensures that the pixel values of the predicted image are within [0,1] interval. We then use IMAGENET_DEFAULT_MEAN and IMAGENET_DEFAULT_STD to normalize the image, following the common practice of processing a natural image in 2D domain.

Secondly, we don't think that our coloring module is an "Adapter". To the best of my knowledge, an "Adapter" is a task-specific module that is inserted in the pre-trained model and directly changes intermediate features. However, the proposed P2P module produces the input to the image model (an image) and doesn't affect its architecture design. We think it is a "Prompting" module, which converts downstream task (point cloud analysis under this condition) to pre-trained task (image classification) to leverage the pre-trained knowledge.

ZhangYuanhan-AI commented 1 year ago

Yes. I agree.

Then very interesting to know how D + C (Fig.3) performs. What do you think?

Have you done such experiments?

wangzy22 commented 1 year ago

That's an interesting idea. However, we haven't tried. Our paper mainly focuses on proposing a new prompting mechanism, thus ablations only compare P2P prompting with other prompting method like VPT.

I'm looking forward to see how well combining these two prompting would perform.

ZhangYuanhan-AI commented 1 year ago

ZhangYuanhan-AI

That's an interesting idea. However, we haven't tried. Our paper mainly focuses on proposing a new prompting mechanism, thus ablations only compare P2P prompting with other prompting method like VPT.

I'm looking forward to see how well combining these two prompting would perform.

Thanks!