Some suggestions on the source code

[PhucNDA]

Hi, thanks for the wonderful state-of-the-art work.

I have some difficulties while dealing with the source code, but everything is okay now. I have some (2) suggestions to make the method/repo look more approachable/attractive to people.

1. I see this source code shares similarity with OV-Seg with OpenVocab-vibe, so it's crucial to have some scripts supporting open-vocab prompting inference.
2. Sometimes people want to extract only visual features to do some extra tasks (distillation, 3D lifting, etc.) so it's recommended to have some instruction dealing with visual & text alignment. Currently, the repo (aligns text and visual via aggregation layers which is different from most OV method) only supports outputting the class-logit of pre-defined dataset.

Again, thanks for significant work, the work surpasses OV-Seg a significant gap. Above are some issues I saw when I was playing with the CAT-Seg. Looking forward to upcoming work.

Thank you, PhucNDA.

[zsc1220]

@PhucNDA Ask for help！！！Do you have the code for suggestion 1 about this project? Is it convenient for open source ? Thank you so much!

[zsc1220]

import numpy as np print(np)

[tofutoo]

你好贴主，我想问问在demo中我的sam模型识别出现了模型识别检查点出错的情况，想询问一下你是否有解决方法呢？不胜感激 Some model parameters or buffers are not found in the checkpoint: sam.image_encoder.blocks.0.attn.proj.{bias, weight} sam.image_encoder.blocks.0.attn.qkv.{bias, weight} sam.image_encoder.blocks.0.attn.{rel_pos_h, rel_pos_w} sam.image_encoder.blocks.0.mlp.lin1.{bias, weight} sam.image_encoder.blocks.0.mlp.lin2.{bias, weight} sam.image_encoder.blocks.0.norm1.{bias, weight} sam.image_encoder.blocks.0.norm2.{bias, weight} sam.image_encoder.blocks.1.attn.proj.{bias, weight} sam.image_encoder.blocks.1.attn.qkv.{bias, weight} sam.image_encoder.blocks.1.attn.{rel_pos_h, rel_pos_w} sam.image_encoder.blocks.1.mlp.lin1.{bias, weight} sam.image_encoder.blocks.1.mlp.lin2.{bias, weight} sam.image_encoder.blocks.1.norm1.{bias, weight} sam.image_encoder.blocks.1.norm2.{bias, weight} sam.image_encoder.blocks.10.attn.proj.{bias, weight} sam.image_encoder.blocks.10.attn.qkv.{bias, weight} sam.image_encoder.blocks.10.attn.{rel_pos_h, rel_pos_w} sam.image_encoder.blocks.10.mlp.lin1.{bias, weight} sam.image_encoder.blocks.10.mlp.lin2.{bias, weight} sam.image_encoder.blocks.10.norm1.{bias, weight} sam.image_encoder.blocks.10.norm2.{bias, weight} sam.image_encoder.blocks.11.attn.proj.{bias, weight} sam.image_encoder.blocks.11.attn.qkv.{bias, weight} sam.image_encoder.blocks.11.attn.{rel_pos_h, rel_pos_w} sam.image_encoder.blocks.11.mlp.lin1.{bias, weight} sam.image_encoder.blocks.11.mlp.lin2.{bias, weight} sam.image_encoder.blocks.11.norm1.{bias, weight} sam.image_encoder.blocks.11.norm2.{bias, weight} sam.image_encoder.blocks.12.attn.proj.{bias, weight} sam.image_encoder.blocks.12.attn.qkv.{bias, weight} sam.image_encoder.blocks.12.attn.{rel_pos_h, rel_pos_w} sam.image_encoder.blocks.12.mlp.lin1.{bias, weight} sam.image_encoder.blocks.12.mlp.lin2.{bias, weight} sam.image_encoder.blocks.12.norm1.{bias, weight} sam.image_encoder.blocks.12.norm2.{bias, weight} sam.image_encoder.blocks.13.attn.proj.{bias, weight} sam.image_encoder.blocks.13.attn.qkv.{bias, weight} sam.image_encoder.blocks.13.attn.{rel_pos_h, rel_pos_w} sam.image_encoder.blocks.13.mlp.lin1.{bias, weight} sam.image_encoder.blocks.13.mlp.lin2.{bias, weight} sam.image_encoder.blocks.13.norm1.{bias, weight} sam.image_encoder.blocks.13.norm2.{bias, weight} sam.image_encoder.blocks.14.attn.proj.{bias, weight} sam.image_encoder.blocks.14.attn.qkv.{bias, weight} sam.image_encoder.blocks.14.attn.{rel_pos_h, rel_pos_w} sam.image_encoder.blocks.14.mlp.lin1.{bias, weight} sam.image_encoder.blocks.14.mlp.lin2.{bias, weight} sam.image_encoder.blocks.14.norm1.{bias, weight} sam.image_encoder.blocks.14.norm2.{bias, weight} sam.image_encoder.blocks.15.attn.proj.{bias, weight} sam.image_encoder.blocks.15.attn.qkv.{bias, weight} sam.image_encoder.blocks.15.attn.{rel_pos_h, rel_pos_w} sam.image_encoder.blocks.15.mlp.lin1.{bias, weight} sam.image_encoder.blocks.15.mlp.lin2.{bias, weight} sam.image_encoder.blocks.15.norm1.{bias, weight} sam.image_encoder.blocks.15.norm2.{bias, weight} sam.image_encoder.blocks.16.attn.proj.{bias, weight} sam.image_encoder.blocks.16.attn.qkv.{bias, weight} sam.image_encoder.blocks.16.attn.{rel_pos_h, rel_pos_w} sam.image_encoder.blocks.16.mlp.lin1.{bias, weight} sam.image_encoder.blocks.16.mlp.lin2.{bias, weight} sam.image_encoder.blocks.16.norm1.{bias, weight} sam.image_encoder.blocks.16.norm2.{bias, weight} sam.image_encoder.blocks.17.attn.proj.{bias, weight} sam.image_encoder.blocks.17.attn.qkv.{bias, weight} sam.image_encoder.blocks.17.attn.{rel_pos_h, rel_pos_w} sam.image_encoder.blocks.17.mlp.lin1.{bias, weight} sam.image_encoder.blocks.17.mlp.lin2.{bias, weight} sam.image_encoder.blocks.17.norm1.{bias, weight} sam.image_encoder.blocks.17.norm2.{bias, weight} sam.image_encoder.blocks.18.attn.proj.{bias, weight} sam.image_encoder.blocks.18.attn.qkv.{bias, weight} sam.image_encoder.blocks.18.attn.{rel_pos_h, rel_pos_w} sam.image_encoder.blocks.18.mlp.lin1.{bias, weight} sam.image_encoder.blocks.18.mlp.lin2.{bias, weight} sam.image_encoder.blocks.18.norm1.{bias, weight} sam.image_encoder.blocks.18.norm2.{bias, weight} sam.image_encoder.blocks.19.attn.proj.{bias, weight} sam.image_encoder.blocks.19.attn.qkv.{bias, weight} sam.image_encoder.blocks.19.attn.{rel_pos_h, rel_pos_w} sam.image_encoder.blocks.19.mlp.lin1.{bias, weight} sam.image_encoder.blocks.19.mlp.lin2.{bias, weight} sam.image_encoder.blocks.19.norm1.{bias, weight} sam.image_encoder.blocks.19.norm2.{bias, weight} sam.image_encoder.blocks.2.attn.proj.{bias, weight} sam.image_encoder.blocks.2.attn.qkv.{bias, weight} sam.image_encoder.blocks.2.attn.{rel_pos_h, rel_pos_w} sam.image_encoder.blocks.2.mlp.lin1.{bias, weight} sam.image_encoder.blocks.2.mlp.lin2.{bias, weight} sam.image_encoder.blocks.2.norm1.{bias, weight} sam.image_encoder.blocks.2.norm2.{bias, weight} sam.image_encoder.blocks.20.attn.proj.{bias, weight} sam.image_encoder.blocks.20.attn.qkv.{bias, weight} sam.image_encoder.blocks.20.attn.{rel_pos_h, rel_pos_w} sam.image_encoder.blocks.20.mlp.lin1.{bias, weight} sam.image_encoder.blocks.20.mlp.lin2.{bias, weight} sam.image_encoder.blocks.20.norm1.{bias, weight} sam.image_encoder.blocks.20.norm2.{bias, weight} sam.image_encoder.blocks.21.attn.proj.{bias, weight} sam.image_encoder.blocks.21.attn.qkv.{bias, weight} sam.image_encoder.blocks.21.attn.{rel_pos_h, rel_pos_w} sam.image_encoder.blocks.21.mlp.lin1.{bias, weight} sam.image_encoder.blocks.21.mlp.lin2.{bias, weight} sam.image_encoder.blocks.21.norm1.{bias, weight} sam.image_encoder.blocks.21.norm2.{bias, weight} sam.image_encoder.blocks.22.attn.proj.{bias, weight} sam.image_encoder.blocks.22.attn.qkv.{bias, weight} sam.image_encoder.blocks.22.attn.{rel_pos_h, rel_pos_w} sam.image_encoder.blocks.22.mlp.lin1.{bias, weight} sam.image_encoder.blocks.22.mlp.lin2.{bias, weight} sam.image_encoder.blocks.22.norm1.{bias, weight} sam.image_encoder.blocks.22.norm2.{bias, weight} sam.image_encoder.blocks.23.attn.proj.{bias, weight} sam.image_encoder.blocks.23.attn.qkv.{bias, weight} sam.image_encoder.blocks.23.attn.{rel_pos_h, rel_pos_w} sam.image_encoder.blocks.23.mlp.lin1.{bias, weight} sam.image_encoder.blocks.23.mlp.lin2.{bias, weight} sam.image_encoder.blocks.23.norm1.{bias, weight} sam.image_encoder.blocks.23.norm2.{bias, weight} sam.image_encoder.blocks.24.attn.proj.{bias, weight} sam.image_encoder.blocks.24.attn.qkv.{bias, weight} sam.image_encoder.blocks.24.attn.{rel_pos_h, rel_pos_w} sam.image_encoder.blocks.24.mlp.lin1.{bias, weight} sam.image_encoder.blocks.24.mlp.lin2.{bias, weight} sam.image_encoder.blocks.24.norm1.{bias, weight} sam.image_encoder.blocks.24.norm2.{bias, weight} sam.image_encoder.blocks.25.attn.proj.{bias, weight} sam.image_encoder.blocks.25.attn.qkv.{bias, weight} sam.image_encoder.blocks.25.attn.{rel_pos_h, rel_pos_w} sam.image_encoder.blocks.25.mlp.lin1.{bias, weight} sam.image_encoder.blocks.25.mlp.lin2.{bias, weight} sam.image_encoder.blocks.25.norm1.{bias, weight} sam.image_encoder.blocks.25.norm2.{bias, weight} sam.image_encoder.blocks.26.attn.proj.{bias, weight} sam.image_encoder.blocks.26.attn.qkv.{bias, weight} sam.image_encoder.blocks.26.attn.{rel_pos_h, rel_pos_w} sam.image_encoder.blocks.26.mlp.lin1.{bias, weight} sam.image_encoder.blocks.26.mlp.lin2.{bias, weight} sam.image_encoder.blocks.26.norm1.{bias, weight} sam.image_encoder.blocks.26.norm2.{bias, weight} sam.image_encoder.blocks.27.attn.proj.{bias, weight} sam.image_encoder.blocks.27.attn.qkv.{bias, weight} sam.image_encoder.blocks.27.attn.{rel_pos_h, rel_pos_w} sam.image_encoder.blocks.27.mlp.lin1.{bias, weight} sam.image_encoder.blocks.27.mlp.lin2.{bias, weight} sam.image_encoder.blocks.27.norm1.{bias, weight} sam.image_encoder.blocks.27.norm2.{bias, weight} sam.image_encoder.blocks.28.attn.proj.{bias, weight} sam.image_encoder.blocks.28.attn.qkv.{bias, weight} sam.image_encoder.blocks.28.attn.{rel_pos_h, rel_pos_w} sam.image_encoder.blocks.28.mlp.lin1.{bias, weight} sam.image_encoder.blocks.28.mlp.lin2.{bias, weight} sam.image_encoder.blocks.28.norm1.{bias, weight} sam.image_encoder.blocks.28.norm2.{bias, weight} sam.image_encoder.blocks.29.attn.proj.{bias, weight} sam.image_encoder.blocks.29.attn.qkv.{bias, weight} sam.image_encoder.blocks.29.attn.{rel_pos_h, rel_pos_w} sam.image_encoder.blocks.29.mlp.lin1.{bias, weight} sam.image_encoder.blocks.29.mlp.lin2.{bias, weight} sam.image_encoder.blocks.29.norm1.{bias, weight} sam.image_encoder.blocks.29.norm2.{bias, weight} sam.image_encoder.blocks.3.attn.proj.{bias, weight} sam.image_encoder.blocks.3.attn.qkv.{bias, weight} sam.image_encoder.blocks.3.attn.{rel_pos_h, rel_pos_w} sam.image_encoder.blocks.3.mlp.lin1.{bias, weight} sam.image_encoder.blocks.3.mlp.lin2.{bias, weight} sam.image_encoder.blocks.3.norm1.{bias, weight} sam.image_encoder.blocks.3.norm2.{bias, weight} sam.image_encoder.blocks.30.attn.proj.{bias, weight} sam.image_encoder.blocks.30.attn.qkv.{bias, weight} sam.image_encoder.blocks.30.attn.{rel_pos_h, rel_pos_w} sam.image_encoder.blocks.30.mlp.lin1.{bias, weight} sam.image_encoder.blocks.30.mlp.lin2.{bias, weight} sam.image_encoder.blocks.30.norm1.{bias, weight} sam.image_encoder.blocks.30.norm2.{bias, weight} sam.image_encoder.blocks.31.attn.proj.{bias, weight} sam.image_encoder.blocks.31.attn.qkv.{bias, weight} sam.image_encoder.blocks.31.attn.{rel_pos_h, rel_pos_w} sam.image_encoder.blocks.31.mlp.lin1.{bias, weight} sam.image_encoder.blocks.31.mlp.lin2.{bias, weight} sam.image_encoder.blocks.31.norm1.{bias, weight} sam.image_encoder.blocks.31.norm2.{bias, weight} sam.image_encoder.blocks.4.attn.proj.{bias, weight} sam.image_encoder.blocks.4.attn.qkv.{bias, weight} sam.image_encoder.blocks.4.attn.{rel_pos_h, rel_pos_w} sam.image_encoder.blocks.4.mlp.lin1.{bias, weight} sam.image_encoder.blocks.4.mlp.lin2.{bias, weight} sam.image_encoder.blocks.4.norm1.{bias, weight} sam.image_encoder.blocks.4.norm2.{bias, weight} sam.image_encoder.blocks.5.attn.proj.{bias, weight} sam.image_encoder.blocks.5.attn.qkv.{bias, weight} sam.image_encoder.blocks.5.attn.{rel_pos_h, rel_pos_w} sam.image_encoder.blocks.5.mlp.lin1.{bias, weight} sam.image_encoder.blocks.5.mlp.lin2.{bias, weight} sam.image_encoder.blocks.5.norm1.{bias, weight} sam.image_encoder.blocks.5.norm2.{bias, weight} sam.image_encoder.blocks.6.attn.proj.{bias, weight} sam.image_encoder.blocks.6.attn.qkv.{bias, weight} sam.image_encoder.blocks.6.attn.{rel_pos_h, rel_pos_w} sam.image_encoder.blocks.6.mlp.lin1.{bias, weight} sam.image_encoder.blocks.6.mlp.lin2.{bias, weight} sam.image_encoder.blocks.6.norm1.{bias, weight} sam.image_encoder.blocks.6.norm2.{bias, weight} sam.image_encoder.blocks.7.attn.proj.{bias, weight} sam.image_encoder.blocks.7.attn.qkv.{bias, weight} sam.image_encoder.blocks.7.attn.{rel_pos_h, rel_pos_w} sam.image_encoder.blocks.7.mlp.lin1.{bias, weight} sam.image_encoder.blocks.7.mlp.lin2.{bias, weight} sam.image_encoder.blocks.7.norm1.{bias, weight} sam.image_encoder.blocks.7.norm2.{bias, weight} sam.image_encoder.blocks.8.attn.proj.{bias, weight} sam.image_encoder.blocks.8.attn.qkv.{bias, weight} sam.image_encoder.blocks.8.attn.{rel_pos_h, rel_pos_w} sam.image_encoder.blocks.8.mlp.lin1.{bias, weight} sam.image_encoder.blocks.8.mlp.lin2.{bias, weight} sam.image_encoder.blocks.8.norm1.{bias, weight} sam.image_encoder.blocks.8.norm2.{bias, weight} sam.image_encoder.blocks.9.attn.proj.{bias, weight} sam.image_encoder.blocks.9.attn.qkv.{bias, weight} sam.image_encoder.blocks.9.attn.{rel_pos_h, rel_pos_w} sam.image_encoder.blocks.9.mlp.lin1.{bias, weight} sam.image_encoder.blocks.9.mlp.lin2.{bias, weight} sam.image_encoder.blocks.9.norm1.{bias, weight} sam.image_encoder.blocks.9.norm2.{bias, weight} sam.image_encoder.neck.0.weight sam.image_encoder.neck.1.{bias, weight} sam.image_encoder.neck.2.weight sam.image_encoder.neck.3.{bias, weight} sam.image_encoder.patch_embed.proj.{bias, weight} sam.image_encoder.pos_embed sam.mask_decoder.iou_prediction_head.layers.0.{bias, weight} sam.mask_decoder.iou_prediction_head.layers.1.{bias, weight} sam.mask_decoder.iou_prediction_head.layers.2.{bias, weight} sam.mask_decoder.iou_token.weight sam.mask_decoder.mask_tokens.weight sam.mask_decoder.output_hypernetworks_mlps.0.layers.0.{bias, weight} sam.mask_decoder.output_hypernetworks_mlps.0.layers.1.{bias, weight} sam.mask_decoder.output_hypernetworks_mlps.0.layers.2.{bias, weight} sam.mask_decoder.output_hypernetworks_mlps.1.layers.0.{bias, weight} sam.mask_decoder.output_hypernetworks_mlps.1.layers.1.{bias, weight} sam.mask_decoder.output_hypernetworks_mlps.1.layers.2.{bias, weight} sam.mask_decoder.output_hypernetworks_mlps.2.layers.0.{bias, weight} sam.mask_decoder.output_hypernetworks_mlps.2.layers.1.{bias, weight} sam.mask_decoder.output_hypernetworks_mlps.2.layers.2.{bias, weight} sam.mask_decoder.output_hypernetworks_mlps.3.layers.0.{bias, weight} sam.mask_decoder.output_hypernetworks_mlps.3.layers.1.{bias, weight} sam.mask_decoder.output_hypernetworks_mlps.3.layers.2.{bias, weight} sam.mask_decoder.output_upscaling.0.{bias, weight} sam.mask_decoder.output_upscaling.1.{bias, weight} sam.mask_decoder.output_upscaling.3.{bias, weight} sam.mask_decoder.transformer.final_attn_token_to_image.k_proj.{bias, weight} sam.mask_decoder.transformer.final_attn_token_to_image.out_proj.{bias, weight} sam.mask_decoder.transformer.final_attn_token_to_image.q_proj.{bias, weight} sam.mask_decoder.transformer.final_attn_token_to_image.v_proj.{bias, weight} sam.mask_decoder.transformer.layers.0.cross_attn_image_to_token.k_proj.{bias, weight} sam.mask_decoder.transformer.layers.0.cross_attn_image_to_token.out_proj.{bias, weight} sam.mask_decoder.transformer.layers.0.cross_attn_image_to_token.q_proj.{bias, weight} sam.mask_decoder.transformer.layers.0.cross_attn_image_to_token.v_proj.{bias, weight} sam.mask_decoder.transformer.layers.0.cross_attn_token_to_image.k_proj.{bias, weight} sam.mask_decoder.transformer.layers.0.cross_attn_token_to_image.out_proj.{bias, weight} sam.mask_decoder.transformer.layers.0.cross_attn_token_to_image.q_proj.{bias, weight} sam.mask_decoder.transformer.layers.0.cross_attn_token_to_image.v_proj.{bias, weight} sam.mask_decoder.transformer.layers.0.mlp.lin1.{bias, weight} sam.mask_decoder.transformer.layers.0.mlp.lin2.{bias, weight} sam.mask_decoder.transformer.layers.0.norm1.{bias, weight} sam.mask_decoder.transformer.layers.0.norm2.{bias, weight} sam.mask_decoder.transformer.layers.0.norm3.{bias, weight} sam.mask_decoder.transformer.layers.0.norm4.{bias, weight} sam.mask_decoder.transformer.layers.0.self_attn.k_proj.{bias, weight} sam.mask_decoder.transformer.layers.0.self_attn.out_proj.{bias, weight} sam.mask_decoder.transformer.layers.0.self_attn.q_proj.{bias, weight} sam.mask_decoder.transformer.layers.0.self_attn.v_proj.{bias, weight} sam.mask_decoder.transformer.layers.1.cross_attn_image_to_token.k_proj.{bias, weight} sam.mask_decoder.transformer.layers.1.cross_attn_image_to_token.out_proj.{bias, weight} sam.mask_decoder.transformer.layers.1.cross_attn_image_to_token.q_proj.{bias, weight} sam.mask_decoder.transformer.layers.1.cross_attn_image_to_token.v_proj.{bias, weight} sam.mask_decoder.transformer.layers.1.cross_attn_token_to_image.k_proj.{bias, weight} sam.mask_decoder.transformer.layers.1.cross_attn_token_to_image.out_proj.{bias, weight} sam.mask_decoder.transformer.layers.1.cross_attn_token_to_image.q_proj.{bias, weight} sam.mask_decoder.transformer.layers.1.cross_attn_token_to_image.v_proj.{bias, weight} sam.mask_decoder.transformer.layers.1.mlp.lin1.{bias, weight} sam.mask_decoder.transformer.layers.1.mlp.lin2.{bias, weight} sam.mask_decoder.transformer.layers.1.norm1.{bias, weight} sam.mask_decoder.transformer.layers.1.norm2.{bias, weight} sam.mask_decoder.transformer.layers.1.norm3.{bias, weight} sam.mask_decoder.transformer.layers.1.norm4.{bias, weight} sam.mask_decoder.transformer.layers.1.self_attn.k_proj.{bias, weight} sam.mask_decoder.transformer.layers.1.self_attn.out_proj.{bias, weight} sam.mask_decoder.transformer.layers.1.self_attn.q_proj.{bias, weight} sam.mask_decoder.transformer.layers.1.self_attn.v_proj.{bias, weight} sam.mask_decoder.transformer.norm_final_attn.{bias, weight} sam.prompt_encoder.mask_downscaling.0.{bias, weight} sam.prompt_encoder.mask_downscaling.1.{bias, weight} sam.prompt_encoder.mask_downscaling.3.{bias, weight} sam.prompt_encoder.mask_downscaling.4.{bias, weight} sam.prompt_encoder.mask_downscaling.6.{bias, weight} sam.prompt_encoder.no_mask_embed.weight sam.prompt_encoder.not_a_point_embed.weight sam.prompt_encoder.pe_layer.positional_encoding_gaussian_matrix sam.prompt_encoder.point_embeddings.0.weight sam.prompt_encoder.point_embeddings.1.weight sam.prompt_encoder.point_embeddings.2.weight sam.prompt_encoder.point_embeddings.3.weight