Some hesitations related to background class label index

[Tavarich]

Hi, Thanks for your work. I met similar problem about environment nbkn865 said as #7, but I haven't try your further response. Now I'm trying to transfer this work to new mmdet2.x. However, mmdet2.x have some different characteristics, one is that the background label is not 0 but the num_classes.

I wonder if your work still efficient after the background label changed.?

Also, I wonder if these semantic vectors start from the "background", that is to say the index of "background" vector(attribute) is 0?

And in mmdetection/tools/test.py, I found code as

 model.bbox_head.seen_bg_weight = torch.from_numpy(seen_bg_weight).cuda()
 model.bbox_head.seen_bg_bias = torch.from_numpy(seen_bg_bias).cuda()

but I think the model has no attr as seen_bg_weight and see_bg_bias. Do you mean it refers to 0 index (last index in new mmdet) of fc layer weights or do you modify something?

Looking forward to your answer, thanks!

[sandipan211]

Hello, I have not tried this code out in mmdet2.x, but once you figure out the dependencies and set up your environment, it should work. However, yes, in the current code, background label is assigned to 0. If you need to change it to num_classes, you just have to make the corresponding changes.

but I think the model has no attr as seen_bg_weight and see_bg_bias. Do you mean it refers to 0 index (last index in new mmdet) of fc layer weights or do you modify something?

When our GAN generator is trained, it is used to generate unseen class features and train a classifier. The weights of this classiifer are copied to the Faster-RCNN classifier to update it. So yes seen_bg weights and biases are referring to the 0 index. You can go through this function definition to understand what's going on and from where seen_bg gets weights and biases. This function is called just before assigning model.seen_bg bias and weights.

[Tavarich]

Thank you for your reply. I have tried adjusting these projects on the new framework mmdet2.x，It apparently run successfully as well. However, the classifier trained on the new framework has poor zero-shot detection performance. I think I may ignore many details about zsd settings. It's really a hard work to modify the project on a framework level (like mmdet2.x). So I have to go back and use the mmdet1.0 framework that you have used before. Luckily, after switching to a lower version of PyTorch, I was able to successfully set up mmdet1.0. I retrained the GAN network and classifier and achieved similar results in your zero-shot detection evaluation. If I have time later, I will try again with a new framework. Any way, thanks.

[sandipan211]

ZSD can be a hard setting to get acquainted with initially, especially if you are relatively new to it, so I understand. However, I'm glad you could reproduce the results.

[EfaJow]

@Tavarich Hi, I also retrained the GAN and classifier with mmdet1.0, however, I couldn't achieve the similar results. Did you changed any part of the code? If you could give me any suggestion, I'd appreciate it.

[Tavarich]

@EfaJow Sorry for the late reply. In fact, I probably didn't make too many improvements, mainly running it according to the process. This project is somewhat complex and requires careful completion of each step. For example, train the backbone using the Seen categories, generate correct Seen category visual features, and so on. Pay attention to the information of the training process (train and val messages), and print the intermediate results (such as TSNE plot), which may help you verify whether the steps are correct.