search

dingjiansw101 / ZegFormer

Official code for "Decoupling Zero-Shot Semantic Segmentation"
Other
163 stars 4 forks source link

Why novel queries not overfit to the "no-object" class? #12

Closed yifliu3 closed 1 year ago

yifliu3 commented 1 year ago

Hello authors, thank you for this great work!

In my understanding, during the training period, instances with novel classes are also exposed to the network, so the queries of these novel instances will match the "no-object" textual embeddings by the cross entropy loss. In this case, the model may tend to ignore the novel instances during inference and cause severe performance drop. But it seems not a problem in the experiements, so I'm wondering if my understanding has something wrong.

dingjiansw101 commented 1 year ago

First, the cross entropy loss does not influence the mask generation of novel classes. Secondly, we want to predict high scores of novel classes and low scores of seen classes for novel instances. From my view, the "no-object" textual embeddings are learnable. The cross-entropy loss will make the "no-object" embeddings dissimilar to the seen class embeddings. If a novel instance is matched with "no-object", the features of a novel instance will also be dissimilar to the seen class embeddings, which may help to predict low scores of seen classes for the novel instances.

yifliu3 commented 1 year ago

Thanks a lot for your clear explanation!

yifliu3 commented 1 year ago

Hi, another two problems are:

  1. If the mask loss of"no-object" is ignored during the training period, then why the mask of novel instances can be predicted well during inference?
  2. During the training, some background things are also classified to be "no-object", so when testing, the background and novel instances are all classified to be "no-object", right?
dingjiansw101 commented 1 year ago
  1. From my understanding, there are some shared features (especially low-level features) between seen and novel classes. So the queries have some generalization ability to the novel classes. But not all novel classes can be segmented well. How to improve the recall and precision of novel class masks is an open problem.
  2. During testing, the "no-object" class scores are ignored. The instances are either assigned to seen or novel classes.
yifliu3 commented 1 year ago

I see, thanks a lot for your clear clarification!