Dynamically changing vocabulary

[joshmyersdean]

Hello!

Thank you for this great work! I was wondering if there's an easy way that I'm missing to dynamically change the custom vocabulary per sample? Or is this a limitation of detectron2 (i.e., not being able to change Metadata)?

[PeizeSun]

Hi, I guess there is a way to change the custom vocabulary per sample. For example, input [custom vocabulary] to [run_on_image], run [self.metadata.thing_classes = custom_vocabulary.split(',')] [classifier = get_clip_embeddings(self.metadata.thing_classes)] [reset_cls_test(self.predictor.model, classifier)] within [run_on_image]. This might work.

[joshmyersdean]

I tried that and unfortunately it did not work for me. I encountered this error

  146 print(custom_vocabulary)
    147 if custom_vocabulary is not None:
--> 148     self.metadata.thing_classes = custom_vocabulary.split(',')
    149     classifier = get_clip_embeddings(self.metadata.thing_classes)
    150     reset_cls_test(self.predictor.model, classifier)

File ~/detectron2/detectron2/data/catalog.py:148, in Metadata.__setattr__(self, key, val)
    146 try:
    147     oldval = getattr(self, key)
--> 148     assert oldval == val, (
    149         "Attribute '{}' in the metadata of '{}' cannot be set "
    150         "to a different value!\n{} != {}".format(key, self.name, oldval, val)
    151     )
    152 except AttributeError:
    153     super().__setattr__(key, val)

AssertionError: Attribute 'thing_classes' in the metadata of '__unused' cannot be set to a different value!
['shirt', ' top', ' sweater', ' cardigan', ' jacket', ' vest', ' pants', ' shorts', ' skirt', ' coat', ' dress', ' jumpsuit', ' cape', ' glasses', ' hat', ' headband', ' tie', ' glove', ' watch', ' belt', ' leg warmer', ' tights', ' sock', ' shoe', ' bag', ' scarf', ' umbrella', ' hood', ' collar', ' lapel', ' epaulette', ' sleeve', ' pocket', ' neckline', ' buckle', ' zipper', ' applique', ' bead', ' bow', ' flower', ' fringe', ' ribbon', ' rivet', ' ruffle', ' sequin', 'tassel'] != ['person']

[PeizeSun]

I tried that and unfortunately it did not work for me. I encountered this error

  146 print(custom_vocabulary)
    147 if custom_vocabulary is not None:
--> 148     self.metadata.thing_classes = custom_vocabulary.split(',')
    149     classifier = get_clip_embeddings(self.metadata.thing_classes)
    150     reset_cls_test(self.predictor.model, classifier)

File ~/detectron2/detectron2/data/catalog.py:148, in Metadata.__setattr__(self, key, val)
    146 try:
    147     oldval = getattr(self, key)
--> 148     assert oldval == val, (
    149         "Attribute '{}' in the metadata of '{}' cannot be set "
    150         "to a different value!\n{} != {}".format(key, self.name, oldval, val)
    151     )
    152 except AttributeError:
    153     super().__setattr__(key, val)

AssertionError: Attribute 'thing_classes' in the metadata of '__unused' cannot be set to a different value!
['shirt', ' top', ' sweater', ' cardigan', ' jacket', ' vest', ' pants', ' shorts', ' skirt', ' coat', ' dress', ' jumpsuit', ' cape', ' glasses', ' hat', ' headband', ' tie', ' glove', ' watch', ' belt', ' leg warmer', ' tights', ' sock', ' shoe', ' bag', ' scarf', ' umbrella', ' hood', ' collar', ' lapel', ' epaulette', ' sleeve', ' pocket', ' neckline', ' buckle', ' zipper', ' applique', ' bead', ' bow', ' flower', ' fringe', ' ribbon', ' rivet', ' ruffle', ' sequin', 'tassel'] != ['person']

How about add [self.metadata = MetadataCatalog.get("__unused")] before [self.metadata.thing_classes = custom_vocabulary.split(',')] ?

[joshmyersdean]

Adding MetadataCatalog.remove("__unused") before that ended up being the trick. Thank you for the help!