evaluation internlm-xcomposer2-vl-7b get 10% acc on mmbench-dev-cn, not 78.3%

InternLM / InternLM-XComposer

InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output

Apache License 2.0

2.47k stars 153 forks source link

class CLIPVisionTower(nn.Module): def __init__(self, vision_tower): super().__init__() self.is_loaded = False self.is_resize_pos = False from modelscope import snapshot_download self.vision_tower_name = snapshot_download("AI-ModelScope/clip-vit-large-patch14-336") ### different self.select_layer = -1 self.select_feature = 'patch' self.load_model() self.resize_pos()

I use the weight from internlm-xcomposer2-vl-7b. The evaluation code warning when loading the model.
weights of ['vit.vision_tower.vision_model_post_layernorm.weight', 'vit.vision_tower.vision_model.post_layernorm.bias'] are newly initialized 
is this the reason for the poor performance in mmbench-dev-cn?

please note the code internlm-xcomposer2-vl-7b/build_mlp.py in modelscope is different from internlm-xcomposer2-vl-7b/build_mlp.py in huggingface
class CLIPVisionTower(nn.Module):

    def __init__(self, vision_tower):
        super().__init__()

        self.is_loaded = False
        self.is_resize_pos = False
        from modelscope import snapshot_download
        self.vision_tower_name = snapshot_download("AI-ModelScope/clip-vit-large-patch14-336")  ### different 
        self.select_layer = -1
        self.select_feature = 'patch'
        self.load_model()
        self.resize_pos()

use the code from branch InternLM-XComposer2 instead of main fix the problem, after fix the import library bug in code, the acc is matched.

InternLM / InternLM-XComposer

evaluation internlm-xcomposer2-vl-7b get 10% acc on mmbench-dev-cn, not 78.3% #198