search

gokayfem / ComfyUI_VLM_nodes

Custom ComfyUI nodes for Vision Language Models, Large Language Models, Image to Music, Text to Music, Consistent and Random Creative Prompt Generation
Apache License 2.0
384 stars 31 forks source link

RuntimeError: Unknown model (vit_so400m_patch14_siglip_384) #45

Closed formulake closed 6 months ago

formulake commented 6 months ago

When I try running either of the moondream nodes I get this traceback error

!!! Exception during processing !!! Traceback (most recent call last): File "F:\Tools\ComfyUI\execution.py", line 148, in recursive_execute obj = class_def() File "F:\Tools\ComfyUI\custom_nodes\ComfyUI_VLM_nodes\nodes\moondream2.py", line 39, in init self.predictor = Moondream2Predictor() File "F:\Tools\ComfyUI\custom_nodes\ComfyUI_VLM_nodes\nodes\moondream2.py", line 24, in init self.model = AutoModelForCausalLM.from_pretrained(self.model_path, trust_remote_code=True).to(self.device).eval() File "F:\Tools\ComfyUI\venv\lib\site-packages\transformers\models\auto\auto_factory.py", line 561, in from_pretrained return model_class.from_pretrained( File "F:\Tools\ComfyUI\venv\lib\site-packages\transformers\modeling_utils.py", line 3462, in from_pretrained model = cls(config, *model_args, **model_kwargs) File "C:\Users\kunal.cache\huggingface\modules\transformers_modules\files_for_moondream2\moondream.py", line 15, in init self.vision_encoder = VisionEncoder() File "C:\Users\kunal.cache\huggingface\modules\transformers_modules\files_for_moondream2\vision_encoder.py", line 98, in init VisualHolder(timm.create_model("vit_so400m_patch14_siglip_384")) File "F:\Tools\ComfyUI\venv\lib\site-packages\timm\models\factory.py", line 67, in create_model raise RuntimeError('Unknown model (%s)' % model_name) RuntimeError: Unknown model (vit_so400m_patch14_siglip_384)

I've located and downloaded the missing vit_so400m_patch14_siglip_384.safetensors file and tried putting it in the moondream2 folder, the clip_vision folder and the CLIP

gokayfem commented 6 months ago

this is related to transformers or timm library i think. you shouldnot put any file inside clip_vision it should automatically find it.

Yonggie commented 4 months ago

Just upgrade the timm lib and it'll work fine.