vojatravel
commented
3 months ago i dont have a solution, but where did you put the models for moondream and which one did you download. it seems im doing something wrong myself
Closed camoody1 closed 2 months ago
vojatravel
commented
3 months ago i dont have a solution, but where did you put the models for moondream and which one did you download. it seems im doing something wrong myself
gokayfem
commented
3 months ago downgrading transformers==4.36.2 fixes the problem if you want to use it. change the transformers in the requirements.txt inside ComfyUI_VLM_nodes.
camoody1
commented
3 months ago I changed the transformers line in the requirements.txt file for ComfyUI_VLM_nodes and this is my new error:
got prompt
Fetching 15 files: 100%|███████████████████████████████████████████████████████████████| 15/15 [00:04<00:00, 3.50it/s]
!!! Exception during processing !!!
Traceback (most recent call last):
File "H:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\utils\import_utils.py", line 1472, in _get_module
File "importlib\__init__.py", line 126, in import_module
File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
File "<frozen importlib._bootstrap>", line 1140, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'transformers.models.cohere.configuration_cohere'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\execution.py", line 148, in recursive_execute
obj = class_def()
^^^^^^^^^^^
File "H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_VLM_nodes\nodes\moondream2.py", line 39, in __init__
self.predictor = Moondream2Predictor()
^^^^^^^^^^^^^^^^^^^^^
File "H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_VLM_nodes\nodes\moondream2.py", line 24, in __init__
self.model = AutoModelForCausalLM.from_pretrained(self.model_path, trust_remote_code=True).to(self.device).eval()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "H:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\models\auto\auto_factory.py", line 540, in from_pretrained
kwargs["quantization_config"] = kwargs_orig["quantization_config"]
^^^^^^^^^^^^^^^^^^^^^^^^^
File "H:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\models\auto\auto_factory.py", line 751, in keys
module_name = model_type_to_module_name(model_type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "H:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\models\auto\auto_factory.py", line 752, in <listcomp>
if module_name not in self._modules:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "H:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\models\auto\auto_factory.py", line 748, in _load_attr_from_module
raise KeyError(key)
^^^^^^^^^^^^^
File "H:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\models\auto\auto_factory.py", line 692, in getattribute_from_module
File "H:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\utils\import_utils.py", line 1462, in __getattr__
File "H:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\utils\import_utils.py", line 1474, in _get_module
RuntimeError: Failed to import transformers.models.cohere.configuration_cohere because of the following error (look up to see its traceback):
No module named 'transformers.models.cohere.configuration_cohere'
Prompt executed in 5.01 secondslets go back to transformers>=4.38.2 and try this option
inside models/LLavacheckpoints/files_for_moondream2 folder
there is a file modelling_phi.py
do this change change line 386
if key_padding_mask is not None:
padding_mask = torch.full(
(batch_size, seqlen), -10000.0, dtype=scores.dtype, device=scores.device
)
padding_mask.masked_fill_(key_padding_mask, 0.0)
scores = scores + rearrange(padding_mask, "b s -> b 1 1 s")from this to this
if key_padding_mask is not None:
padding_mask = torch.full(
(batch_size, seqlen_k),
-10000.0,
dtype=scores.dtype,
device=scores.device,
)
key_padding_mask = key_padding_mask[:, :seqlen_k]
padding_mask.masked_fill_(key_padding_mask, 0.0)
scores = scores + rearrange(padding_mask, "b s -> b 1 1 s")I made the changes and restarted Comfy. It worked once. I then restarted Comfy, again, just to make sure it was going to stick. And now, it's not working, again. Every time I restart Comfy, it seems to be overwriting the changes I had made in the modelling_phi.py file. Here is the error message, now:
got prompt
!!! Exception during processing !!!
Traceback (most recent call last):
File "H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_VLM_nodes\nodes\moondream2.py", line 68, in moondream2_generate_predictions
response = self.predictor.generate_predictions(temp_path, text_input)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_VLM_nodes\nodes\moondream2.py", line 33, in generate_predictions
generated_text = self.model.answer_question(enc_image, question, self.tokenizer)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Christopher\.cache\huggingface\modules\transformers_modules\files_for_moondream2\moondream.py", line 92, in answer_question
answer = self.generate(
^^^^^^^^^^^^^^
File "C:\Users\Christopher\.cache\huggingface\modules\transformers_modules\files_for_moondream2\moondream.py", line 76, in generate
output_ids = self.text_model.generate(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "H:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "H:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\generation\utils.py", line 1527, in generate
result = self._greedy_search(
^^^^^^^^^^^^^^^^^^^^
File "H:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\generation\utils.py", line 2411, in _greedy_search
outputs = self(
^^^^^
File "H:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "H:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Christopher\.cache\huggingface\modules\transformers_modules\files_for_moondream2\modeling_phi.py", line 709, in forward
hidden_states = self.transformer(
^^^^^^^^^^^^^^^^^
File "H:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "H:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Christopher\.cache\huggingface\modules\transformers_modules\files_for_moondream2\modeling_phi.py", line 675, in forward
else func(*args)
^^^^^^^^^^^
File "H:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "H:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Christopher\.cache\huggingface\modules\transformers_modules\files_for_moondream2\modeling_phi.py", line 541, in forward
attn_outputs = self.mixer(
^^^^^^^^^^^
File "H:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "H:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Christopher\.cache\huggingface\modules\transformers_modules\files_for_moondream2\modeling_phi.py", line 514, in forward
attn_output_function(x, past_key_values, attention_mask)
File "C:\Users\Christopher\.cache\huggingface\modules\transformers_modules\files_for_moondream2\modeling_phi.py", line 494, in _forward_cross_attn
return attn_func(
^^^^^^^^^^
File "C:\Users\Christopher\.cache\huggingface\modules\transformers_modules\files_for_moondream2\modeling_phi.py", line 491, in <lambda>
else lambda fn, *args, **kwargs: fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "H:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "H:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "H:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\amp\autocast_mode.py", line 16, in decorate_autocast
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "H:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\amp\autocast_mode.py", line 16, in decorate_autocast
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Christopher\.cache\huggingface\modules\transformers_modules\files_for_moondream2\modeling_phi.py", line 318, in forward
padding_mask.masked_fill_(key_padding_mask, 0.0)
RuntimeError: The expanded size of the tensor (753) must match the existing size (754) at non-singleton dimension 1. Target sizes: [1, 753]. Tensor sizes: [1, 754]
Prompt executed in 9.64 secondsokay. inside ComfyUI_VLM_nodes/nodes
there is a moondream2.py
change the snapshot_download() function with this
old
class Moondream2Predictor:
def __init__(self):
self.model_path = snapshot_download("vikhyatk/moondream2",
local_dir=files_for_moondream2,
force_download=False, # Set to True if you always want to download, regardless of local copy
local_files_only=False, # Set to False to allow downloading if not available locally
revision="2024-03-04", # Specify the revision date for version control
local_dir_use_symlinks="auto", # or set to True/False based on your symlink preference
ignore_patterns=["*.bin", "*.jpg", "*.png"]) # Customize based on neednew
self.model_path = files_for_moondream2 this way it wont overwrite it.
camoody1
commented
3 months ago That change seemed to do the trick... until I performed an "Update All" from within Comfy Manager. That seems to have overwritten the two files, again. But before the update, I was able to restart ComfyUI a couple of times and it kept working.
gokayfem
commented
3 months ago when you make update the moondream2.py gets overwritten thats why
camoody1
commented
3 months ago when you make update the moondream2.py gets overwritten thats why
Well, yeah. I guessed that. 😂
Is this a code change you can make so overwriting won't be an issue? Or are you keeping it as is, even though it seems to break the node for some people? 😁
gokayfem
commented
3 months ago moondream2 is a new model, the creator constantly changing his codes, i will wait until he is finished with that. then i will fix the version.
camoody1
commented
3 months ago @gokayfem Thank you for your help working through this.
gokayfem
commented
3 months ago @camoody1
https://huggingface.co/gokaygokay/moondream-prompt
i fine tuned moondream2 to give stable diffusion prompts directly from the images, i will add this to repository soon.
camoody1
commented
3 months ago @gokayfem Nice! Thank youi
I'm getting the following error when trying to run the MoonDream2 node:
I have an RTX 3060 12GB using Windows. Any idea what could be the issue?