search

EvilBT / ComfyUI_SLK_joy_caption_two

ComfyUI Node
GNU General Public License v3.0
180 stars 5 forks source link

Joy_caption_two直接报错， #1

Open ohahaps2 opened 1 month ago

ohahaps2 commented 1 month ago

ComfyUI Error Report

Error Details

删除手动下载文件，让它自动下载models\clip\siglip-so400m-patch14-384 也是一样的错误，该怎么办 大佬？

ohahaps2 commented 1 month ago

pip install requirements.txt后新的错误又出来了. rope_scaling must be a dictionary with two fields, type and factor, got {'factor': 8.0, 'high_freq_factor': 4.0, 'low_freq_factor': 1.0, 'original_max_position_embeddings': 8192, 'rope_type': 'llama3'}

ComfyUI Error Report

Error Details

2024-10-12 20:19:38,924 - root - INFO - got prompt 2024-10-12 20:19:40,746 - root - INFO - Requested to load SiglipVisionTransformer 2024-10-12 20:19:40,747 - root - INFO - Loading 1 new model 2024-10-12 20:19:41,034 - root - INFO - loaded completely 0.0 1618.345947265625 True 2024-10-12 20:19:41,569 - root - INFO - Requested to load ImageAdapter 2024-10-12 20:19:41,569 - root - INFO - Loading 1 new model 2024-10-12 20:19:41,583 - root - INFO - loaded completely 0.0 82.078125 True 2024-10-12 20:19:41,937 - root - ERROR - !!! Exception during processing !!! rope_scaling must be a dictionary with two fields, type and factor, got {'factor': 8.0, 'high_freq_factor': 4.0, 'low_freq_factor': 1.0, 'original_max_position_embeddings': 8192, 'rope_type': 'llama3'} 2024-10-12 20:19:41,940 - root - ERROR - Traceback (most recent call last): File "H:\sd3\ComfyUI\execution.py", line 323, in execute output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "H:\sd3\ComfyUI\execution.py", line 198, in get_output_data return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "H:\sd3\ComfyUI\execution.py", line 169, in _map_node_over_list process_inputs(input_dict, i) File "H:\sd3\ComfyUI\execution.py", line 158, in process_inputs results.append(getattr(obj, func)(inputs)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "H:\sd3\ComfyUI\custom_nodes\ComfyUI_SLK_joy_caption_two\joy_caption_two_node.py", line 374, in generate text_model = joy_two_pipeline.llm.load_llm_model(joy_two_pipeline.model) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "H:\sd3\ComfyUI\custom_nodes\ComfyUI_SLK_joy_caption_two\joy_caption_two_node.py", line 172, in load_llm_model text_model = AutoModelForCausalLM.from_pretrained(text_model_path, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "H:\sd3\python_embeded\Lib\site-packages\transformers\models\auto\auto_factory.py", line 523, in from_pretrained config, kwargs = AutoConfig.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "H:\sd3\python_embeded\Lib\site-packages\transformers\models\auto\configuration_auto.py", line 958, in from_pretrained return config_class.from_dict(config_dict, unused_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "H:\sd3\python_embeded\Lib\site-packages\transformers\configuration_utils.py", line 768, in from_dict config = cls(**config_dict) ^^^^^^^^^^^^^^^^^^ File "H:\sd3\python_embeded\Lib\site-packages\transformers\models\llama\configuration_llama.py", line 161, in init self._rope_scaling_validation() File "H:\sd3\python_embeded\Lib\site-packages\transformers\models\llama\configuration_llama.py", line 182, in _rope_scaling_validation raise ValueError( ValueError: rope_scaling must be a dictionary with two fields, type and factor, got {'factor': 8.0, 'high_freq_factor': 4.0, 'low_freq_factor': 1.0, 'original_max_position_embeddings': 8192, 'rope_type': 'llama3'}

2024-10-12 20:19:41,943 - root - INFO - Prompt executed in 3.00 seconds 2024-10-12 20:20:47,288 - root - INFO - got prompt 2024-10-12 20:20:47,538 - root - ERROR - !!! Exception during processing !!! rope_scaling must be a dictionary with two fields, type and factor, got {'factor': 8.0, 'high_freq_factor': 4.0, 'low_freq_factor': 1.0, 'original_max_position_embeddings': 8192, 'rope_type': 'llama3'} 2024-10-12 20:20:47,539 - root - ERROR - Traceback (most recent call last): File "H:\sd3\ComfyUI\execution.py", line 323, in execute output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "H:\sd3\ComfyUI\execution.py", line 198, in get_output_data return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "H:\sd3\ComfyUI\execution.py", line 169, in _map_node_over_list process_inputs(input_dict, i) File "H:\sd3\ComfyUI\execution.py", line 158, in process_inputs results.append(getattr(obj, func)(inputs)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "H:\sd3\ComfyUI\custom_nodes\ComfyUI_SLK_joy_caption_two\joy_caption_two_node.py", line 374, in generate text_model = joy_two_pipeline.llm.load_llm_model(joy_two_pipeline.model) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "H:\sd3\ComfyUI\custom_nodes\ComfyUI_SLK_joy_caption_two\joy_caption_two_node.py", line 172, in load_llm_model text_model = AutoModelForCausalLM.from_pretrained(text_model_path, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "H:\sd3\python_embeded\Lib\site-packages\transformers\models\auto\auto_factory.py", line 523, in from_pretrained config, kwargs = AutoConfig.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "H:\sd3\python_embeded\Lib\site-packages\transformers\models\auto\configuration_auto.py", line 958, in from_pretrained return config_class.from_dict(config_dict, unused_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "H:\sd3\python_embeded\Lib\site-packages\transformers\configuration_utils.py", line 768, in from_dict config = cls(**config_dict) ^^^^^^^^^^^^^^^^^^ File "H:\sd3\python_embeded\Lib\site-packages\transformers\models\llama\configuration_llama.py", line 161, in init self._rope_scaling_validation() File "H:\sd3\python_embeded\Lib\site-packages\transformers\models\llama\configuration_llama.py", line 182, in _rope_scaling_validation raise ValueError( ValueError: rope_scaling must be a dictionary with two fields, type and factor, got {'factor': 8.0, 'high_freq_factor': 4.0, 'low_freq_factor': 1.0, 'original_max_position_embeddings': 8192, 'rope_type': 'llama3'}

2024-10-12 20:20:47,540 - root - INFO - Prompt executed in 0.23 seconds 2024-10-12 20:21:45,770 - root - INFO - got prompt 2024-10-12 20:21:45,928 - root - ERROR - !!! Exception during processing !!! rope_scaling must be a dictionary with two fields, type and factor, got {'factor': 8.0, 'high_freq_factor': 4.0, 'low_freq_factor': 1.0, 'original_max_position_embeddings': 8192, 'rope_type': 'llama3'} 2024-10-12 20:21:45,929 - root - ERROR - Traceback (most recent call last): File "H:\sd3\ComfyUI\execution.py", line 323, in execute output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "H:\sd3\ComfyUI\execution.py", line 198, in get_output_data return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "H:\sd3\ComfyUI\execution.py", line 169, in _map_node_over_list process_inputs(input_dict, i) File "H:\sd3\ComfyUI\execution.py", line 158, in process_inputs results.append(getattr(obj, func)(inputs)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "H:\sd3\ComfyUI\custom_nodes\ComfyUI_SLK_joy_caption_two\joy_caption_two_node.py", line 374, in generate text_model = joy_two_pipeline.llm.load_llm_model(joy_two_pipeline.model) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "H:\sd3\ComfyUI\custom_nodes\ComfyUI_SLK_joy_caption_two\joy_caption_two_node.py", line 172, in load_llm_model text_model = AutoModelForCausalLM.from_pretrained(text_model_path, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "H:\sd3\python_embeded\Lib\site-packages\transformers\models\auto\auto_factory.py", line 523, in from_pretrained config, kwargs = AutoConfig.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "H:\sd3\python_embeded\Lib\site-packages\transformers\models\auto\configuration_auto.py", line 958, in from_pretrained return config_class.from_dict(config_dict, unused_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "H:\sd3\python_embeded\Lib\site-packages\transformers\configuration_utils.py", line 768, in from_dict config = cls(**config_dict) ^^^^^^^^^^^^^^^^^^ File "H:\sd3\python_embeded\Lib\site-packages\transformers\models\llama\configuration_llama.py", line 161, in init self._rope_scaling_validation() File "H:\sd3\python_embeded\Lib\site-packages\transformers\models\llama\configuration_llama.py", line 182, in _rope_scaling_validation raise ValueError( ValueError: rope_scaling must be a dictionary with two fields, type and factor, got {'factor': 8.0, 'high_freq_factor': 4.0, 'low_freq_factor': 1.0, 'original_max_position_embeddings': 8192, 'rope_type': 'llama3'}

2024-10-12 20:21:45,931 - root - INFO - Prompt executed in 0.14 seconds 2024-10-12 20:23:21,945 - root - INFO - got prompt 2024-10-12 20:23:22,178 - root - ERROR - !!! Exception during processing !!! rope_scaling must be a dictionary with two fields, type and factor, got {'factor': 8.0, 'high_freq_factor': 4.0, 'low_freq_factor': 1.0, 'original_max_position_embeddings': 8192, 'rope_type': 'llama3'} 2024-10-12 20:23:22,179 - root - ERROR - Traceback (most recent call last): File "H:\sd3\ComfyUI\execution.py", line 323, in execute output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "H:\sd3\ComfyUI\execution.py", line 198, in get_output_data return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "H:\sd3\ComfyUI\execution.py", line 169, in _map_node_over_list process_inputs(input_dict, i) File "H:\sd3\ComfyUI\execution.py", line 158, in process_inputs results.append(getattr(obj, func)(inputs)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "H:\sd3\ComfyUI\custom_nodes\ComfyUI_SLK_joy_caption_two\joy_caption_two_node.py", line 374, in generate text_model = joy_two_pipeline.llm.load_llm_model(joy_two_pipeline.model) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "H:\sd3\ComfyUI\custom_nodes\ComfyUI_SLK_joy_caption_two\joy_caption_two_node.py", line 172, in load_llm_model text_model = AutoModelForCausalLM.from_pretrained(text_model_path, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "H:\sd3\python_embeded\Lib\site-packages\transformers\models\auto\auto_factory.py", line 523, in from_pretrained config, kwargs = AutoConfig.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "H:\sd3\python_embeded\Lib\site-packages\transformers\models\auto\configuration_auto.py", line 958, in from_pretrained return config_class.from_dict(config_dict, unused_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "H:\sd3\python_embeded\Lib\site-packages\transformers\configuration_utils.py", line 768, in from_dict config = cls(**config_dict) ^^^^^^^^^^^^^^^^^^ File "H:\sd3\python_embeded\Lib\site-packages\transformers\models\llama\configuration_llama.py", line 161, in init self._rope_scaling_validation() File "H:\sd3\python_embeded\Lib\site-packages\transformers\models\llama\configuration_llama.py", line 182, in _rope_scaling_validation raise ValueError( ValueError: rope_scaling must be a dictionary with two fields, type and factor, got {'factor': 8.0, 'high_freq_factor': 4.0, 'low_freq_factor': 1.0, 'original_max_position_embeddings': 8192, 'rope_type': 'llama3'}

2024-10-12 20:23:22,181 - root - INFO - Prompt executed in 0.21 seconds


## Attached Workflow
Please make sure that workflow does not contain any sensitive information such as API keys or passwords.
EvilBT commented 1 month ago

你这里感觉应该是模型下载不完整之类的问题，建议从国内镜像下载整个目录下的文件，再复制到指定目录下，我截图里有确定的位置的，你再试试

ohahaps2 commented 1 month ago

这个不用下吧，看你说用那个bnb-4bit小显存，我就没下载那个大的 “”“把整个文件夹内的内容复制到 models\LLM\Meta-Llama-3.1-8B-Instruct-bnb-4bit 下

国外:meta-llama/Meta-Llama-3.1-8B

国内:hf/meta-llama/Meta-Llama-3.1-8B”“”

biubiubiyabiu commented 1 month ago

输入类型（torch.cuda.HalfTensor）和权重类型（torch.FloatTensor）应该相同，大佬，这是啥问题？

ohahaps2 commented 1 month ago

全部重新下载了 还是一样的报第二个错。

biubiubiyabiu commented 1 month ago

ComfyUI Error Report

Error Details

ohahaps2 commented 1 month ago

我照网上的办法 修改了models\clip\siglip-so400m-patch14-384\config.json "rms_norm_eps": 1e-05, "rope_scaling": { "factor": 8.0, "type": "dynamic" 就能正常工作了 ，说是transformers新版本 rope_scaling 一些参数变了。

EvilBT commented 1 month ago

ComfyUI Error Report ComfyUI 错误报告

Error Details 错误详情

  • Node Type: Joy_caption_two节点类型: Joy_caption_two
  • Exception Type: RuntimeError异常类型：运行时错误
  • Exception Message: Input type (torch.cuda.HalfTensor) and weight type (torch.FloatTensor) should be the same异常消息：输入类型（torch.cuda.HalfTensor）和权重类型（torch.FloatTensor）应保持一致

Stack Trace 堆栈跟踪

![Uploading 微信截图_20241013013201.png…]()

你的截图看不到呀，这个应该是系统默认的创建张量类型不一致导致的，因为我的系统比较老旧，不能够充分测试，所以你再提供一下详细的截图给我看看？我也方便更新代码，我的系统是rtx2080 8G的显卡，默认是能正常使用的，但是我也遇到过类似的，结果现在还是在你这边的环境下遇到了

biubiubiyabiu commented 1 month ago

ComfyUI 错误报告 ComfyUI 错误报告

错误详情 错误详情

  • 节点类型: Joy_caption_two 节点类型: Joy_caption_two
  • Exception Type: RuntimeError异常类型：运行时错误
  • 异常消息：输入类型（torch.cuda.HalfTensor）和权重类型（torch.FloatTensor）应该相同异常消息：输入类型（torch.cuda.HalfTensor）和权重类型（torch.FloatTensor）应保持一致

堆栈跟踪

![Uploading 微信截图_20241013013201.png…]()

你的截图看不到呀，这个应该是系统默认的创建张量类型不一致导致的，因为我的系统比较旧，无法充分测试，所以你再提供一下详细的截图给我看看吗？我也方便更新代码，我的系统是rtx2080 8G的显卡，默认是能正常使用的，但是我也遇到过类似的，结果现在还是在你的环境下遇到了 got prompt Failed to validate prompt for output 27:

  • (prompt):
  • Required input is missing: text
  • ShowText|pysssss 27:
  • Required input is missing: text Output will be ignored Failed to validate prompt for output 10:
  • (prompt):
  • Required input is missing: image
  • WD14Tagger|pysssss 10:
  • Required input is missing: image Output will be ignored F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\ComfyUI\models\clip\siglip-so400m-patch14-384 Loading VLM's custom vision model Requested to load SiglipVisionTransformer Loading 1 new model loaded completely 0.0 1618.345947265625 True !!! Exception during processing !!! Input type (torch.cuda.HalfTensor) and weight type (torch.FloatTensor) should be the same Traceback (most recent call last): File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\ComfyUI\execution.py", line 323, in execute output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\ComfyUI\execution.py", line 198, in get_output_data return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\ComfyUI\execution.py", line 169, in _map_node_over_list process_inputs(input_dict, i) File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\ComfyUI\execution.py", line 158, in process_inputs results.append(getattr(obj, func)(inputs)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_SLK_joy_caption_two\joy_caption_two_node.py", line 331, in generate vision_outputs = joy_two_pipeline.clip_model.encode_image(pixel_values) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_SLK_joy_caption_two\joy_caption_two_node.py", line 65, in encode_image vision_outputs = self.model(pixel_values=pixel_values, output_hidden_states=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1562, in _call_impl return forward_call(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\models\siglip\modeling_siglip.py", line 1088, in forward hidden_states = self.embeddings(pixel_values, interpolate_pos_encoding=interpolate_pos_encoding) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1562, in _call_impl return forward_call(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\models\siglip\modeling_siglip.py", line 311, in forward patch_embeds = self.patch_embedding(pixel_values) # shape = [, width, grid, grid] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1553, in _wrapped_call_impl return self._call_impl(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1562, in _call_impl return forward_call(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\conv.py", line 458, in forward return self._conv_forward(input, self.weight, self.bias) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\conv.py", line 454, in _conv_forward return F.conv2d(input, weight, bias, self.stride, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: Input type (torch.cuda.HalfTensor) and weight type (torch.FloatTensor) should be the same

Prompt executed in 48.35 seconds Exception in thread Thread-8 (): Traceback (most recent call last): File "threading.py", line 1045, in _bootstrap_inner File "threading.py", line 982, in run File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Manager\glob\manager_server.py", line 1354, in threading.Thread(target=lambda: asyncio.run(default_cache_update())).start() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "asyncio\runners.py", line 190, in run File "asyncio\runners.py", line 118, in run File "asyncio\base_events.py", line 654, in run_until_complete File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Manager\glob\manager_server.py", line 1351, in default_cache_update await asyncio.gather(a, b, c, d, e) File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Manager\glob\manager_server.py", line 1338, in get_cache json_obj = await core.get_data(uri, True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Manager\glob\manager_core.py", line 621, in get_data json_text = await resp.text() ^^^^^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\python_embeded\Lib\site-packages\aiohttp\client_reqrep.py", line 1176, in text await self.read() File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\python_embeded\Lib\site-packages\aiohttp\client_reqrep.py", line 1134, in read self._body = await self.content.read() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\python_embeded\Lib\site-packages\aiohttp\streams.py", line 383, in read block = await self.readany() ^^^^^^^^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\python_embeded\Lib\site-packages\aiohttp\streams.py", line 405, in readany await self._wait("readany") File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\python_embeded\Lib\site-packages\aiohttp\streams.py", line 311, in _wait with self._timer: File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\python_embeded\Lib\site-packages\aiohttp\helpers.py", line 713, in exit raise asyncio.TimeoutError from None TimeoutError

biubiubiyabiu commented 1 month ago

我照网上的办法 修改了models\clip\siglip-so400m-patch14-384\config.json "rms_norm_eps": 1e-05, "rope_scaling": { "factor": 8.0, "type": "dynamic" 就能正常工作了 ，说是transformers新版本 rope_scaling 一些参数变了。

你是怎么修改的呀？

EvilBT commented 1 month ago

更新一下代码，我修复了部分BUG，指定了加载类型，或者能修复这个BUG

biubiubiyabiu commented 1 month ago

更新一下代码，我修复了部分BUG，指定了加载类型，或者能修复这个BUG

mixed dtype (CPU): expect parameter to have scalar type of Float
重新在manger里安装了。运行又报这个错， 怎么解呀？

biubiubiyabiu commented 1 month ago

更新一下代码，我修复了部分BUG，指定了加载类型，或者能修复这个BUG

Mixed dtype (CPU): Expect parameter to have scalar type of Float 重新在manger里安装了。运行又报这个错，怎么解呀？

got prompt Failed to validate prompt for output 27:

EvilBT commented 1 month ago

更新一下代码，我修复了部分BUG，指定了加载类型，或者能修复这个BUG

Mixed dtype (CPU): Expect parameter to have scalar type of Float 重新在manger里安装了。运行又报这个错，怎么解呀？

got prompt Failed to validate prompt for output 27:

  • (prompt):

    • Required input is missing: text

  • ShowText|pysssss 27:

    • Required input is missing: text Output will be ignored Failed to validate prompt for output 10:

  • (prompt):

    • Required input is missing: image

  • WD14Tagger|pysssss 10:

    • Required input is missing: image Output will be ignored F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\ComfyUI\models\clip\siglip-so400m-patch14-384 Loading VLM's custom vision model Requested to load SiglipVisionTransformer Loading 1 new model loaded completely 0.0 809.1729736328125 True !!! Exception during processing !!! mixed dtype (CPU): expect parameter to have scalar type of Float Traceback (most recent call last): File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\ComfyUI\execution.py", line 323, in execute output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\ComfyUI\execution.py", line 198, in get_output_data return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\ComfyUI\execution.py", line 169, in _map_node_over_list process_inputs(input_dict, i) File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\ComfyUI\execution.py", line 158, in process_inputs results.append(getattr(obj, func)(inputs)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_SLK_joy_caption_two\joy_caption_two_node.py", line 332, in generate vision_outputs = joy_two_pipeline.clip_model.encode_image(pixel_values) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_SLK_joy_caption_two\joy_caption_two_node.py", line 66, in encode_image vision_outputs = self.model(pixel_values=pixel_values, output_hidden_states=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1562, in _call_impl return forward_call(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\models\siglip\modeling_siglip.py", line 1090, in forward encoder_outputs = self.encoder( ^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1562, in _call_impl return forward_call(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\models\siglip\modeling_siglip.py", line 901, in forward layer_outputs = encoder_layer( ^^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1553, in _wrapped_call_impl return self._call_impl(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1562, in _call_impl return forward_call(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\models\siglip\modeling_siglip.py", line 642, in forward hidden_states = self.layer_norm1(hidden_states) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1562, in _call_impl return forward_call(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\normalization.py", line 202, in forward return F.layer_norm( ^^^^^^^^^^^^^ File "F:\AI\ComfyUI_windows_portable_nvidia (1)\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\functional.py", line 2576, in layer_norm return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: mixed dtype (CPU): expect parameter to have scalar type of Float

贴一下你的transformers版本？

ohahaps2 commented 1 month ago

我照网上的办法 修改了models\clip\siglip-so400m-patch14-384\config.json "rms_norm_eps": 1e-05, "rope_scaling": { "factor": 8.0, "type": "dynamic" 就能正常工作了 ，说是transformers新版本 rope_scaling 一些参数变了。

你是怎么修改的呀？

右键编辑 config.json，最最最下面 找到 rope_scaling之后那些相似字段，变成我贴的那一段，就好了。

VAVAVAAA commented 1 month ago

我照网上的办法 修改了models\clip\siglip-so400m-patch14-384\config.json "rms_norm_eps": 1e-05, "rope_scaling": { "factor": 8.0, "type": "dynamic" 就能正常工作了 ，说是transformers新版本 rope_scaling 一些参数变了。

你是怎么修改的呀？

右键编辑 config.json，最最最下面 找到 rope_scaling之后那些相似字段，变成我贴的那一段，就好了。

兄弟，我没找到你说的这部分代码啊，siglip-so400m-patch14-384抱脸官方仓库里也没有，我这也出现报错了

EvilBT commented 1 month ago

你们的 transformers 版本是多少？我在云服务器遇到这个 rope_scaling must be a dictionary with two fields, type and factor, got {'factor': 8.0, 'high_freq_factor': 4.0, 'low_freq_factor': 1.0, 'original_max_position_embeddings': 8192, 'rope_type': 'llama3'} 这个错误时，更新 transformers到最新的就可以了

VAVAVAAA commented 1 month ago

你们的 transformers 版本是多少？我在云服务器遇到这个 rope_scaling must be a dictionary with two fields, type and factor, got {'factor': 8.0, 'high_freq_factor': 4.0, 'low_freq_factor': 1.0, 'original_max_position_embeddings': 8192, 'rope_type': 'llama3'} 这个错误时，更新 transformers到最新的就可以了

transformers==4.44.2

spetangle commented 1 month ago

同样的错误：

ComfyUI Error Report

Error Details

  • Node Type: Joy_caption_two_advanced
  • Exception Type: RuntimeError

  • Exception Message: Input type (torch.FloatTensor) and weight type (torch.HalfTensor) should be the same or input should be a MKLDNN tensor and weight is a dense tensor

    Stack Trace

    
File "E:\AIGC\ComfyUI\execution.py", line 323, in execute
output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "E:\AIGC\ComfyUI\execution.py", line 198, in get_output_data
return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "E:\AIGC\ComfyUI\execution.py", line 169, in _map_node_over_list
process_inputs(input_dict, i)

File "E:\AIGC\ComfyUI\execution.py", line 158, in process_inputs
results.append(getattr(obj, func)(**inputs))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "E:\AIGC\ComfyUI\custom_nodes\ComfyUI_SLK_joy_caption_two\joy_caption_two_node.py", line 496, in generate
vision_outputs = joy_two_pipeline.clip_model.encode_image(pixel_values)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "E:\AIGC\ComfyUI\custom_nodes\ComfyUI_SLK_joy_caption_two\joy_caption_two_node.py", line 67, in encode_image
vision_outputs = self.model(pixel_values=pixel_values, output_hidden_states=True)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "E:\AIGC\ComfyUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "E:\AIGC\ComfyUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "E:\AIGC\ComfyUI\venv\Lib\site-packages\transformers\models\siglip\modeling_siglip.py", line 1087, in forward
hidden_states = self.embeddings(pixel_values, interpolate_pos_encoding=interpolate_pos_encoding)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "E:\AIGC\ComfyUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "E:\AIGC\ComfyUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "E:\AIGC\ComfyUI\venv\Lib\site-packages\transformers\models\siglip\modeling_siglip.py", line 310, in forward
patch_embeds = self.patch_embedding(pixel_values)  # shape = [*, width, grid, grid]
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "E:\AIGC\ComfyUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "E:\AIGC\ComfyUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "E:\AIGC\ComfyUI\venv\Lib\site-packages\torch\nn\modules\conv.py", line 460, in forward
return self._conv_forward(input, self.weight, self.bias)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "E:\AIGC\ComfyUI\venv\Lib\site-packages\torch\nn\modules\conv.py", line 456, in _conv_forward
return F.conv2d(input, weight, bias, self.stride,
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

## System Information
- **ComfyUI Version:** v0.2.3
- **Arguments:** E:\AIGC\ComfyUI\main.py --auto-launch --preview-method auto --disable-cuda-malloc --lowvram --preview-method auto --use-split-cross-attention
- **OS:** nt
- **Python Version:** 3.11.7 (tags/v3.11.7:fa7a6f2, Dec  4 2023, 19:24:49) [MSC v.1937 64 bit (AMD64)]
- **Embedded Python:** false
- **PyTorch Version:** 2.2.0+cu118
## Devices

已经尝试过：

  1. 重新下载siglip-so400m-patch14-384；
  2. 尝试修改siglip-so400m-patch14-384的config.json；
  3. 重启安装依赖文件；
  4. transformers==4.44.2，降级至4.37.0； 以上4个办法都不成功。 image
EvilBT commented 1 month ago

你们先帮我测试一下，我先不改代码，你们在 joy_caption_two_node.py 的第132行左右的这里添加这两行代码：

        image_adapter.load_state_dict(torch.load(adapter_path, map_location=self.offload_device, weights_only=True))
        image_adapter.eval()

中间添加这两行，指定加载格式：

        image_adapter.load_state_dict(torch.load(adapter_path, map_location=self.offload_device, weights_only=True))
        # 新增加的两行
        img_dtype = text_encoder_dtype()
        image_adapter = image_adapter.to(img_dtype)
        # 就是指定转换到统一格式
        image_adapter.eval()

然后重启尝试一下，看看还会不会出现精度不一致的问题：

Exception Message: Input type (torch.cuda.HalfTensor) and weight type (torch.FloatTensor) should be the same

如果可以的话，你们反馈一下，我再更新代码

design2024-cc commented 3 weeks ago

我照网上的办法 修改了models\clip\siglip-so400m-patch14-384\config.json "rms_norm_eps": 1e-05, "rope_scaling": { "factor": 8.0, "type": "dynamic" 就能正常工作了 ，说是transformers新版本 rope_scaling 一些参数变了。

你是怎么修改的呀？

右键编辑 config.json，最最最下面 找到 rope_scaling之后那些相似字段，变成我贴的那一段，就好了。

兄弟，我没找到你说的这部分代码啊，siglip-so400m-patch14-384抱脸官方仓库里也没有，我这也出现报错了

兄弟，在这里面models\LLM\Meta-Llama-3.1-8B-bnb-4bit，他应该搞错文件夹了，我在这里找到了，跟他的一样代码，但按照改了也不行

design2024-cc commented 3 weeks ago

你们先帮我测试一下，我先不改代码，你们在 joy_caption_two_node.py 的第132行左右的这里添加这两行代码：

        image_adapter.load_state_dict(torch.load(adapter_path, map_location=self.offload_device, weights_only=True))
        image_adapter.eval()

中间添加这两行，指定加载格式：

        image_adapter.load_state_dict(torch.load(adapter_path, map_location=self.offload_device, weights_only=True))
        # 新增加的两行
        img_dtype = text_encoder_dtype()
        image_adapter = image_adapter.to(img_dtype)
        # 就是指定转换到统一格式
        image_adapter.eval()

然后重启尝试一下，看看还会不会出现精度不一致的问题：

Exception Message: Input type (torch.cuda.HalfTensor) and weight type (torch.FloatTensor) should be the same

如果可以的话，你们反馈一下，我再更新代码

UP加油啊，对你的期望很大啊！

EvilBT commented 3 weeks ago

你们先帮我测试一下，我先不改代码，你们在 joy_caption_two_node.py 的第132行左右的这里添加这两行代码：

        image_adapter.load_state_dict(torch.load(adapter_path, map_location=self.offload_device, weights_only=True))
        image_adapter.eval()

中间添加这两行，指定加载格式：

        image_adapter.load_state_dict(torch.load(adapter_path, map_location=self.offload_device, weights_only=True))
        # 新增加的两行
        img_dtype = text_encoder_dtype()
        image_adapter = image_adapter.to(img_dtype)
        # 就是指定转换到统一格式
        image_adapter.eval()

然后重启尝试一下，看看还会不会出现精度不一致的问题：

Exception Message: Input type (torch.cuda.HalfTensor) and weight type (torch.FloatTensor) should be the same

如果可以的话，你们反馈一下，我再更新代码

UP加油啊，对你的期望很大啊！

主要是我复现不了，就挺尴尬，有时候可能就是安装依赖的问题，有时是系统环境不一样的问题，那你也帮我尝试一下，先把代码更新到最新，然后在 joy_caption_two_node.py 的第64行那一个方法添加下面一行：

    def encode_image(self, pixel_values):
        #print(f"{id(self)}之前 in JoyClipVisionModel: {next(self.model.parameters()).device}")  # 打印模型参数的设备
        load_models_gpu([self.patcher], force_full_load=True, force_patch_weights=True)
        #print(f"之后 in JoyClipVisionModel: {next(self.model.parameters()).device}")  # 打印模型参数的设备
        # 将图像数据转换为模型的 dtype 添加下面这行
        pixel_values = pixel_values.to(dtype=self.type)
        vision_outputs = self.model(pixel_values=pixel_values, output_hidden_states=True)
        return vision_outputs

保存重启ComfyUI试一下

spetangle commented 3 weeks ago

ComfyUI Error Report

Error Details


# 新增加的两行
        img_dtype = text_encoder_dtype()
        image_adapter = image_adapter.to(img_dtype)

# 将图像数据转换为模型的 dtype 添加下面这行
        pixel_values = pixel_values.to(dtype=self.type)

上面两条已添加，这是报错信息
design2024-cc commented 3 weeks ago

各位兄弟，我搞定了，先说一下我的这个过程，仅供参考。我用的是秋叶最新版，下载完秋叶包后，所有东西都不更新，直接重新安装该插件，运行comfyui后自动安装依赖，可能提出现红色错误提示，先不管。下一步一定要仔细看github上的说明，尤其是文件名和文件夹的名称，哪怕是下划线，一定要跟说明的一模一样，如果是自行下载的模型，所有文件夹和里面的文件一定要跟Hugging Face命名一样，仔细对照，一步错即全错。

GaiNianK commented 2 weeks ago

我的解决了，直接卸载插件重新安装就好了，模型不用重新下载，用回之前下载的模型文件就好了