Closed jwooldridge234 closed 1 month ago
Yeah my bad, I forgot to actually push the update that allows using those. Can you try now?
Just did. Looks like it's reliant on CUDA for the node (I get this error):
Error occurred when executing LoadChatGLM3:
Torch not compiled with CUDA enabled
File "/Users/jackwooldridge/ComfyUI/execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jackwooldridge/ComfyUI/execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jackwooldridge/ComfyUI/execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jackwooldridge/ComfyUI/custom_nodes/ComfyUI-KwaiKolorsWrapper/nodes.py", line 124, in loadmodel
text_encoder.quantize(8)
File "/Users/jackwooldridge/ComfyUI/custom_nodes/ComfyUI-KwaiKolorsWrapper/kolors/models/modeling_chatglm.py", line 852, in quantize
quantize(self.encoder, weight_bit_width)
File "/Users/jackwooldridge/ComfyUI/custom_nodes/ComfyUI-KwaiKolorsWrapper/kolors/models/quantization.py", line 157, in quantize
weight=layer.self_attention.query_key_value.weight.to(torch.cuda.current_device()),
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jackwooldridge/.pyenv/versions/3.12.4/lib/python3.12/site-packages/torch/cuda/__init__.py", line 778, in current_device
_lazy_init()
File "/Users/jackwooldridge/.pyenv/versions/3.12.4/lib/python3.12/site-packages/torch/cuda/__init__.py", line 284, in _lazy_init
raise AssertionError("Torch not compiled with CUDA enabled")
MPS has limited support for quantization (though Huggingface's new Quanto package works pretty well) so I don't know if this is easily fixable on your end. I can look into options for Mac and maybe do a pull request.
Yeah looks like quantization doesn't support MPS.
Hi there!
Tried running this on MacOS, using the Load ChatGLM3 Model node and the 8bit and 4bit safetensors, and I get some interesting errors:
4bit model in fp16, q8, q4 (makes no difference):
8bit model in any quant:
I'm on the latest version of ComfyUI and I made sure to install requirements.txt after pulling your latest version. Breaks just using python main.py and no parameters.