TypeError: Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead.

[jacobb0631]

Trying to run tooncrafter through the node on M1 Max MBP, using example workflow in the examples folder, when using auto dtype it runs the model in fp16 and the vae in fp32 which then causes this error: RuntimeError: Input type (c10::Half) and bias type (float) should be the same, running it with fp32 set to both gave me a CUDA error but I simply swapped the "comfyui.git/app/custom_nodes/ComfyUI-DynamiCrafterWrapper/lvdm/models/samplers/ddim.py", line 21, in register_buffer attr = attr.to(torch.device("cuda"))" from cuda to mps, doing this I ran into another error: TypeError: Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead.

Using fp16 for vae dtype causes the following error (all other models are also set to fp16 here): comfyui.git/app/env/lib/python3.10/site-packages/torch/nn/functional.py", line 4089, in interpolate return torch._C._nn._upsample_bicubic2d_aa(input, output_size, align_corners, scale_factors) RuntimeError: "compute_index_ranges_weights" not implemented for 'Half'

I'm fine with any method of running it but just can't seem to do so currently, any idea on how to fix it?

[jacobb0631]

I managed to get past this error by changing the register buffer like so: def register_buffer(self, name, attr): if type(attr) == torch.Tensor: if attr.device != torch.device("cuda"): if torch.cuda.is_available(): attr = attr.to(torch.device("cuda")) elif hasattr(torch.backends, "mps") and torch.backends.mps.is_available(): attr = attr.to(torch.device("mps"), torch.float32) else: attr = attr.to(torch.device("cpu"))

Now I'm running into the following error: AttributeError: 'DDIMSampler' object has no attribute 'alphas_cumprod_prev' Here's the full log: [Start proxy] Local Sharing http://127.0.0.1:8188 Proxy Started {"target":"http://127.0.0.1:8188","proxy":"http://192.168.1.15:42421"} got prompt Prompt executor has been patched by Job Iterator! [rgthree] Using rgthree's optimized recursive execution. [rgthree] First run patching recursive_output_delete_if_changed and recursive_will_execute. [rgthree] Note: If execution seems broken due to forward ComfyUI changes, you can disable the optimization from rgthree settings in ComfyUI. Loading model from: /Users/jacobbrown/pinokio/api/comfyui.git/app/models/checkpoints/dynamicrafter/tooncrafter_512_interp-fp16.safetensors no module 'xformers'. Processing without... no module 'xformers'. Processing without... LatentVisualDiffusion: Running in v-prediction mode AE working on z of shape (1, 4, 32, 32) = 4096 dimensions. Working with z of shape (1, 4, 32, 32) = 4096 dimensions. vanilla making attention of type 'vanilla' with 512 in_channels memory-efficient-cross-attn-fusion making attention of type 'memory-efficient-cross-attn-fusion' with 512 in_channels memory-efficient-cross-attn-fusion making attention of type 'memory-efficient-cross-attn-fusion' with 512 in_channels

model checkpoint loaded. Model using dtype: torch.float32 Loading model from: /Users/jacobbrown/pinokio/api/comfyui.git/app/models/clip_vision/CLIP-ViT-H-14-laion2B-s32B-b79K.safetensors clip missing: ['text_model.encoder.layers.23.layer_norm1.weight', 'text_model.encoder.layers.23.layer_norm1.bias', 'text_model.encoder.layers.23.self_attn.q_proj.weight', 'text_model.encoder.layers.23.self_attn.q_proj.bias', 'text_model.encoder.layers.23.self_attn.k_proj.weight', 'text_model.encoder.layers.23.self_attn.k_proj.bias', 'text_model.encoder.layers.23.self_attn.v_proj.weight', 'text_model.encoder.layers.23.self_attn.v_proj.bias', 'text_model.encoder.layers.23.self_attn.out_proj.weight', 'text_model.encoder.layers.23.self_attn.out_proj.bias', 'text_model.encoder.layers.23.layer_norm2.weight', 'text_model.encoder.layers.23.layer_norm2.bias', 'text_model.encoder.layers.23.mlp.fc1.weight', 'text_model.encoder.layers.23.mlp.fc1.bias', 'text_model.encoder.layers.23.mlp.fc2.weight', 'text_model.encoder.layers.23.mlp.fc2.bias', 'text_projection.weight'] Loading model from: /Users/jacobbrown/pinokio/api/comfyui.git/app/models/clip/stable-diffusion-2-1-clip.safetensors Requested to load SD2ClipModel Loading 1 new model VAE using dtype: torch.float32 Requested to load CLIPVisionModelProjection Loading 1 new model !!! Exception during processing!!! 'DDIMSampler' object has no attribute 'alphas_cumprod_prev' Traceback (most recent call last): File "/Users/jacobbrown/pinokio/api/comfyui.git/app/execution.py", line 151, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) File "/Users/jacobbrown/pinokio/api/comfyui.git/app/execution.py", line 81, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) File "/Users/jacobbrown/pinokio/api/comfyui.git/app/execution.py", line 74, in map_node_over_list results.append(getattr(obj, func)(*slice_dict(input_data_all, i))) File "/Users/jacobbrown/pinokio/api/comfyui.git/app/customnodes/ComfyUI-DynamiCrafterWrapper/nodes.py", line 640, in process samples, = ddim_sampler.sample(S=steps, File "/Users/jacobbrown/pinokio/api/comfyui.git/app/env/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(args, *kwargs) File "/Users/jacobbrown/pinokio/api/comfyui.git/app/custom_nodes/ComfyUI-DynamiCrafterWrapper/lvdm/models/samplers/ddim.py", line 107, in sample self.make_schedule(ddim_num_steps=S, ddim_discretize=timestep_spacing, ddim_eta=eta, verbose=schedule_verbose) File "/Users/jacobbrown/pinokio/api/comfyui.git/app/custom_nodes/ComfyUI-DynamiCrafterWrapper/lvdm/models/samplers/ddim.py", line 59, in make_schedule (1 - self.alphas_cumprod_prev) / (1 - self.alphas_cumprod) ( AttributeError: 'DDIMSampler' object has no attribute 'alphas_cumprod_prev'

Prompt executed in 17.23 seconds

Looked up the error and don't see anything relevant, any help is appreciated :)

[kijai]

That last error has to be caused by your edit, though quickly looking it should work. I pushed some changes now that may or may not help, I don't have mps device to test anything with so it's pretty tedious to troubleshoot :/

[jacobb0631]

Running the newest version gave this error (running the example with all set to fp32): TypeError: Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead. fp32 on all is still the only working dtype (which is expected because of "compute_index_ranges_weights" not implemented for 'Half') Just changing the line to: attr = attr.to(torch.device(device), torch.float32) made it so the steps actually worked (farthest it's gone so far) but then ran into this error: Assertion failed: (0 <= mpsAxis && mpsAxis < 4 && "Runtime canonicalization must simplify reduction axes to minor 4 dimensions."), function getKernelAxes, file GPUReductionOps.mm, line 31. Abort trap: 6 (env) (base) <> /Users/jacobbrown/pinokio/bin/miniconda/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d '

Unsure what the problem is, it's not running out of memory (memory pressure is entirely green and it uses a max of around 30gb)

Here's the full log: Start proxy] Local Sharing http://127.0.0.1:8188 Proxy Started {"target":"http://127.0.0.1:8188","proxy":"http://192.168.1.15:42421"} FETCH DATA from: /Users/jacobbrown/pinokio/api/comfyui.git/app/custom_nodes/ComfyUI-Manager/extension-node-map.json [DONE] got prompt Prompt executor has been patched by Job Iterator! [rgthree] Using rgthree's optimized recursive execution. [rgthree] First run patching recursive_output_delete_if_changed and recursive_will_execute. [rgthree] Note: If execution seems broken due to forward ComfyUI changes, you can disable the optimization from rgthree settings in ComfyUI. Loading model from: /Users/jacobbrown/pinokio/api/comfyui.git/app/models/checkpoints/dynamicrafter/tooncrafter_512_interp-fp16.safetensors no module 'xformers'. Processing without... no module 'xformers'. Processing without... LatentVisualDiffusion: Running in v-prediction mode AE working on z of shape (1, 4, 32, 32) = 4096 dimensions. Working with z of shape (1, 4, 32, 32) = 4096 dimensions. vanilla making attention of type 'vanilla' with 512 in_channels memory-efficient-cross-attn-fusion making attention of type 'memory-efficient-cross-attn-fusion' with 512 in_channels memory-efficient-cross-attn-fusion making attention of type 'memory-efficient-cross-attn-fusion' with 512 in_channels

model checkpoint loaded. Model using dtype: torch.float32 Loading model from: /Users/jacobbrown/pinokio/api/comfyui.git/app/models/clip_vision/CLIP-ViT-H-14-laion2B-s32B-b79K.safetensors clip missing: ['text_model.encoder.layers.23.layer_norm1.weight', 'text_model.encoder.layers.23.layer_norm1.bias', 'text_model.encoder.layers.23.self_attn.q_proj.weight', 'text_model.encoder.layers.23.self_attn.q_proj.bias', 'text_model.encoder.layers.23.self_attn.k_proj.weight', 'text_model.encoder.layers.23.self_attn.k_proj.bias', 'text_model.encoder.layers.23.self_attn.v_proj.weight', 'text_model.encoder.layers.23.self_attn.v_proj.bias', 'text_model.encoder.layers.23.self_attn.out_proj.weight', 'text_model.encoder.layers.23.self_attn.out_proj.bias', 'text_model.encoder.layers.23.layer_norm2.weight', 'text_model.encoder.layers.23.layer_norm2.bias', 'text_model.encoder.layers.23.mlp.fc1.weight', 'text_model.encoder.layers.23.mlp.fc1.bias', 'text_model.encoder.layers.23.mlp.fc2.weight', 'text_model.encoder.layers.23.mlp.fc2.bias', 'text_projection.weight'] Loading model from: /Users/jacobbrown/pinokio/api/comfyui.git/app/models/clip/stable-diffusion-2-1-clip.safetensors Requested to load SD2ClipModel Loading 1 new model VAE using dtype: torch.float32 Requested to load CLIPVisionModelProjection Loading 1 new model DDIM Sampler: 100%|███████████████████████████████████████████████████████████████████████████████| 20/20 [03:05<00:00, 9.26s/it] Assertion failed: (0 <= mpsAxis && mpsAxis < 4 && "Runtime canonicalization must simplify reduction axes to minor 4 dimensions."), function getKernelAxes, file GPUReductionOps.mm, line 31. Abort trap: 6 (env) (base) <> /Users/jacobbrown/pinokio/bin/miniconda/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d '

Disconnected...

The real reason this reply took so long is I decided to try running in CPU only incase it's helpful to you at all: [Start proxy] Local Sharing http://127.0.0.1:8188 Proxy Started {"target":"http://127.0.0.1:8188","proxy":"http://192.168.1.15:42421"} got prompt Prompt executor has been patched by Job Iterator! [rgthree] Using rgthree's optimized recursive execution. [rgthree] First run patching recursive_output_delete_if_changed and recursive_will_execute. [rgthree] Note: If execution seems broken due to forward ComfyUI changes, you can disable the optimization from rgthree settings in ComfyUI. Loading model from: /Users/jacobbrown/pinokio/api/comfyui.git/app/models/checkpoints/dynamicrafter/tooncrafter_512_interp-fp16.safetensors no module 'xformers'. Processing without... no module 'xformers'. Processing without... LatentVisualDiffusion: Running in v-prediction mode AE working on z of shape (1, 4, 32, 32) = 4096 dimensions. Working with z of shape (1, 4, 32, 32) = 4096 dimensions. vanilla making attention of type 'vanilla' with 512 in_channels memory-efficient-cross-attn-fusion making attention of type 'memory-efficient-cross-attn-fusion' with 512 in_channels memory-efficient-cross-attn-fusion making attention of type 'memory-efficient-cross-attn-fusion' with 512 in_channels

model checkpoint loaded. Model using dtype: torch.float32 Loading model from: /Users/jacobbrown/pinokio/api/comfyui.git/app/models/clip_vision/CLIP-ViT-H-14-laion2B-s32B-b79K.safetensors clip missing: ['text_model.encoder.layers.23.layer_norm1.weight', 'text_model.encoder.layers.23.layer_norm1.bias', 'text_model.encoder.layers.23.self_attn.q_proj.weight', 'text_model.encoder.layers.23.self_attn.q_proj.bias', 'text_model.encoder.layers.23.self_attn.k_proj.weight', 'text_model.encoder.layers.23.self_attn.k_proj.bias', 'text_model.encoder.layers.23.self_attn.v_proj.weight', 'text_model.encoder.layers.23.self_attn.v_proj.bias', 'text_model.encoder.layers.23.self_attn.out_proj.weight', 'text_model.encoder.layers.23.self_attn.out_proj.bias', 'text_model.encoder.layers.23.layer_norm2.weight', 'text_model.encoder.layers.23.layer_norm2.bias', 'text_model.encoder.layers.23.mlp.fc1.weight', 'text_model.encoder.layers.23.mlp.fc1.bias', 'text_model.encoder.layers.23.mlp.fc2.weight', 'text_model.encoder.layers.23.mlp.fc2.bias', 'text_projection.weight'] Loading model from: /Users/jacobbrown/pinokio/api/comfyui.git/app/models/clip/stable-diffusion-2-1-clip.safetensors Requested to load SD2ClipModel Loading 1 new model VAE using dtype: torch.float32 Requested to load CLIPVisionModelProjection Loading 1 new model DDIM Sampler: 100%|████████████████████████████████████████████████████| 20/20 [18:35<00:00, 55.77s/it] VAE using dtype: torch.float32 !!! Exception during processing!!! XFormers not available, it is required for ToonCrafter decoder. Alternatively you can use a standard VAE Decode -node instead, but this has a negative effect on the image quality though. Traceback (most recent call last): File "/Users/jacobbrown/pinokio/api/comfyui.git/app/execution.py", line 151, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) File "/Users/jacobbrown/pinokio/api/comfyui.git/app/execution.py", line 81, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) File "/Users/jacobbrown/pinokio/api/comfyui.git/app/execution.py", line 74, in map_node_over_list results.append(getattr(obj, func)(**slice_dict(input_data_all, i))) File "/Users/jacobbrown/pinokio/api/comfyui.git/app/custom_nodes/ComfyUI-DynamiCrafterWrapper/nodes.py", line 740, in process raise Exception("XFormers not available, it is required for ToonCrafter decoder. Alternatively you can use a standard VAE Decode -node instead, but this has a negative effect on the image quality though.") Exception: XFormers not available, it is required for ToonCrafter decoder. Alternatively you can use a standard VAE Decode -node instead, but this has a negative effect on the image quality though.

Prompt executed in 1156.30 seconds

CPU only with low vram mode in all fp32 works perfectly as intended (video comes out fine) :) If you can get MPS working that'd be nice but at least cpu works now on my mbp which is cool, thanks for the pushes and if you want me to test anything I will gladly.

[kijai]

Does the error happen still in the main node? It looks like it samples all steps just fine at least.

[jacobb0631]

Does the error happen still in the main node? It looks like it samples all steps just fine at least.

Yes it’s the main block that is errorring in the MPS run, it is the decoder block that is erroring in the cpu run with example notebook (normal not low vram) but that’s because I don’t have xformers installed

Also like to add that setting PYTORCH_ENABLE_MPS_FALLBACK to 1 and using PyTorch nightly didn't change anything (still getting the same error), it'd be neat if the sampling steps ran on the gpu then whatever is goofing it just fell back to the cpu but I cannot seem to get that to happen