RuntimeError: MPS: Unsupported Border padding mode

[pimboreel]

After trying different dimensions on a Mac M2 Max 64GB, only 256 x 256 on a batch size of 16 would provide a stable output. besides that, RIFE VFI gives me an error, preventing me from upscaling the resolution and frame rate:

pimboreel@bass-mbp-3 ComfyUI % python3 main.py             
** ComfyUI start up time: 2023-11-25 17:11:44.409575

Prestartup times for custom nodes:
   0.0 seconds: /Users/pimboreel/Dropbox/AI/ComfyUI/custom_nodes/ComfyUI-Manager

Total VRAM 65536 MB, total RAM 65536 MB
Set vram state to: SHARED
Device: mps
VAE dtype: torch.float32
Using sub quadratic optimization for cross attention, if you have memory or speed issues try using: --use-split-cross-attention
### Loading: ComfyUI-Manager (V1.3)
### ComfyUI Revision: 1730 [eff24ea6] | Released on '2023-11-24'

Import times for custom nodes:
   0.0 seconds: /Users/pimboreel/Dropbox/AI/ComfyUI/custom_nodes/ComfyUI-Frame-Interpolation
   0.0 seconds: /Users/pimboreel/Dropbox/AI/ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved
   0.0 seconds: /Users/pimboreel/Dropbox/AI/ComfyUI/custom_nodes/ComfyUI-VideoHelperSuite
   0.1 seconds: /Users/pimboreel/Dropbox/AI/ComfyUI/custom_nodes/ComfyUI-Manager

Starting server

To see the GUI go to: http://127.0.0.1:8188
FETCH DATA from: /Users/pimboreel/Dropbox/AI/ComfyUI/custom_nodes/ComfyUI-Manager/extension-node-map.json
got prompt
model_type EPS
adm 0
Using split attention in VAE
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
Using split attention in VAE
missing {'cond_stage_model.clip_l.logit_scale', 'cond_stage_model.clip_l.text_projection'}
left over keys: dict_keys(['cond_stage_model.clip_l.transformer.text_model.embeddings.position_ids'])
[AnimateDiffEvo] - INFO - Loading motion LoRA v2_lora_PanRight.ckpt
Global Step: 10000
[AnimateDiffEvo] - INFO - Loading motion LoRA v2_lora_ZoomIn.ckpt
Global Step: 10000
[AnimateDiffEvo] - INFO - Loading motion module mm_sd_v14.ckpt
[AnimateDiffEvo] - INFO - Applying a v2 LoRA (v2_lora_ZoomIn.ckpt) to a v1 motion model.
[AnimateDiffEvo] - INFO - Applying a v2 LoRA (v2_lora_PanRight.ckpt) to a v1 motion model.
Requested to load SD1ClipModel
Loading 1 new model
[AnimateDiffEvo] - INFO - Regular AnimateDiff activated - latents passed in (16) less or equal to context_length None.
[AnimateDiffEvo] - INFO - Injecting motion module mm_sd_v14.ckpt version v1.
Requested to load BaseModel
Loading 1 new model
 92%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊                    | 23/25 [01:21<00:07,  3.55s/it]/opt/homebrew/lib/python3.11/site-packages/torchsde/_brownian/brownian_interval.py:599: UserWarning: Should have ta>=t0 but got ta=0.02916753478348255 and t0=0.029168.
  warnings.warn(f"Should have ta>=t0 but got ta={ta} and t0={self._start}.")
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 25/25 [01:26<00:00,  3.45s/it]
[AnimateDiffEvo] - INFO - Ejecting motion module mm_sd_v14.ckpt version v1.
[AnimateDiffEvo] - INFO - Cleaning motion module from unet.
[AnimateDiffEvo] - INFO - Removing motion module mm_sd_v14.ckpt from cache
Using split attention in VAE
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
Using split attention in VAE
Leftover VAE keys ['model_ema.decay', 'model_ema.num_updates']
ERROR:root:!!! Exception during processing !!!
ERROR:root:Traceback (most recent call last):
  File "/Users/pimboreel/Dropbox/AI/ComfyUI/execution.py", line 153, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pimboreel/Dropbox/AI/ComfyUI/execution.py", line 83, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pimboreel/Dropbox/AI/ComfyUI/execution.py", line 76, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pimboreel/Dropbox/AI/ComfyUI/custom_nodes/ComfyUI-Frame-Interpolation/vfi_models/rife/__init__.py", line 92, in vfi
    generic_frame_loop(frames, clear_cache_after_n_frames, multiplier, return_middle_frame, *args,
  File "/Users/pimboreel/Dropbox/AI/ComfyUI/custom_nodes/ComfyUI-Frame-Interpolation/vfi_utils.py", line 165, in generic_frame_loop
    middle_frame = return_middle_frame_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pimboreel/Dropbox/AI/ComfyUI/custom_nodes/ComfyUI-Frame-Interpolation/vfi_models/rife/__init__.py", line 86, in return_middle_frame
    return model(frame_0, frame_1, timestep, scale_list, in_fast_mode, in_ensemble)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pimboreel/Dropbox/AI/ComfyUI/custom_nodes/ComfyUI-Frame-Interpolation/vfi_models/rife/rife_arch.py", line 531, in forward
    warped_img0 = warp(img0, flow[:, :2])
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pimboreel/Dropbox/AI/ComfyUI/custom_nodes/ComfyUI-Frame-Interpolation/vfi_models/rife/rife_arch.py", line 59, in warp
    return torch.nn.functional.grid_sample(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/torch/nn/functional.py", line 4304, in grid_sample
    return torch.grid_sampler(input, grid, mode_enum, padding_mode_enum, align_corners)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: MPS: Unsupported Border padding mode

I have set the unlimited_area_hack to true according to this thread.

[Kosinkadink]

Looks like MPS is not compatible with something RIFE is doing. Do you still get stuff out if you don't use RIFE?

[pimboreel]

Yes, there is still an H.264 video file being produced. I highlighted the RIFE in the screenshot below for you to see the signal flow. I am running Ventura 13.2.1.

The VAE (in the upper part of the screenshot) decodes the KSampler, spits out the video and then crashes when trying to pass the VAE output to the first RIFE. Disabling the first RIFE and only using the second one gives the same error.

[Kosinkadink]

Gotcha, looks like it's an issue with MDS not working with the code in RIFE. I'd recommend opening up an issue with the repo for that node, as I'm not gonna be able to help ya with a RIFE-related issue.

[pimboreel]

Will do that, thanks!

[meisa233]

modify the last line of loss/warplayer.py

torch.nn.functional.grid_sample(input=tenInput, grid=g, mode='bilinear', padding_mode='border',
                                               align_corners=True)

to

torch.nn.functional.grid_sample(input=tenInput, grid=g, mode='bilinear', padding_mode='reflection',
                                               align_corners=True)

[gmmarc]

@meisa233 solution worked for me on MacOS. Thank you!

[meqaniqal]

I have comfyui installed on my m1 mac in pinokio with vfi nodes. I also installed taichi somehow a couple weeks ago. Until I read @maisa233's suggestion, I had given up, and although I couldn't find a warplayer.py in my comfyui installation (only in automatic1111), it turned out that editing the file rife.arc in /Users//pinokio/api/comfy.git/app/custom_nodes/ComfyUI-Frame-Interpolation/vfi_models/rife/rife_arch.py to switch padding_mode to "reflection" made rife work, finally. I tried similar modifications to the arch files for the other vfi models , but the only other vfi nodes besides rife that worked for me after this were ifunet and ifrnet.