Is there existing or planned Windows support?

[Gyramuur]

I've got the model.ckpt and all the requirements installed through miniconda, but I don't think I can run .sh files.

[l33tx0]

after activating the conda env , cd to the root directory of cloned project Try with the following command that resemble the sh file

python scripts\evaluation\inference.py --seed 123 --mode "base" --ckpt_path checkpoints\base_1024_v1\model.ckpt --config configs\inference_t2v_1024_v1.0.yaml --savedir results\base_1024_test --n_samples 1 --bs 1 --height 576 --width 1024 --unconditional_guidance_scale 12.0 --ddim_steps 50 --ddim_eta 1.0 --prompt_file prompts\test_prompts.txt --fps 28

[Gyramuur]

after activating the conda env , cd to the root directory of cloned project Try with the following command that resemble the sh file

python scripts\evaluation\inference.py --seed 123 --mode "base" --ckpt_path checkpoints\base_1024_v1\model.ckpt --config configs\inference_t2v_1024_v1.0.yaml --savedir results\base_1024_test --n_samples 1 --bs 1 --height 576 --width 1024 --unconditional_guidance_scale 12.0 --ddim_steps 50 --ddim_eta 1.0 --prompt_file prompts\test_prompts.txt --fps 28

That allows me to run it, thanks. But unfortunately now I'm getting a new issue:

@CoLVDM Inference: 2023-11-01-15-53-39 Global seed set to 123 WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for: PyTorch 2.0.0+cu118 with CUDA 1108 (you have 2.1.0+cu118) Python 3.8.10 (you have 3.8.5) Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers) Memory-efficient attention, SwiGLU, sparse and more won't be available. Set XFORMERS_MORE_DETAILS=1 for more details A matching Triton is not available, some optimizations will not be enabled. Error caught was: No module named 'triton' AE working on z of shape (1, 4, 64, 64) = 16384 dimensions.

model checkpoint loaded. [rank:0] 2/2 samples loaded. [rank:0] batch-1 (1)x1 ... DDIM scale True ddim device cuda:0 C:\Users\Makazi\miniconda3-waifu\envs\VideoCrafter\lib\site-packages\torch\utils\checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants. warnings.warn( C:\Users\Makazi\miniconda3-waifu\envs\VideoCrafter\lib\site-packages\torch\utils\checkpoint.py:61: UserWarning: None of the inputs have requires_grad=True. Gradients will be None warnings.warn( Traceback (most recent call last): File "scripts\evaluation\inference.py", line 137, in run_inference(args, gpu_num, rank) File "scripts\evaluation\inference.py", line 122, in run_inference batch_samples = batch_ddim_sampling(model, cond, noise_shape, args.n_samples, \ File "E:\waifu\VideoCrafter-main\scripts\evaluation\funcs.py", line 50, in batch_ddimsampling samples, = ddim_sampler.sample(S=ddim_steps, File "C:\Users\Makazi\miniconda3-waifu\envs\VideoCrafter\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(*args, kwargs) File "E:\waifu\VideoCrafter-main\scripts\evaluation....\lvdm\models\samplers\ddim.py", line 114, in sample samples, intermediates = self.ddim_sampling(conditioning, size, File "C:\Users\Makazi\miniconda3-waifu\envs\VideoCrafter\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(*args, *kwargs) File "E:\waifu\VideoCrafter-main\scripts\evaluation....\lvdm\models\samplers\ddim.py", line 193, in ddim_sampling outs = self.p_sample_ddim(img, cond, ts, index=index, use_original_steps=ddim_use_original_steps, File "C:\Users\Makazi\miniconda3-waifu\envs\VideoCrafter\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(args, kwargs) File "E:\waifu\VideoCrafter-main\scripts\evaluation....\lvdm\models\samplers\ddim.py", line 230, in p_sample_ddim e_t = self.model.apply_model(x, t, c, kwargs) File "E:\waifu\VideoCrafter-main\scripts\evaluation....\lvdm\models\ddpm3d.py", line 522, in apply_model x_recon = self.model(x_noisy, t, cond, kwargs) File "C:\Users\Makazi\miniconda3-waifu\envs\VideoCrafter\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "C:\Users\Makazi\miniconda3-waifu\envs\VideoCrafter\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(args, kwargs) File "E:\waifu\VideoCrafter-main\scripts\evaluation....\lvdm\models\ddpm3d.py", line 712, in forward out = self.diffusion_model(x, t, context=cc, kwargs) File "C:\Users\Makazi\miniconda3-waifu\envs\VideoCrafter\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "C:\Users\Makazi\miniconda3-waifu\envs\VideoCrafter\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(args, kwargs) File "E:\waifu\VideoCrafter-main\scripts\evaluation....\lvdm\modules\networks\openaimodel3d.py", line 556, in forward h = module(h, emb, context=context, batch_size=b) File "C:\Users\Makazi\miniconda3-waifu\envs\VideoCrafter\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "C:\Users\Makazi\miniconda3-waifu\envs\VideoCrafter\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, *kwargs) File "E:\waifu\VideoCrafter-main\scripts\evaluation....\lvdm\modules\networks\openaimodel3d.py", line 41, in forward x = layer(x, context) File "C:\Users\Makazi\miniconda3-waifu\envs\VideoCrafter\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) File "C:\Users\Makazi\miniconda3-waifu\envs\VideoCrafter\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, kwargs) File "E:\waifu\VideoCrafter-main\scripts\evaluation....\lvdm\modules\attention.py", line 272, in forward x = block(x, context=context) File "C:\Users\Makazi\miniconda3-waifu\envs\VideoCrafter\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "C:\Users\Makazi\miniconda3-waifu\envs\VideoCrafter\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(args, kwargs) File "E:\waifu\VideoCrafter-main\scripts\evaluation....\lvdm\modules\attention.py", line 214, in forward return checkpoint(self._forward, input_tuple, self.parameters(), self.checkpoint) File "E:\waifu\VideoCrafter-main\scripts\evaluation....\lvdm\common.py", line 92, in checkpoint return ckpt(func, inputs) File "C:\Users\Makazi\miniconda3-waifu\envs\VideoCrafter\lib\site-packages\torch_compile.py", line 24, in inner return torch._dynamo.disable(fn, recursive)(args, kwargs) File "C:\Users\Makazi\miniconda3-waifu\envs\VideoCrafter\lib\site-packages\torch_dynamo\eval_frame.py", line 328, in _fn return fn(*args, *kwargs) File "C:\Users\Makazi\miniconda3-waifu\envs\VideoCrafter\lib\site-packages\torch_dynamo\external_utils.py", line 17, in inner return fn(args, kwargs) File "C:\Users\Makazi\miniconda3-waifu\envs\VideoCrafter\lib\site-packages\torch\utils\checkpoint.py", line 451, in checkpoint return CheckpointFunction.apply(function, preserve, args) File "C:\Users\Makazi\miniconda3-waifu\envs\VideoCrafter\lib\site-packages\torch\autograd\function.py", line 539, in apply return super().apply(args, kwargs) # type: ignore[misc] File "C:\Users\Makazi\miniconda3-waifu\envs\VideoCrafter\lib\site-packages\torch\utils\checkpoint.py", line 230, in forward outputs = run_function(args) File "E:\waifu\VideoCrafter-main\scripts\evaluation....\lvdm\modules\attention.py", line 217, in _forward x = self.attn1(self.norm1(x), context=context if self.disable_self_attn else None, mask=mask) + x File "C:\Users\Makazi\miniconda3-waifu\envs\VideoCrafter\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) File "C:\Users\Makazi\miniconda3-waifu\envs\VideoCrafter\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) File "E:\waifu\VideoCrafter-main\scripts\evaluation....\lvdm\modules\attention.py", line 154, in efficient_forward out = xformers.ops.memory_efficient_attention(q, k, v, attn_bias=None, op=None) File "C:\Users\Makazi\miniconda3-waifu\envs\VideoCrafter\lib\site-packages\xformers\ops\fmha__init.py", line 192, in memory_efficient_attention return _memory_efficient_attention( File "C:\Users\Makazi\miniconda3-waifu\envs\VideoCrafter\lib\site-packages\xformers\ops\fmha__init__.py", line 290, in _memory_efficient_attention return _memory_efficient_attention_forward( File "C:\Users\Makazi\miniconda3-waifu\envs\VideoCrafter\lib\site-packages\xformers\ops\fmha\init__.py", line 306, in _memory_efficient_attention_forward op = _dispatch_fw(inp) File "C:\Users\Makazi\miniconda3-waifu\envs\VideoCrafter\lib\site-packages\xformers\ops\fmha\dispatch.py", line 104, in _dispatch_fw return _run_priority_list( File "C:\Users\Makazi\miniconda3-waifu\envs\VideoCrafter\lib\site-packages\xformers\ops\fmha\dispatch.py", line 79, in _run_priority_list raise NotImplementedError(msg) NotImplementedError: No operator found for memory_efficient_attention_forward with inputs: query : shape=(80, 9216, 1, 64) (torch.float32) key : shape=(80, 9216, 1, 64) (torch.float32) value : shape=(80, 9216, 1, 64) (torch.float32) attn_bias : <class 'NoneType'> p : 0.0 flshattF is not supported because: xFormers wasn't build with CUDA support dtype=torch.float32 (supported: {torch.bfloat16, torch.float16}) Operator wasn't built - see python -m xformers.info for more info tritonflashattF is not supported because: xFormers wasn't build with CUDA support dtype=torch.float32 (supported: {torch.bfloat16, torch.float16}) Operator wasn't built - see python -m xformers.info for more info triton is not available requires A100 GPU cutlassF is not supported because: xFormers wasn't build with CUDA support Operator wasn't built - see python -m xformers.info for more info smallkF is not supported because: xFormers wasn't build with CUDA support max(query.shape[-1] != value.shape[-1]) > 32 Operator wasn't built - see python -m xformers.info for more info unsupported embed per head: 64

[zephirusgit]

I am testing in Windows with a 12GB RTX 2060 He is processing, I guess something is doing: D I will leave it until use cuda118 xformers to cuda118 pip3 install -U xformers --index-url https://download.pytorch.org/whl/cu118 torch from cuda118 pip3 install torch==2.1.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

launch with python ./scripts/evaluation/inference.py --seed 123 --mode 'base' --ckpt_path 'checkpoints/i2v_512_v1/model.ckpt' --config 'configs/inference_i2v_512_v1.0.yaml' --savedir results/i2v_512_test --n_samples 1 --bs 1 --height 320 --width 512 --unconditional_guidance_scale 12.0 --ddim_steps 50 --ddim_eta 1.0 --prompt_file "./prompts/i2v_prompts/test_prompts.txt" --fps 8 or python ./scripts/evaluation/inference.py --seed 123 --mode 'base' --ckpt_path 'checkpoints/base_1024_v1/model.ckpt' --config 'configs/inference_t2v_1024_v1.0.yaml' --savedir results/base_1024_test --n_samples 1 --bs 1 --height 576 --width 1024 --unconditional_guidance_scale 12.0 --ddim_steps 50 --ddim_eta 1.0 --prompt_file "./prompts/test_prompts.txt" --fps 28

oviously they have to have the CUDA installed, I use miniconda, You have to download the checkpoints in their corresponding folder, within the git. will also download a file called Open_clip_pytorch_Model.bin of 4GB (in hugginface folder temps)

I could launch it and it's processing, I guess something will be doing .. You are using the 12GB of Vram.

(videocrafter) PS H:\ia\VideoCrafter-main> python ./scripts/evaluation/inference.py --seed 123 --mode 'base' --ckpt_path 'checkpoints/i2v_512_v1/model.ckpt' --config 'configs/inference_i2v_512_v1.0.yaml' --savedir results/i2v_512_test --n_samples 1 --bs 1 --height 320 --width 512 --unconditional_guidance_scale 12.0 --ddim_steps 50 --ddim_eta 1.0 --prompt_file "./prompts/i2v_prompts/test_prompts.txt" --fps 8 @CoLVDM Inference: 2023-11-13-18-08-02 Global seed set to 123 A matching Triton is not available, some optimizations will not be enabled. Error caught was: No module named 'triton' AE working on z of shape (1, 4, 64, 64) = 16384 dimensions.

model checkpoint loaded. [rank:0] 2/2 samples loaded. [rank:0] batch-1 (1)x1 ... DDIM scale True ddim device cuda:0 C:\Users\ultim\miniconda3\envs\videocrafter\lib\site-packages\torch\utils\checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants. warnings.warn( C:\Users\ultim\miniconda3\envs\videocrafter\lib\site-packages\torch\utils\checkpoint.py:61: UserWarning: None of the inputs have requires_grad=True. Gradients will be None warnings.warn(

If I finish I tell them but I don't know what it does ..

[zephirusgit]

continue processing but is alive!!! in result see a horses video xD

[zephirusgit]

https://github.com/AILab-CVC/VideoCrafter/assets/20031912/e5a89638-1978-473e-8fdb-86b52c3e7750

[zephirusgit]

I still have not to be doing but the one who is quite decent, except the size and the water mark is that of diffusers, there for example creates a video every 20 seconds, of course smaller and does not have the ability to define an image I think.

[zephirusgit]

https://github.com/AILab-CVC/VideoCrafter/assets/20031912/46f5c1dd-cf04-41e3-9845-903c4b29982f