problems about i2v demo

[LeonJoe13]

thank you for your efforts in this great work, when I run the provided i2v demo, I got this video, I don't know why, could you help me ? here is log and result: [2023-12-19 10:43:15,802] INFO: Going into it2v_fullid_img_text inference on 0 gpu [2023-12-19 10:43:15,817] INFO: Loading ViT-H-14 model config. [2023-12-19 10:43:26,889] INFO: Loading pretrained ViT-H-14 weights (models/open_clip_pytorch_model.bin). [2023-12-19 10:44:24,635] INFO: Restored from models/v2-1_512-ema-pruned.ckpt [2023-12-19 10:44:54,441] INFO: Load model from models/i2vgen_xl_00854500.pth with status [2023-12-19 10:44:55,889] INFO: There are 3 videos. with 4 times [2023-12-19 10:44:55,889] INFO: Skip ## To test our images, it is recommended to run one data point at a time (i.e., uncommenting only one line at a time), which should reproduce our results. [2023-12-19 10:44:55,889] INFO: Skip ## To test our images, it is recommended to run one data point at a time (i.e., uncommenting only one line at a time), which should reproduce our results. [2023-12-19 10:44:55,889] INFO: Skip ## To test our images, it is recommended to run one data point at a time (i.e., uncommenting only one line at a time), which should reproduce our results. [2023-12-19 10:44:55,889] INFO: Skip ## To test our images, it is recommended to run one data point at a time (i.e., uncommenting only one line at a time), which should reproduce our results. [2023-12-19 10:44:55,893] INFO: Skip #data/test_images/img_0001.jpg|||A green frog floats on the surface of the water on green lotus leaves, with several pink lotus flowers, in a Chinese painting style. [2023-12-19 10:44:55,893] INFO: Skip #data/test_images/img_0001.jpg|||A green frog floats on the surface of the water on green lotus leaves, with several pink lotus flowers, in a Chinese painting style. [2023-12-19 10:44:55,893] INFO: Skip #data/test_images/img_0001.jpg|||A green frog floats on the surface of the water on green lotus leaves, with several pink lotus flowers, in a Chinese painting style. [2023-12-19 10:44:55,898] INFO: Skip #data/test_images/img_0001.jpg|||A green frog floats on the surface of the water on green lotus leaves, with several pink lotus flowers, in a Chinese painting style. [2023-12-19 10:44:55,898] INFO: [8]/[3] Begin to sample data/test_images/img_0002.png|||A blonde girl in jeans ... [2023-12-19 10:44:58,454] INFO: GPU Memory used 16.52 GB [2023-12-19 10:49:32,112] INFO: Save video to dir workspace/experiments/test_list_for_i2vgen/img_0002_01_00_A_blonde_girl_in_jeans_08.mp4: [2023-12-19 10:49:32,112] INFO: [9]/[3] Begin to sample data/test_images/img_0002.png|||A blonde girl in jeans ... [2023-12-19 10:49:32,746] INFO: GPU Memory used 31.60 GB [2023-12-19 10:53:58,297] INFO: Save video to dir workspace/experiments/test_list_for_i2vgen/img_0002_01_00_A_blonde_girl_in_jeans_09.mp4: [2023-12-19 10:53:58,297] INFO: [10]/[3] Begin to sample data/test_images/img_0002.png|||A blonde girl in jeans ... [2023-12-19 10:53:59,041] INFO: GPU Memory used 31.73 GB [2023-12-19 10:58:26,552] INFO: Save video to dir workspace/experiments/test_list_for_i2vgen/img_0002_01_00_A_blonde_girl_in_jeans_10.mp4: [2023-12-19 10:58:26,553] INFO: [11]/[3] Begin to sample data/test_images/img_0002.png|||A blonde girl in jeans ... [2023-12-19 10:58:27,266] INFO: GPU Memory used 31.73 GB [2023-12-19 11:03:03,906] INFO: Save video to dir workspace/experiments/test_list_for_i2vgen/img_0002_01_00_A_blonde_girl_in_jeans_11.mp4: [2023-12-19 11:03:03,907] INFO: Congratulations! The inference is completed!

[Steven-SWZhang]

Hi, thank you for your interest in our work. Can I ask what you have changed? This result seems abnormal, it looks like it might be a configuration error with Diffusion.

[LeonJoe13]

with

Hi, thank you for your interest in our work. Can I ask what you have changed? This result seems abnormal, it looks like it might be a configuration error with Diffusion.

thanks for your reply, I didn't change anything, except the env, here is my env: python 3.9.18 absl-py 1.4.0 accelerate 0.22.0 aiofiles 23.2.1 aiohttp 3.8.5 aiosignal 1.3.1 altair 5.0.1 annotated-types 0.5.0 antlr4-python3-runtime 4.9.3 anyio 3.7.1 asttokens 2.4.1 async-timeout 4.0.3 attrs 23.1.0 av 11.0.0 cachetools 5.3.1 certifi 2023.7.22 chardet 5.1.0 charset-normalizer 3.2.0 click 8.1.7 cmake 3.27.2 contourpy 1.1.0 cycler 0.11.0 datasets 2.14.4 decorator 5.1.1 decord 0.6.0 diffusers 0.21.4 dill 0.3.7 easydict 1.10 einops 0.6.1 exceptiongroup 1.1.3 executing 2.0.1 fairscale 0.4.6 fastapi 0.103.0 ffmpeg-python 0.2.0 ffmpy 0.3.1 filelock 3.12.2 flash-attn 0.2.1 fonttools 4.42.1 frozenlist 1.4.0 fsspec 2023.6.0 ftfy 6.1.1 future 0.18.3 google-auth 2.22.0 google-auth-oauthlib 1.0.0 gradio 3.41.2 gradio_client 0.5.0 grpcio 1.57.0 h11 0.14.0 httpcore 0.17.3 httpx 0.24.1 huggingface-hub 0.16.4 idna 3.4 imageio 2.15.0 imageio-ffmpeg 0.4.3 imgaug 0.4.0 importlib-metadata 6.8.0 importlib-resources 6.0.1 ipdb 0.13.13 ipython 8.18.1 jedi 0.19.1 Jinja2 3.1.2 joblib 1.3.2 jsonschema 4.19.0 jsonschema-specifications 2023.7.1 kiwisolver 1.4.5 lazy_loader 0.3 lightning-utilities 0.9.0 lit 16.0.6 lpips 0.1.4 Markdown 3.4.4 MarkupSafe 2.1.3 matplotlib 3.7.2 matplotlib-inline 0.1.6 motion-vector-extractor 1.0.6 mpmath 1.3.0 multidict 6.0.4 multiprocess 0.70.15 mypy-extensions 1.0.0 networkx 3.1 numpy 1.24.4 nvidia-cublas-cu11 11.10.3.66 nvidia-cuda-cupti-cu11 11.7.101 nvidia-cuda-nvrtc-cu11 11.7.99 nvidia-cuda-runtime-cu11 11.7.99 nvidia-cudnn-cu11 8.5.0.96 nvidia-cufft-cu11 10.9.0.58 nvidia-curand-cu11 10.2.10.91 nvidia-cusolver-cu11 11.4.0.1 nvidia-cusparse-cu11 11.7.4.91 nvidia-nccl-cu11 2.14.3 nvidia-nvtx-cu11 11.7.91 oauthlib 3.2.2 omegaconf 2.3.0 open-clip-torch 2.0.2 opencv-python 4.4.0.46 opencv-python-headless 4.7.0.68 orjson 3.9.5 packaging 23.2 pandas 2.0.3 parso 0.8.3 pexpect 4.9.0 Pillow 9.5.0 pip 23.3.1 pkgconfig 1.5.5 pkgutil_resolve_name 1.3.10 prompt-toolkit 3.0.43 protobuf 4.24.2 psutil 5.9.5 ptyprocess 0.7.0 pure-eval 0.2.2 pyarrow 13.0.0 pyasn1 0.5.0 pyasn1-modules 0.3.0 pydantic 2.3.0 pydantic_core 2.6.3 pyDeprecate 0.3.1 pydub 0.25.1 Pygments 2.17.2 pynvml 11.5.0 pyparsing 3.0.9 pyre-extensions 0.0.23 python-dateutil 2.8.2 python-multipart 0.0.6 pytorch-lightning 1.4.2 pytz 2023.3 PyWavelets 1.5.0 PyYAML 6.0.1 referencing 0.30.2 regex 2023.8.8 requests 2.31.0 requests-oauthlib 1.3.1 rotary-embedding-torch 0.2.1 rpds-py 0.9.2 rsa 4.9 sacremoses 0.1.1 safetensors 0.3.3 scikit-image 0.20.0 scikit-learn 1.3.2 scipy 1.9.1 semantic-version 2.10.0 setuptools 68.2.2 shapely 2.0.2 simplejson 3.18.4 six 1.16.0 sniffio 1.3.0 stack-data 0.6.3 starlette 0.27.0 sympy 1.12 tensorboard 2.14.0 tensorboard-data-server 0.7.1 threadpoolctl 3.2.0 tifffile 2023.12.9 timm 0.9.12 tokenizers 0.12.1 tomli 2.0.1 toolz 0.12.0 torch 1.13.1 torchdiffeq 0.2.3 torchmetrics 0.6.0 torchsde 0.2.6 torchvision 0.14.1 tqdm 4.66.1 traitlets 5.14.0 trampoline 0.1.2 transformers 4.18.0 triton 2.0.0.dev20221120 typing_extensions 4.9.0 typing-inspect 0.9.0 tzdata 2023.3 urllib3 1.26.16 uvicorn 0.23.2 wcwidth 0.2.12 websockets 11.0.3 Werkzeug 2.3.7 wheel 0.41.2 xformers 0.0.16 xxhash 3.3.0 yarl 1.9.2 zipp 3.16.2

[LeonJoe13]

if I follow your env, it can't work in NVIDIA-SMI 470.182.03 Driver Version: 470.182.03 CUDA Version: 11.8 my env is : absl-py 2.0.0 aiohttp 3.9.1 aiosignal 1.3.1 asttokens 2.4.1 async-timeout 4.0.3 attrs 23.1.0 backcall 0.2.0 cachetools 5.3.2 certifi 2023.11.17 chardet 5.1.0 charset-normalizer 3.3.2 click 8.1.7 cmake 3.28.1 decorator 5.1.1 easydict 1.10 einops 0.7.0 executing 2.0.1 fairscale 0.4.6 ffmpeg 1.4 filelock 3.13.1 flash-attn 0.2.0 frozenlist 1.4.1 fsspec 2023.12.2 ftfy 6.1.1 future 0.18.3 google-auth 2.25.2 google-auth-oauthlib 1.0.0 grpcio 1.60.0 huggingface-hub 0.19.4 idna 3.6 imageio 2.15.0 importlib-metadata 7.0.0 ipdb 0.13.13 ipython 8.12.3 jedi 0.19.1 joblib 1.3.2 lazy_loader 0.3 Markdown 3.5.1 MarkupSafe 2.1.3 matplotlib-inline 0.1.6 motion-vector-extractor 1.0.6 multidict 6.0.4 mypy-extensions 1.0.0 networkx 3.1 numpy 1.24.4 oauthlib 3.2.2 open-clip-torch 2.0.2 opencv-python 4.4.0.46 opencv-python-headless 4.7.0.68 packaging 23.2 parso 0.8.3 pexpect 4.9.0 pickleshare 0.7.5 Pillow 10.1.0 pip 23.3.1 pkgconfig 1.5.5 prompt-toolkit 3.0.43 protobuf 4.25.1 ptyprocess 0.7.0 pure-eval 0.2.2 pyasn1 0.5.1 pyasn1-modules 0.3.0 pyDeprecate 0.3.1 Pygments 2.17.2 pynvml 11.5.0 pyre-extensions 0.0.23 pytorch-lightning 1.4.2 PyWavelets 1.4.1 PyYAML 6.0.1 regex 2023.10.3 requests 2.31.0 requests-oauthlib 1.3.1 rotary-embedding-torch 0.2.1 rsa 4.9 sacremoses 0.1.1 scikit-image 0.20.0 scikit-learn 1.3.2 scipy 1.9.1 setuptools 68.2.2 simplejson 3.18.4 six 1.16.0 stack-data 0.6.3 tensorboard 2.14.0 tensorboard-data-server 0.7.2 threadpoolctl 3.2.0 tifffile 2023.7.10 tokenizers 0.12.1 tomli 2.0.1 torch 1.12.0+cu113 torchaudio 0.12.0 torchdiffeq 0.2.3 torchmetrics 0.6.0 torchsde 0.2.6 torchvision 0.13.0+cu113 tqdm 4.66.1 traitlets 5.14.0 trampoline 0.1.2 transformers 4.18.0 triton 2.0.0.dev20221120 typing_extensions 4.9.0 typing-inspect 0.9.0 urllib3 2.1.0 wcwidth 0.2.12 Werkzeug 3.0.1 wheel 0.41.2 xformers 0.0.13 yarl 1.9.4 zipp 3.17.0

the terminal shows: Exception: Failed to invoke function <function inference_i2vgen_entrance at 0x7f6699cecd30>, with CUDA error: no kernel image is available for execution on the device CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

[BaiMoHan]

same issue. I meet the hint: no kernel image is available for execution on the device

[Steven-SWZhang]

Hi, what is the status of your GPU memory usage?

[mingzhang1998]

Any solution to this? I tried to run the demo inference cmd of i2v, but it kept giving me error like this. I tried this model on a 32G V100 GPU and also on a 40G A100 GPU, and they returned the same error.

[rlrahulkanojia]

You need a 80GB GPU for running the inference.