Hanbo-Cheng / DAWN-pytorch

Offical implement of Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for talking head Video Generation
154 stars 9 forks source link

Error opening output files: No such file or directory #7

Closed nitinmukesh closed 3 days ago

nitinmukesh commented 3 days ago

After fixing the last error, this new error popped In this path cache\ood_test_1009\real_female_1\video\cache\target_audio.mp4. cache folder is not getting created inside video folder

(DAWN) C:\ai\DAWN-pytorch>run_ood_test\run_DM_v0_df_test_128_both_pose_blink.bat

(DAWN) C:\ai\DAWN-pytorch>REM Set variables

(DAWN) C:\ai\DAWN-pytorch>set test_name=ood_test_1009

(DAWN) C:\ai\DAWN-pytorch>set time_tag=tmp1009

(DAWN) C:\ai\DAWN-pytorch>set audio_path=WRA_MarcoRubio_000.wav

(DAWN) C:\ai\DAWN-pytorch>set image_path=real_female_1.jpeg

(DAWN) C:\ai\DAWN-pytorch>set cache_path=cache\tmp1009

(DAWN) C:\ai\DAWN-pytorch>set audio_emb_path=cache\target_audio.npy

(DAWN) C:\ai\DAWN-pytorch>set video_output_path=cache\

(DAWN) C:\ai\DAWN-pytorch>REM Activate the 3DDFA Conda environment and run the first script

(DAWN) C:\ai\DAWN-pytorch>call conda activate 3DDFA
">>>>>>>>>>>>>>1"
C:\ai\DAWN-pytorch\extract_init_states
C:\Users\nitin\miniconda3\envs\3DDFA\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py:69: UserWarning: Specified provider 'CUDAExecutionProvider' is not in available provider names.Available providers: 'AzureExecutionProvider, CPUExecutionProvider'
  warnings.warn(
">>>>>>>>>>>>>>2"
C:\ai\DAWN-pytorch
Loading the Wav2Vec2 Processor...
Ignored unknown kwarg option normalize
Ignored unknown kwarg option normalize
Ignored unknown kwarg option normalize
Ignored unknown kwarg option normalize
Loading the HuBERT Model...
2024-11-10 21:15:00.506843: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: SSE SSE2 SSE3 SSE4.1 SSE4.2 AVX AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.

TFHubertModel has backpropagation operations that are NOT supported on CPU. If you wish to train/fine-tune this model, you need a GPU or a TPU
2024-11-10 21:15:00.965718: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x213aaab28f0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2024-11-10 21:15:00.965943: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2024-11-10 21:15:00.977868: I .\tensorflow/compiler/jit/device_compiler.h:186] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.
All TF 2.0 model weights were used when initializing HubertModel.

All the weights of HubertModel were initialized from the TF 2.0 model.
If your task is similar to the task the model of the checkpoint was trained on, you can already use HubertModel for predictions without further training.
fnum525,hubersize1050
">>>>>>>>>>>>>>3"
C:\ai\DAWN-pytorch\PBnet
C:\Users\nitin\miniconda3\envs\DAWN\lib\site-packages\torch\nn\modules\transformer.py:282: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
  warnings.warn(f"enable_nested_tensor is True, but self.use_nested_tensor is False because {why_not_sparsity_fast_path}")
Restore weights..
eval!
eval!
">>>>>>>>>>>>>>4"
C:\ai\DAWN-pytorch
-j-of-tr-ddim0020_1.00
RESTORE_FROM: .\pretrain_models\DAWN_128.pth
cond scale: 1.0
sampling step: 20
C:\Users\nitin\miniconda3\envs\DAWN\lib\site-packages\torch\functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\TensorShape.cpp:3527.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
=> loading checkpoint '.\pretrain_models\DAWN_128.pth'
=> loaded checkpoint '.\pretrain_models\DAWN_128.pth'
torch.Size([1, 6])
torch.Size([1, 8])
sampling loop time step: 100%|████████████████████████████████████████████████████████| 20/20 [00:22<00:00,  1.12s/it]
DDIM time 22.328591346740723
C:\Users\nitin\miniconda3\envs\DAWN\lib\site-packages\torch\nn\functional.py:4296: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
  warnings.warn(
generation time 24.30060076713562
ffmpeg version 6.1-full_build-www.gyan.dev Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 12.2.0 (Rev10, Built by MSYS2 project)
  configuration: --enable-gpl --enable-version3 --enable-static --pkg-config=pkgconf --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-bzlib --enable-lzma --enable-libsnappy --enable-zlib --enable-librist --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-libbluray --enable-libcaca --enable-sdl2 --enable-libaribb24 --enable-libaribcaption --enable-libdav1d --enable-libdavs2 --enable-libuavs3d --enable-libzvbi --enable-librav1e --enable-libsvtav1 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libaom --enable-libjxl --enable-libopenjpeg --enable-libvpx --enable-mediafoundation --enable-libass --enable-frei0r --enable-libfreetype --enable-libfribidi --enable-libharfbuzz --enable-liblensfun --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-ffnvcodec --enable-nvdec --enable-nvenc --enable-dxva2 --enable-d3d11va --enable-libvpl --enable-libshaderc --enable-vulkan --enable-libplacebo --enable-opencl --enable-libcdio --enable-libgme --enable-libmodplug --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libshine --enable-libtheora --enable-libtwolame --enable-libvo-amrwbenc --enable-libcodec2 --enable-libilbc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-ladspa --enable-libbs2b --enable-libflite --enable-libmysofa --enable-librubberband --enable-libsoxr --enable-chromaprint
  libavutil      58. 29.100 / 58. 29.100
  libavcodec     60. 31.102 / 60. 31.102
  libavformat    60. 16.100 / 60. 16.100
  libavdevice    60.  3.100 / 60.  3.100
  libavfilter     9. 12.100 /  9. 12.100
  libswscale      7.  5.100 /  7.  5.100
  libswresample   4. 12.100 /  4. 12.100
  libpostproc    57.  3.100 / 57.  3.100
Trailing option(s) found in the command: may be ignored.
[aist#0:0/pcm_s16le @ 000002cb76bb6780] Guessed Channel Layout: mono
Input #0, wav, from 'C:\ai\DAWN-pytorch\tmpt_0wnoak.wav':
  Duration: 00:00:08.00, bitrate: 256 kb/s
  Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 16000 Hz, 1 channels, s16, 256 kb/s
Input #1, mov,mp4,m4a,3gp,3g2,mj2, from 'C:\ai\DAWN-pytorch\tmpq353_e86.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2mp41
    encoder         : Lavf58.76.100
  Duration: 00:00:08.00, start: 0.000000, bitrate: 131 kb/s
  Stream #1:0[0x1](und): Video: mpeg4 (Simple Profile) (mp4v / 0x7634706D), yuv420p, 128x128 [SAR 1:1 DAR 1:1], 129 kb/s, 25 fps, 25 tbr, 12800 tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
[out#0/mp4 @ 000002cb76bd4c00] Error opening output cache\\ood_test_1009\real_female_1\video\cache\target_audio.mp4: No such file or directory
Error opening output file cache\\ood_test_1009\real_female_1\video\cache\target_audio.mp4.
Error opening output files: No such file or directory
Permission denied: Unable to delete C:\ai\DAWN-pytorch\tmpt_0wnoak.wav.
Permission denied: Unable to delete C:\ai\DAWN-pytorch\tmpq353_e86.mp4.
29.944566 seconds
cache\\ood_test_1009\real_female_1\video
C:\ai\DAWN-pytorch\cache>tree /F
Folder PATH listing for volume Windows-SSD
Volume serial number is CE9F-A6AE
C:.
│   target_audio.npy
│
├───ood_test_1009
│   └───real_female_1
│       ├───img
│       │   └───cache
│       │       └───target_audio_1.00
│       │           ├───gt
│       │           └───samp
│       │                   000_c_1.00.png
│       │                   001_c_1.00.png
│       │                   002_c_1.00.png
│       │                   003_c_1.00.png
│       │                   004_c_1.00.png
│       │                   005_c_1.00.png
│       │                   006_c_1.00.png
│       │                   007_c_1.00.png
│       │                   008_c_1.00.png
│       │                   009_c_1.00.png
│       │                   010_c_1.00.png
│       │                   011_c_1.00.png
│       │                   012_c_1.00.png
│       │                   013_c_1.00.png
│       │                   014_c_1.00.png
│       │                   015_c_1.00.png
│       │                   016_c_1.00.png
│       │                   017_c_1.00.png
│       │                   018_c_1.00.png
│       │                   019_c_1.00.png
│       │                   020_c_1.00.png
│       │                   021_c_1.00.png
│       │                   022_c_1.00.png
│       │                   023_c_1.00.png
│       │                   024_c_1.00.png
│       │                   025_c_1.00.png
│       │                   026_c_1.00.png
│       │                   027_c_1.00.png
│       │                   028_c_1.00.png
│       │                   029_c_1.00.png
│       │                   030_c_1.00.png
│       │                   031_c_1.00.png
│       │                   032_c_1.00.png
│       │                   033_c_1.00.png
│       │                   034_c_1.00.png
│       │                   035_c_1.00.png
│       │                   036_c_1.00.png
│       │                   037_c_1.00.png
│       │                   038_c_1.00.png
│       │                   039_c_1.00.png
│       │                   040_c_1.00.png
│       │                   041_c_1.00.png
│       │                   042_c_1.00.png
│       │                   043_c_1.00.png
│       │                   044_c_1.00.png
│       │                   045_c_1.00.png
│       │                   046_c_1.00.png
│       │                   047_c_1.00.png
│       │                   048_c_1.00.png
│       │                   049_c_1.00.png
│       │                   050_c_1.00.png
│       │                   051_c_1.00.png
│       │                   052_c_1.00.png
│       │                   053_c_1.00.png
│       │                   054_c_1.00.png
│       │                   055_c_1.00.png
│       │                   056_c_1.00.png
│       │                   057_c_1.00.png
│       │                   058_c_1.00.png
│       │                   059_c_1.00.png
│       │                   060_c_1.00.png
│       │                   061_c_1.00.png
│       │                   062_c_1.00.png
│       │                   063_c_1.00.png
│       │                   064_c_1.00.png
│       │                   065_c_1.00.png
│       │                   066_c_1.00.png
│       │                   067_c_1.00.png
│       │                   068_c_1.00.png
│       │                   069_c_1.00.png
│       │                   070_c_1.00.png
│       │                   071_c_1.00.png
│       │                   072_c_1.00.png
│       │                   073_c_1.00.png
│       │                   074_c_1.00.png
│       │                   075_c_1.00.png
│       │                   076_c_1.00.png
│       │                   077_c_1.00.png
│       │                   078_c_1.00.png
│       │                   079_c_1.00.png
│       │                   080_c_1.00.png
│       │                   081_c_1.00.png
│       │                   082_c_1.00.png
│       │                   083_c_1.00.png
│       │                   084_c_1.00.png
│       │                   085_c_1.00.png
│       │                   086_c_1.00.png
│       │                   087_c_1.00.png
│       │                   088_c_1.00.png
│       │                   089_c_1.00.png
│       │                   090_c_1.00.png
│       │                   091_c_1.00.png
│       │                   092_c_1.00.png
│       │                   093_c_1.00.png
│       │                   094_c_1.00.png
│       │                   095_c_1.00.png
│       │                   096_c_1.00.png
│       │                   097_c_1.00.png
│       │                   098_c_1.00.png
│       │                   099_c_1.00.png
│       │                   100_c_1.00.png
│       │                   101_c_1.00.png
│       │                   102_c_1.00.png
│       │                   103_c_1.00.png
│       │                   104_c_1.00.png
│       │                   105_c_1.00.png
│       │                   106_c_1.00.png
│       │                   107_c_1.00.png
│       │                   108_c_1.00.png
│       │                   109_c_1.00.png
│       │                   110_c_1.00.png
│       │                   111_c_1.00.png
│       │                   112_c_1.00.png
│       │                   113_c_1.00.png
│       │                   114_c_1.00.png
│       │                   115_c_1.00.png
│       │                   116_c_1.00.png
│       │                   117_c_1.00.png
│       │                   118_c_1.00.png
│       │                   119_c_1.00.png
│       │                   120_c_1.00.png
│       │                   121_c_1.00.png
│       │                   122_c_1.00.png
│       │                   123_c_1.00.png
│       │                   124_c_1.00.png
│       │                   125_c_1.00.png
│       │                   126_c_1.00.png
│       │                   127_c_1.00.png
│       │                   128_c_1.00.png
│       │                   129_c_1.00.png
│       │                   130_c_1.00.png
│       │                   131_c_1.00.png
│       │                   132_c_1.00.png
│       │                   133_c_1.00.png
│       │                   134_c_1.00.png
│       │                   135_c_1.00.png
│       │                   136_c_1.00.png
│       │                   137_c_1.00.png
│       │                   138_c_1.00.png
│       │                   139_c_1.00.png
│       │                   140_c_1.00.png
│       │                   141_c_1.00.png
│       │                   142_c_1.00.png
│       │                   143_c_1.00.png
│       │                   144_c_1.00.png
│       │                   145_c_1.00.png
│       │                   146_c_1.00.png
│       │                   147_c_1.00.png
│       │                   148_c_1.00.png
│       │                   149_c_1.00.png
│       │                   150_c_1.00.png
│       │                   151_c_1.00.png
│       │                   152_c_1.00.png
│       │                   153_c_1.00.png
│       │                   154_c_1.00.png
│       │                   155_c_1.00.png
│       │                   156_c_1.00.png
│       │                   157_c_1.00.png
│       │                   158_c_1.00.png
│       │                   159_c_1.00.png
│       │                   160_c_1.00.png
│       │                   161_c_1.00.png
│       │                   162_c_1.00.png
│       │                   163_c_1.00.png
│       │                   164_c_1.00.png
│       │                   165_c_1.00.png
│       │                   166_c_1.00.png
│       │                   167_c_1.00.png
│       │                   168_c_1.00.png
│       │                   169_c_1.00.png
│       │                   170_c_1.00.png
│       │                   171_c_1.00.png
│       │                   172_c_1.00.png
│       │                   173_c_1.00.png
│       │                   174_c_1.00.png
│       │                   175_c_1.00.png
│       │                   176_c_1.00.png
│       │                   177_c_1.00.png
│       │                   178_c_1.00.png
│       │                   179_c_1.00.png
│       │                   180_c_1.00.png
│       │                   181_c_1.00.png
│       │                   182_c_1.00.png
│       │                   183_c_1.00.png
│       │                   184_c_1.00.png
│       │                   185_c_1.00.png
│       │                   186_c_1.00.png
│       │                   187_c_1.00.png
│       │                   188_c_1.00.png
│       │                   189_c_1.00.png
│       │                   190_c_1.00.png
│       │                   191_c_1.00.png
│       │                   192_c_1.00.png
│       │                   193_c_1.00.png
│       │                   194_c_1.00.png
│       │                   195_c_1.00.png
│       │                   196_c_1.00.png
│       │                   197_c_1.00.png
│       │                   198_c_1.00.png
│       │                   199_c_1.00.png
│       │
│       └───video
└───tmp1009
        dri_blink.npy
        dri_pose.npy
Hanbo-Cheng commented 3 days ago

In DM_3\test_demo\test_VIDEO_hdtf_df_wpose_face_cond_init_ca_newae_ood_256_2.py line 99: directory_name = (args.source_img_path).split('/')[-1].split('.')[0] the .split('/') should be replaced to .split('\') for windows system.

The ffmpeg has no permission to create a directory I suppose.

Hanbo-Cheng commented 3 days ago

Thank you for pointing it out. I need to explain this in the readme.

nitinmukesh commented 3 days ago

I tried .split('\') as single backslash will escape '.

Still same error

[out#0/mp4 @ 000001ba5a234a80] Error opening output cache\ood_test_1009\real_female_1\video\cache\target_audio.mp4: No such file or directory Error opening output file cache\ood_test_1009\real_female_1\video\cache\target_audio.mp4. Error opening output files: No such file or directory Permission denied: Unable to delete C:\ai\DAWN-pytorch\tmpyscct4sk.wav. Permission denied: Unable to delete C:\ai\DAWN-pytorch\tmpdi7uxexc.mp4.

Hanbo-Cheng commented 3 days ago

Plus, it seems that you missed the first step (using 3DDFA to extract the initial state of the portrait). Although I use a default value in the code when the initial state is missing, it will usually cause worse results. If you have any problems when extracting the initial states, please let me know.

Hanbo-Cheng commented 3 days ago

I intend to save the video in cache\ood_test_1009\real_female_1\video\target_audio.mp4, maybe you should check the similar problem in DM_3\test_demo\test_VIDEO_hdtf_df_wpose_face_cond_init_ca_newae_ood_256_2.py, such as line 238.

I will try to think of a way that is compatible with the Windows platform.

nitinmukesh commented 3 days ago

For the time being solved the issue using

directory_name = (args.source_img_path).split('\\')[-1].split('.')[0]
CKPT_DIR = os.path.join(args.save_path, directory_name ,'video')
os.makedirs(CKPT_DIR, exist_ok=True)
IMG_DIR = os.path.join(args.save_path, directory_name, 'img')
os.makedirs(IMG_DIR, exist_ok=True)
VID_DIR = os.path.join(CKPT_DIR, 'cache')
os.makedirs(VID_DIR, exist_ok=True)
Input #0, wav, from 'C:\ai\DAWN-pytorch\tmpry627sy_.wav':
  Duration: 00:00:08.00, bitrate: 256 kb/s
  Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 16000 Hz, 1 channels, s16, 256 kb/s
Input #1, mov,mp4,m4a,3gp,3g2,mj2, from 'C:\ai\DAWN-pytorch\tmpv7prl144.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2mp41
    encoder         : Lavf58.76.100
  Duration: 00:00:08.00, start: 0.000000, bitrate: 131 kb/s
  Stream #1:0[0x1](und): Video: mpeg4 (Simple Profile) (mp4v / 0x7634706D), yuv420p, 128x128 [SAR 1:1 DAR 1:1], 129 kb/s, 25 fps, 25 tbr, 12800 tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
Stream mapping:
  Stream #1:0 -> #0:0 (copy)
  Stream #0:0 -> #0:1 (pcm_s16le (native) -> aac (native))
Press [q] to stop, [?] for help
Output #0, mp4, to 'cache\\ood_test_1009\real_female_1\video\cache\target_audio.mp4':
  Metadata:
    encoder         : Lavf60.16.100
  Stream #0:0(und): Video: mpeg4 (Simple Profile) (mp4v / 0x7634706D), yuv420p, 128x128 [SAR 1:1 DAR 1:1], q=2-31, 129 kb/s, 25 fps, 25 tbr, 12800 tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
  Stream #0:1: Audio: aac (LC) (mp4a / 0x6134706D), 16000 Hz, stereo, fltp, 128 kb/s
    Metadata:
      encoder         : Lavc60.31.102 aac
[out#0/mp4 @ 000001ca83c051c0] video:127kB audio:80kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 2.370734%
size=     212kB time=00:00:07.96 bitrate= 218.2kbits/s speed=  25x
[aac @ 000001ca83c18f40] Qavg: 63553.680
Permission denied: Unable to delete C:\ai\DAWN-pytorch\tmpry627sy_.wav.
Permission denied: Unable to delete C:\ai\DAWN-pytorch\tmpv7prl144.mp4.
29.8978938 seconds
cache\\ood_test_1009\real_female_1\video