search

ashawkey / RAD-NeRF

Real-time Neural Radiance Talking Portrait Synthesis via Audio-spatial Decomposition
MIT License
862 stars 151 forks source link

AttributeError: _2D in process.py #60

Closed gloomiebloomie closed 1 year ago

gloomiebloomie commented 1 year ago

[INFO] ===== extract audio from data/ian/ian.mp4 to data/ian/aud.wav ===== ffmpeg version 4.2.7-0ubuntu0.1 Copyright (c) 2000-2022 the FFmpeg developers built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1) configuration: --prefix=/usr --extra-version=0ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared libavutil 56. 31.100 / 56. 31.100 libavcodec 58. 54.100 / 58. 54.100 libavformat 58. 29.100 / 58. 29.100 libavdevice 58. 8.100 / 58. 8.100 libavfilter 7. 57.100 / 7. 57.100 libavresample 4. 0. 0 / 4. 0. 0 libswscale 5. 5.100 / 5. 5.100 libswresample 3. 5.100 / 3. 5.100 libpostproc 55. 5.100 / 55. 5.100 Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'data/ian/ian.mp4': Metadata: major_brand : isom minor_version : 512 compatible_brands: isomiso2avc1mp41 encoder : Lavf60.4.100 Duration: 00:01:29.36, start: 0.000000, bitrate: 376 kb/s Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 512x512 [SAR 1:1 DAR 1:1], 165 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default) Metadata: handler_name : VideoHandler encoder : Lavc60.6.101 libx264 Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 205 kb/s (default) Metadata: handler_name : SoundHandler File 'data/ian/aud.wav' already exists. Overwrite ? [y/N] y Stream mapping: Stream #0:1 -> #0:0 (aac (native) -> pcm_s16le (native)) Press [q] to stop, [?] for help Output #0, wav, to 'data/ian/aud.wav': Metadata: major_brand : isom minor_version : 512 compatible_brands: isomiso2avc1mp41 ISFT : Lavf58.29.100 Stream #0:0(und): Audio: pcm_s16le ([1][0][0][0] / 0x0001), 16000 Hz, stereo, s16, 512 kb/s (default) Metadata: handler_name : SoundHandler encoder : Lavc58.54.100 pcm_s16le size= 5571kB time=00:01:29.21 bitrate= 511.5kbits/s speed= 341x
video:0kB audio:5571kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.001367% [INFO] ===== extracted audio ===== [INFO] ===== extract audio labels for data/ian/aud.wav ===== ALSA lib confmisc.c:767:(parse_card) cannot find card '0' ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory ALSA lib conf.c:5220:(snd_config_expand) Evaluate error: No such file or directory ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM sysdefault ALSA lib confmisc.c:767:(parse_card) cannot find card '0' ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory ALSA lib conf.c:5220:(snd_config_expand) Evaluate error: No such file or directory ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM sysdefault ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.front ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.rear ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.center_lfe ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.side ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.surround21 ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.surround21 ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.surround40 ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.surround41 ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.surround50 ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.surround51 ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.surround71 ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.iec958 ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.iec958 ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.iec958 ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.hdmi ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.hdmi ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.modem ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.modem ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline ALSA lib confmisc.c:767:(parse_card) cannot find card '0' ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory ALSA lib conf.c:5220:(snd_config_expand) Evaluate error: No such file or directory ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM default ALSA lib confmisc.c:767:(parse_card) cannot find card '0' ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory ALSA lib conf.c:5220:(snd_config_expand) Evaluate error: No such file or directory ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM default ALSA lib confmisc.c:767:(parse_card) cannot find card '0' ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory ALSA lib conf.c:5220:(snd_config_expand) Evaluate error: No such file or directory ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM dmix [WARN] audio has 2 channels, only use the first. [INFO] loaded audio stream data/ian/aud.wav: (1426091,) [INFO] loading ASR model cpierse/wav2vec2-large-xlsr-53-esperanto... /usr/local/lib/python3.10/dist-packages/transformers/configuration_utils.py:380: UserWarning: Passing gradient_checkpointing to a config initialization is deprecated and will be removed in v5 Transformers. Using model.gradient_checkpointing_enable() instead, or if you are using the Trainer API, pass gradient_checkpointing=True in your TrainingArguments. warnings.warn( Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. 2023-06-23 16:03:36.111749: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT [START] balumo jne enus ien boro s an hjaĝo deco de kor ar ietinke mal fes juris denovana paŝinma nin maro ajnito a korpo listo nere aj nirekitapi mon nurde maifes si malmoneŭa j es mam in taj aŭ iĝon aame sajdi de rokvo enug isol s enŝa ajnas malo temas pli mon de faji menesvor la sta empael ol monde vosteri s avensaga sojni legaŭ longe despan n aj niciu mek ana ple ŝo sparŝandos plumie nekspasa pa en kapŝo maluco krakte soa kenŝajnu mano min pozo ajnice ger aŭ longe truata opegu ŝajn i ne pransaglan tajn tu kalŝim mane jasompendara nĝoj de diru pa jfane samteng aj maŭaj senĝoje boni pa j em aŭdojn nĝeni ra ten sodaŭvol aŭ raj tian bondon aĉ iuj saka insvuaj havuon la feriek al ombrulino ha aŭlonge estas vena ŭ bredas an ĉio aĉtas benmonda al mondo aŭdas kmaĝi taj afbodio muen al monce dancaj de emeĝf ron salen sesvidu pato vaj fakso li aj momo ŝe drandan fajni n aŭ poero nav ŝert mallonge ko opes antaŭ akvena ŝulkho de mal tematemon ajni t ie nafis pas a navsĉeuo ku da feis farea francu anĉa sed ne donpiu [END] [INFO] save all feats for training purpose... [INFO] saved logits to data/ian/aud_eo.npy [INFO] ===== extracted audio labels ===== [INFO] ===== extract images from data/ian/ian.mp4 to data/ian/ori_imgs ===== ffmpeg version 4.2.7-0ubuntu0.1 Copyright (c) 2000-2022 the FFmpeg developers built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1) configuration: --prefix=/usr --extra-version=0ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared libavutil 56. 31.100 / 56. 31.100 libavcodec 58. 54.100 / 58. 54.100 libavformat 58. 29.100 / 58. 29.100 libavdevice 58. 8.100 / 58. 8.100 libavfilter 7. 57.100 / 7. 57.100 libavresample 4. 0. 0 / 4. 0. 0 libswscale 5. 5.100 / 5. 5.100 libswresample 3. 5.100 / 3. 5.100 libpostproc 55. 5.100 / 55. 5.100 Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'data/ian/ian.mp4': Metadata: major_brand : isom minor_version : 512 compatible_brands: isomiso2avc1mp41 encoder : Lavf60.4.100 Duration: 00:01:29.36, start: 0.000000, bitrate: 376 kb/s Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 512x512 [SAR 1:1 DAR 1:1], 165 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default) Metadata: handler_name : VideoHandler encoder : Lavc60.6.101 libx264 Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 205 kb/s (default) Metadata: handler_name : SoundHandler Stream mapping: Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native)) Press [q] to stop, [?] for help [swscaler @ 0x5616d6548a80] deprecated pixel format used, make sure you did set range correctly Output #0, image2, to 'data/ian/ori_imgs/%d.jpg': Metadata: major_brand : isom minor_version : 512 compatible_brands: isomiso2avc1mp41 encoder : Lavf58.29.100 Stream #0:0(und): Video: mjpeg, yuvj420p(pc), 512x512 [SAR 1:1 DAR 1:1], q=1-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc (default) Metadata: handler_name : VideoHandler encoder : Lavc58.54.100 mjpeg Side data: cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: -1 frame= 2234 fps=239 q=1.0 Lsize=N/A time=00:01:29.36 bitrate=N/A speed=9.55x
video:134913kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown [INFO] ===== extracted images ===== [INFO] ===== extract semantics from data/ian/ori_imgs to data/ian/parsing ===== [INFO] loading model... 100% 2234/2234 [03:11<00:00, 11.66it/s] [INFO] ===== extracted semantics ===== [INFO] ===== extract background image from data/ian/ori_imgs ===== 100% 112/112 [01:32<00:00, 1.21it/s] [INFO] ===== extracted background image ===== [INFO] ===== extract torso and gt images for data/ian ===== 100% 2234/2234 [02:29<00:00, 14.99it/s] [INFO] ===== extracted torso and gt images ===== [INFO] ===== extract face landmarks from data/ian/ori_imgs ===== Traceback (most recent call last): File "/content/RAD-NeRF/data_utils/process.py", line 393, in extract_landmarks(ori_imgs_dir) File "/content/RAD-NeRF/data_utils/process.py", line 50, in extract_landmarks fa = face_alignment.FaceAlignment(face_alignment.LandmarksType._2D, flip_input=False) File "/usr/lib/python3.10/enum.py", line 437, in getattr raise AttributeError(name) from None AttributeError: _2D

JiadiLee commented 1 year ago

_2D landmarks type has been deleted in the new version of the face_alignment library. You can try change "_2D" to "TWO_D" for test. By the way, I also encountered "Unknown PCM cards" problem. What does this warning mean? Does this affect the final result?

gloomiebloomie commented 1 year ago

Update those errors didn't seem to effect finishing the processing but I'm just starting training now and it runs I'll update you after training if it works.

ahkimkoo commented 7 months ago

_2D landmarks type has been deleted in the new version of the face_alignment library. You can try change "_2D" to "TWO_D" for test. By the way, I also encountered "Unknown PCM cards" problem. What does this warning mean? Does this affect the final result?

it works