search

YuvalNirkin / fsgan

FSGAN - Official PyTorch Implementation
https://nirkin.com/fsgan
Creative Commons Zero v1.0 Universal
754 stars 147 forks source link

Failed to read frame during preprocessing for reenact.py #99

Closed harrodjordan closed 3 years ago

harrodjordan commented 3 years ago

In running the following code:

python ~/data/projects/fsgan/inference/reenact.py ~/data/datasets/tom_noaudio.mp4 -t ~/data/datasets/ts_noaudio.mp4 -o ~/data/outputs/out1.mp4 --encoder_codec mp4v

I consistently get the following error message during the pre-process videos step: "AssertionError: Failed to read frame from video in index: 53" It seems to be an error with opencv, and has persisted when opencv was installed using pip as well as when building it from source. I am using Ubuntu 16.04

I've tried re-encoding the video using ffmpeg -i <input> -r 25 <output>, as well as using --encoder_codec mp4v, and neither have worked. The videos were originally exported as h.264, 25fps, with the video encoded as avc1 and the audio as mp4, so I also tried stripping the audio using ffmpeg and using only the avc1 video, which also didn't work. I've tried those listed in #3 as well, with no success.

I plan to try exporting them as mpeg4 files tomorrow, but I wanted to see whether there was another known workaround for this issue.

I've included a truncated version of the full output below, only removing some of the "h.264 @" warnings for brevity: 


=> using GPU devices: 0
=> Loading face pose model: "hopenet_robust_alpha1.pth"...
=> Loading face landmarks model: "hr18_wflw_landmarks.pth"...
=> Loading face segmentation model: "celeba_unet_256_1_2_segmentation_v2.pth"...
=> Loading face reenactment model: "nfv_msrunet_256_1_2_reenactment_v2.1.pth"...
=> Computing face poses for video: "tom_noaudio_seq00.mp4"...
  0%|                                                                                                                                                         | 0/2 [00:00<?, ?batches/s][h264 @ 0x55ab2e89ca40] decode_slice_header error
[h264 @ 0x55ab2e89ca40] AVC: nal size 1909
[h264 @ 0x55ab36dd4940] AVC: nal size 1909
[h264 @ 0x55ab3889f420] AVC: nal size 753864392
[h264 @ 0x55ab3889f420] no frame!
[h264 @ 0x55ab2fd52a80] AVC: nal size 777904183
[h264 @ 0x55ab2fd52a80] AVC: nal size 777904183
[h264 @ 0x55ab2fd52a80] no frame!
[h264 @ 0x55ab335aecc0] reference picture missing during reorder
[h264 @ 0x55ab335aecc0] reference picture missing during reorder
[h264 @ 0x55ab335aecc0] reference picture missing during reorder
[h264 @ 0x55ab335aecc0] reference picture missing during reorder
[h264 @ 0x55ab335aecc0] reference picture missing during reorder
[h264 @ 0x55ab335aecc0] Missing reference picture, default is 65536
[h264 @ 0x55ab335aecc0] Missing reference picture, default is 65536
[h264 @ 0x55ab335aecc0] Missing reference picture, default is 65536
[h264 @ 0x55ab335aecc0] Missing reference picture, default is 65536
[h264 @ 0x55ab335aecc0] Missing reference picture, default is 65536
[h264 @ 0x55ab2e683060] mmco: unref short failure
[h264 @ 0x55ab2e683060] illegal short term buffer state detected
[h264 @ 0x55ab2e683060] Missing reference picture, default is 65530
[h264 @ 0x55ab388652c0] reference picture missing during reorder
[h264 @ 0x55ab388652c0] reference picture missing during reorder
[h264 @ 0x55ab388652c0] Missing reference picture, default is 65530
[h264 @ 0x55ab388652c0] Missing reference picture, default is 65530
[h264 @ 0x55ab2e32e700] mmco: unref short failure
[h264 @ 0x55ab36dd4940] reference picture missing during reorder
[h264 @ 0x55ab36dd4940] Missing reference picture, default is 65534
[h264 @ 0x55ab2f065840] mmco: unref short failure
  0%|                                                                                                                                                         | 0/2 [00:00<?, ?batches/s]
Traceback (most recent call last):
  File "/home/paperspace/data/projects/fsgan/inference/reenact.py", line 444, in <module>
    main(**vars(parser.parse_args()))
  File "/home/paperspace/data/projects/fsgan/inference/reenact.py", line 438, in main
    face_reenactment(source[0], target[0], output, select_source, select_target)
  File "/home/paperspace/data/projects/fsgan/inference/reenact.py", line 220, in __call__
    source_cache_dir, source_seq_file_path, _ = self.cache(source_path)
  File "/home/paperspace/data/projects/fsgan/preprocess/preprocess_video.py", line 468, in cache
    self.process_pose(input_path, output_dir, seq_file_path)
  File "/home/paperspace/data/projects/fsgan/preprocess/preprocess_video.py", line 252, in process_pose
    for i, frame in enumerate(tqdm(in_vid_loader, unit='batches', file=sys.stdout)):
  File "/home/paperspace/anaconda3/lib/python3.7/site-packages/tqdm/std.py", line 1167, in __iter__
    for obj in iterable:
  File "/home/paperspace/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 435, in __next__
    data = self._next_data()
  File "/home/paperspace/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1085, in _next_data
    return self._process_data(data)
  File "/home/paperspace/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1111, in _process_data
    data.reraise()
  File "/home/paperspace/anaconda3/lib/python3.7/site-packages/torch/_utils.py", line 428, in reraise
    raise self.exc_type(msg)
AssertionError: Caught AssertionError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/paperspace/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 198, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/paperspace/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/paperspace/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/paperspace/data/projects/fsgan/datasets/video_inference_dataset.py", line 45, in __getitem__
    assert frame_bgr is not None, 'Failed to read frame from video in index: %d' % index
AssertionError: Failed to read frame from video in index: 53
YuvalNirkin commented 3 years ago

Might be an issue with your OpenCV library. Did you try installing it from source like in the installation guide?

harrodjordan commented 3 years ago

Yep, same issue. Also tried installing the contrib and headless versions from source with the same error - was never able to resolve the issue and ended up switching to a different faceswap method.