search

williamyang1991 / FRESCO

[CVPR 2024] FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation
https://www.mmlab-ntu.com/project/fresco/
Other
734 stars 71 forks source link

ebsynth generation fails #26

Closed boxabirds closed 8 months ago

boxabirds commented 8 months ago

Both the web ui and directly run_fresco.py are failing at the same point. Here's my output. It suggests the run_fresco.pyfile is not producing the expected files. 

(FRESCO) gruntus FRESCO % python ./run_fresco.py ./config/config_carturn.yaml
====================================================================================================
loading configuration...
batch_size: 8
cond_scale: 0.7
controlnet_type: hed
end_opt_step: 15
file_path: ./data/car-turn.mp4
gmflow_path: ./model/gmflow_sintel-0c07dcb3.pth
max_process: 4
maxinterv: 5
mininterv: 5
num_inference_steps: 20
num_warmup_steps: 5
prompt: a red car turns in the winter
run_ebsynth: False
save_path: ./output/car-turn/
sd_path: SG161222/Realistic_Vision_V2.0
seed: 0
sod_path: ./model/epoch_resnet.pth
use_controlnet: True
use_freeu: False
use_salinecy: True

====================================================================================================
creating models...
create optical flow estimation model successfully!
create saliency detection model successfully!
create controlnet model-hed successfully!
unet/diffusion_pytorch_model.safetensors not found
Loading pipeline components...:  57%|████████▌      | 4/7 [00:01<00:01,  2.57it/s]/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  return self.fget.__get__(instance, owner)()
Loading pipeline components...: 100%|███████████████| 7/7 [00:01<00:00,  3.77it/s]
create diffusion model SG161222/Realistic_Vision_V2.0 successfully!

====================================================================================================
key frame selection for "./data/car-turn.mp4"...
processing 3 batches:
keyframe indexes [[0, 5, 10, 15, 20, 25, 30, 35], [40, 45, 50, 55, 60], [65, 70, 75]]

====================================================================================================
video to video translation...
processing batch [1/3] with 8 frames
/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
100%|█████████████████████████████████████████████| 15/15 [00:26<00:00,  1.73s/it]
processing batch [2/3] with 5 frames
100%|█████████████████████████████████████████████| 15/15 [00:22<00:00,  1.51s/it]
processing batch [3/3] with 3 frames
100%|█████████████████████████████████████████████| 15/15 [00:16<00:00,  1.11s/it]

====================================================================================================
to translate full video with ebsynth, install ebsynth and run:

python video_blend.py ./output/car-turn/ --key keys --key_ind 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 --output ./output/car-turn/blend.mp4 --fps 10 --n_proc 4 -ps



====================================================================================================
Done
(FRESCO) gruntus FRESCO % python video_blend.py ./output/car-turn/ --key keys --key_ind 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75  --output ./output/car-turn/blend.mp4 --fps 10 --n_proc 4 -ps
ic| f"about to call ebsynth with args": 'about to call ebsynth with args'
    args: ([0, 1, 2, 3], <blender.video_sequence.VideoSequence object at 0x7721bead1cd0>)
ic| f"about to call ebsynth with args": 'about to call ebsynth with args'
    args: ([4, 5, 6, 7], <blender.video_sequence.VideoSequence object at 0x7721bead1cd0>)
ic| f"about to call ebsynth with args": 'about to call ebsynth with args'
    args: ([8, 9, 10, 11],
           <blender.video_sequence.VideoSequence object at 0x7721bead1cd0>)
ic| f"about to call ebsynth with args": 'about to call ebsynth with args'
    args: ([12, 13, 14], <blender.video_sequence.VideoSequence object at 0x7721bead1cd0>)
/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
/home/julian/.local/share/virtualenvs/FRESCO-jUStlEeO/lib/python3.11/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
[ WARN:0@2.959] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_0/0001.jpg'): can't open/read file: check file path/integrity
Process Process-1:
Traceback (most recent call last):
  File "/home/julian/.asdf/installs/python/3.11.8/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/julian/.asdf/installs/python/3.11.8/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 112, in process_sequences
    process_one_sequence(i, video_sequence)
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 99, in process_one_sequence
    cmd += ' ' + g.get_cmd(j, w)
                 ^^^^^^^^^^^^^^^
  File "/home/julian/sambashare/expts/FRESCO/./src/ebsynth/blender/guide.py", line 96, in get_cmd
    warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/julian/sambashare/expts/FRESCO/./src/ebsynth/flow/flow_utils.py", line 230, in warp
    if len(img.shape) == 2:
           ^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'shape'
[ WARN:0@3.933] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_20/0021.jpg'): can't open/read file: check file path/integrity
Process Process-2:
Traceback (most recent call last):
  File "/home/julian/.asdf/installs/python/3.11.8/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/julian/.asdf/installs/python/3.11.8/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 112, in process_sequences
    process_one_sequence(i, video_sequence)
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 99, in process_one_sequence
    cmd += ' ' + g.get_cmd(j, w)
                 ^^^^^^^^^^^^^^^
  File "/home/julian/sambashare/expts/FRESCO/./src/ebsynth/blender/guide.py", line 96, in get_cmd
    warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/julian/sambashare/expts/FRESCO/./src/ebsynth/flow/flow_utils.py", line 230, in warp
    if len(img.shape) == 2:
           ^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'shape'
[ WARN:0@6.547] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_40/0041.jpg'): can't open/read file: check file path/integrity
Process Process-3:
Traceback (most recent call last):
  File "/home/julian/.asdf/installs/python/3.11.8/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/julian/.asdf/installs/python/3.11.8/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 112, in process_sequences
    process_one_sequence(i, video_sequence)
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 99, in process_one_sequence
    cmd += ' ' + g.get_cmd(j, w)
                 ^^^^^^^^^^^^^^^
  File "/home/julian/sambashare/expts/FRESCO/./src/ebsynth/blender/guide.py", line 96, in get_cmd
    warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/julian/sambashare/expts/FRESCO/./src/ebsynth/flow/flow_utils.py", line 230, in warp
    if len(img.shape) == 2:
           ^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'shape'
[ WARN:0@11.736] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_60/0061.jpg'): can't open/read file: check file path/integrity
Process Process-4:
Traceback (most recent call last):
  File "/home/julian/.asdf/installs/python/3.11.8/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/julian/.asdf/installs/python/3.11.8/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 112, in process_sequences
    process_one_sequence(i, video_sequence)
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 99, in process_one_sequence
    cmd += ' ' + g.get_cmd(j, w)
                 ^^^^^^^^^^^^^^^
  File "/home/julian/sambashare/expts/FRESCO/./src/ebsynth/blender/guide.py", line 96, in get_cmd
    warped_img = flow_calc.warp(prev_img, self.flows[i - 1],
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/julian/sambashare/expts/FRESCO/./src/ebsynth/flow/flow_utils.py", line 230, in warp
    if len(img.shape) == 2:
           ^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'shape'
ebsynth: 12.073577880859375
[ WARN:0@12.866] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_0/0001.jpg'): can't open/read file: check file path/integrity
[ WARN:0@12.866] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_0/0002.jpg'): can't open/read file: check file path/integrity
[ WARN:0@12.866] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_0/0003.jpg'): can't open/read file: check file path/integrity
[ WARN:0@12.866] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_0/0004.jpg'): can't open/read file: check file path/integrity
[ WARN:0@12.866] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_5/0005.jpg'): can't open/read file: check file path/integrity
[ WARN:0@12.866] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_5/0001.jpg'): can't open/read file: check file path/integrity
[ WARN:0@12.866] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_5/0002.jpg'): can't open/read file: check file path/integrity
[ WARN:0@12.866] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_5/0003.jpg'): can't open/read file: check file path/integrity
[ WARN:0@12.866] global loadsave.cpp:248 findDecoder imread_('./output/car-turn/out_5/0004.jpg'): can't open/read file: check file path/integrity
Traceback (most recent call last):
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 311, in <module>
    main(args)
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 271, in main
    process_seq(video_sequence, i, blend_histogram, blend_gradient)
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 203, in process_seq
    dist1s.append(load_error(bin_a, img_shape))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/julian/sambashare/expts/FRESCO/video_blend.py", line 164, in load_error
    with open(bin_path, 'rb') as fp:
         ^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: './output/car-turn/out_0/0001.bin'
``
boxabirds commented 8 months ago

Here's what my directory looks like

(FRESCO) gruntus FRESCO % tree output/car-turn/
output/car-turn/
├── blend
├── keys
│   ├── 0000.png
│   ├── 0005.png
│   ├── 0010.png
│   ├── 0015.png
│   ├── 0020.png
│   ├── 0025.png
│   ├── 0030.png
│   ├── 0035.png
│   ├── 0040.png
│   ├── 0045.png
│   ├── 0050.png
│   ├── 0055.png
│   ├── 0060.png
│   ├── 0065.png
│   ├── 0070.png
│   └── 0075.png
├── out_0
│   └── 0000.jpg
├── out_10
├── out_15
├── out_20
│   └── 0020.jpg
├── out_25
├── out_30
├── out_35
├── out_40
│   └── 0040.jpg
├── out_45
├── out_5
├── out_50
├── out_55
├── out_60
│   └── 0060.jpg
├── out_65
├── out_70
├── out_75
├── tmp
│   ├── flow_f_0000.npy
│   ├── flow_f_0000.png
│   ├── flow_f_0001.npy
│   ├── flow_f_0001.png
│   ├── flow_f_0002.npy
│   ├── flow_f_0002.png
│   ├── flow_f_0003.npy
│   ├── flow_f_0003.png
│   ├── flow_f_0020.npy
│   ├── flow_f_0020.png
│   ├── flow_f_0021.npy
│   ├── flow_f_0021.png
│   ├── flow_f_0022.npy
│   ├── flow_f_0022.png
│   ├── flow_f_0023.npy
│   ├── flow_f_0023.png
│   ├── flow_f_0040.npy
│   ├── flow_f_0040.png
│   ├── flow_f_0041.npy
│   ├── flow_f_0041.png
│   ├── flow_f_0042.npy
│   ├── flow_f_0042.png
│   ├── flow_f_0043.npy
│   ├── flow_f_0043.png
│   ├── flow_f_0060.npy
│   ├── flow_f_0060.png
│   ├── flow_f_0061.npy
│   ├── flow_f_0061.png
│   ├── flow_f_0062.npy
│   ├── flow_f_0062.png
│   ├── flow_f_0063.npy
│   ├── flow_f_0063.png
│   ├── out_0
│   │   ├── edge_0000.jpg
│   │   ├── edge_0001.jpg
│   │   ├── edge_0002.jpg
│   │   ├── edge_0003.jpg
│   │   ├── edge_0004.jpg
│   │   ├── pos_0000.jpg
│   │   ├── pos_0001.jpg
│   │   ├── pos_0002.jpg
│   │   ├── pos_0003.jpg
│   │   ├── pos_0004.jpg
│   │   ├── temporal_0000.jpg
│   │   └── temporal_0001.jpg
│   ├── out_10
│   ├── out_15
│   ├── out_20
│   │   ├── edge_0020.jpg
│   │   ├── edge_0021.jpg
│   │   ├── edge_0022.jpg
│   │   ├── edge_0023.jpg
│   │   ├── edge_0024.jpg
│   │   ├── pos_0020.jpg
│   │   ├── pos_0021.jpg
│   │   ├── pos_0022.jpg
│   │   ├── pos_0023.jpg
│   │   ├── pos_0024.jpg
│   │   ├── temporal_0020.jpg
│   │   └── temporal_0021.jpg
│   ├── out_25
│   ├── out_30
│   ├── out_35
│   ├── out_40
│   │   ├── edge_0040.jpg
│   │   ├── edge_0041.jpg
│   │   ├── edge_0042.jpg
│   │   ├── edge_0043.jpg
│   │   ├── edge_0044.jpg
│   │   ├── pos_0040.jpg
│   │   ├── pos_0041.jpg
│   │   ├── pos_0042.jpg
│   │   ├── pos_0043.jpg
│   │   ├── pos_0044.jpg
│   │   ├── temporal_0040.jpg
│   │   └── temporal_0041.jpg
│   ├── out_45
│   ├── out_5
│   ├── out_50
│   ├── out_55
│   ├── out_60
│   │   ├── edge_0060.jpg
│   │   ├── edge_0061.jpg
│   │   ├── edge_0062.jpg
│   │   ├── edge_0063.jpg
│   │   ├── edge_0064.jpg
│   │   ├── pos_0060.jpg
│   │   ├── pos_0061.jpg
│   │   ├── pos_0062.jpg
│   │   ├── pos_0063.jpg
│   │   ├── pos_0064.jpg
│   │   ├── temporal_0060.jpg
│   │   └── temporal_0061.jpg
│   ├── out_65
│   ├── out_70
│   └── out_75
└── video
    ├── 0000.png
    ├── 0001.png
    ├── 0002.png
    ├── 0003.png
    ├── 0004.png
    ├── 0005.png
    ├── 0006.png
    ├── 0007.png
    ├── 0008.png
    ├── 0009.png
    ├── 0010.png
    ├── 0011.png
    ├── 0012.png
    ├── 0013.png
    ├── 0014.png
    ├── 0015.png
    ├── 0016.png
    ├── 0017.png
    ├── 0018.png
    ├── 0019.png
    ├── 0020.png
    ├── 0021.png
    ├── 0022.png
    ├── 0023.png
    ├── 0024.png
    ├── 0025.png
    ├── 0026.png
    ├── 0027.png
    ├── 0028.png
    ├── 0029.png
    ├── 0030.png
    ├── 0031.png
    ├── 0032.png
    ├── 0033.png
    ├── 0034.png
    ├── 0035.png
    ├── 0036.png
    ├── 0037.png
    ├── 0038.png
    ├── 0039.png
    ├── 0040.png
    ├── 0041.png
    ├── 0042.png
    ├── 0043.png
    ├── 0044.png
    ├── 0045.png
    ├── 0046.png
    ├── 0047.png
    ├── 0048.png
    ├── 0049.png
    ├── 0050.png
    ├── 0051.png
    ├── 0052.png
    ├── 0053.png
    ├── 0054.png
    ├── 0055.png
    ├── 0056.png
    ├── 0057.png
    ├── 0058.png
    ├── 0059.png
    ├── 0060.png
    ├── 0061.png
    ├── 0062.png
    ├── 0063.png
    ├── 0064.png
    ├── 0065.png
    ├── 0066.png
    ├── 0067.png
    ├── 0068.png
    ├── 0069.png
    ├── 0070.png
    ├── 0071.png
    ├── 0072.png
    ├── 0073.png
    ├── 0074.png
    └── 0075.png

36 directories, 176 files
williamyang1991 commented 8 months ago

I think your issue is similar to https://github.com/williamyang1991/Rerender_A_Video/issues/32 and https://github.com/williamyang1991/Rerender_A_Video/issues/38 Myabe you can reinstall ebsynth?

boxabirds commented 8 months ago

I'm using ubuntu: where's the best place to log errors around execution of ebsynth? I don't mind trying to figure out how to install ebsynth but I'd like to know what's going on with it first. Do you have a script or command line example I can run to test ebsyth separately too?

williamyang1991 commented 8 months ago

https://github.com/williamyang1991/FRESCO/blob/9fe1be71b6c21890b5bc92659026f9586440266e/video_blend.py#L24

you just turn on the log by setting OPEN_EBSYNTH_LOG = True

boxabirds commented 8 months ago

It was a permissions problem --- install.py needs to chmod +x the ebsynth binary under linux. 

./src/ebsynth/deps/ebsynth/bin/ebsynth -style /home/julian/sambashare/expts/FRESCO/output/car-turn/keys/0020.png -guide /home/julian/sambashare/expts/FRESCO/output/car-turn/video/0020.png /home/julian/sambashare/expts/FRESCO/output/car-turn/video/0021.png -weight 6 -guide /home/julian/sambashare/expts/FRESCO/output/car-turn/tmp/out_20/edge_0020.jpg /home/julian/sambashare/expts/FRESCO/output/car-turn/tmp/out_20/edge_0021.jpg -weight 0.5 -guide /home/julian/sambashare/expts/FRESCO/output/car-turn/tmp/out_20/temporal_0020.jpg /home/julian/sambashare/expts/FRESCO/output/car-turn/tmp/out_20/temporal_0021.jpg -weight 0.5 -guide /home/julian/sambashare/expts/FRESCO/output/car-turn/tmp/out_20/pos_0020.jpg /home/julian/sambas/home/julian/sambashare/expts/FRESCO/run_fresco.ipynbhare/expts/FRESCO/output/car-turn/tmp/out_20/pos_0021.jpg -weight 2 -output /home/julian/sambashare/expts/FRESCO/output/car-turn/out_20/0021.jpg -searchvoteiters 12 -patchmatchiters 6
/bin/sh: 1: ./src/ebsynth/deps/ebsynth/bin/ebsynth: Permission denied
williamyang1991 commented 8 months ago

Yes, this is a common issue I summarized in my previous project image

boxabirds commented 8 months ago

Yeah it’s a bug in the install script.