search

yiranran / Audio-driven-TalkingFace-HeadPose

Code for "Audio-driven Talking Face Video Generation with Learning-based Personalized Head Pose" (Arxiv 2020) and "Predicting Personalized Head Movement From Short Video and Speech Signal" (TMM 2022)
https://ieeexplore.ieee.org/document/9894719
721 stars 146 forks source link

No 60_net_G.pth file #8

Closed kryzhikov closed 4 years ago

kryzhikov commented 4 years ago

Running this project on colab Getting error on running train19_news1.py here is colab https://colab.research.google.com/drive/1FXoqSLC_y6UpDDcbxGefwMDBh9UTipvZ

And error 

/content/Audio-driven-TalkingFace-HeadPose/render-to-video
19_news/11 11_bmold_win3
sh: 1: matlab: not found
loading models/model-r100-ii/model 0
[20:08:28] src/nnvm/legacy_json_util.cc:209: Loading symbol saved by previous version v1.2.0. Attempting to upgrade...
[20:08:28] src/nnvm/legacy_json_util.cc:217: Symbol successfully upgraded!

Segmentation fault: 11

Stack trace:
  [bt] (0) /usr/local/lib/python3.6/dist-packages/mxnet/libmxnet.so(+0x3c27360) [0x7f214559b360]
loading models/model-r100-ii/model 0
[20:08:30] src/nnvm/legacy_json_util.cc:209: Loading symbol saved by previous version v1.2.0. Attempting to upgrade...
[20:08:30] src/nnvm/legacy_json_util.cc:217: Symbol successfully upgraded!

Segmentation fault: 11

Stack trace:
  [bt] (0) /usr/local/lib/python3.6/dist-packages/mxnet/libmxnet.so(+0x3c27360) [0x7f5400af1360]
----------------- Options ---------------
                       Nw: 3                             
                    alpha: 0.3                           
                attention: 1                             
               batch_size: 1                             
                    beta1: 0.5                           
          checkpoints_dir: ./checkpoints                 
           continue_train: True                             [default: False]
                crop_size: 256                           
                 dataroot: 11_bmold_win3                    [default: None]
             dataset_mode: aligned_feature_multi         
                direction: AtoB                          
              display_env: memory_seq_11                    [default: main]
             display_freq: 400                           
               display_id: 1                             
            display_ncols: 4                             
             display_port: 8097                          
           display_server: http://localhost              
          display_winsize: 256                           
         do_saturate_mask: False                         
                    epoch: 0                                [default: latest]
              epoch_count: 1                             
                 gan_mode: vanilla                       
                  gpu_ids: 0                             
            iden_feat_dim: 512                           
            iden_feat_dir: arcface/iden_feat/            
               iden_thres: 0.98                          
                init_gain: 0.02                          
                init_type: normal                        
                 input_nc: 3                             
                  isTrain: True                             [default: None]
                lambda_L1: 100.0                         
              lambda_mask: 2.0                              [default: 0.1]
       lambda_mask_smooth: 1e-05                         
                load_iter: 0                                [default: 0]
                load_size: 286                           
                       lr: 0.0001                           [default: 0.0002]
           lr_decay_iters: 50                            
                lr_policy: linear                        
         max_dataset_size: inf                           
                 mem_size: 30000                         
                    model: memory_seq                       [default: cycle_gan]
               n_layers_D: 3                             
                     name: memory_seq_p2p/11                [default: experiment_name]
                      ndf: 64                            
                     netD: basic                         
                     netG: unetac_adain_256              
                      ngf: 64                            
                    niter: 60                               [default: 100]
              niter_decay: 0                                [default: 100]
               no_dropout: False                         
                  no_flip: False                         
                  no_html: False                         
                     norm: batch                         
              num_threads: 4                             
                output_nc: 3                             
                    phase: train                         
                pool_size: 0                             
               preprocess: resize_and_crop               
               print_freq: 100                           
             resizemethod: lanczos                       
             save_by_iter: False                         
          save_epoch_freq: 5                             
         save_latest_freq: 5000                          
           serial_batches: False                         
         spatial_feat_dim: 512                           
                   suffix:                               
                    top_k: 256                           
         update_html_freq: 1000                          
                  verbose: False                         
----------------- End -------------------
dataset [AlignedFeatureMultiDataset] was created
The number of training images = 298
initialize network with normal
initialize network with normal
model [MemorySeqModel] was created
loading the model from ./checkpoints/memory_seq_p2p/0_net_G.pth
loading the model from ./checkpoints/memory_seq_p2p/0_net_D.pth
loading the model from ./checkpoints/memory_seq_p2p/0_net_mem.pth
---------- Networks initialized -------------
[Network G] Total number of parameters : 259.056 M
[Network D] Total number of parameters : 2.775 M
[Network mem] Total number of parameters : 11.952 M
-----------------------------------------------
Setting up a new session...
Exception in user code:
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 159, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/util/connection.py", line 80, in create_connection
    raise err
  File "/usr/local/lib/python3.6/dist-packages/urllib3/util/connection.py", line 70, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.6/http/client.py", line 1254, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1300, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1249, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1036, in _send_output
    self.send(msg)
  File "/usr/lib/python3.6/http/client.py", line 974, in send
    self.connect()
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 181, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 168, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f506009f198>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.6/dist-packages/urllib3/util/retry.py", line 399, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8097): Max retries exceeded with url: /env/memory_seq_11 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f506009f198>: Failed to establish a new connection: [Errno 111] Connection refused',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/visdom/__init__.py", line 711, in _send
    data=json.dumps(msg),
  File "/usr/local/lib/python3.6/dist-packages/visdom/__init__.py", line 677, in _handle_post
    r = self.session.post(url, data=data)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 581, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8097): Max retries exceeded with url: /env/memory_seq_11 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f506009f198>: Failed to establish a new connection: [Errno 111] Connection refused',))
[Errno 99] Cannot assign requested address
[Errno 99] Cannot assign requested address
[Errno 99] Cannot assign requested address
Visdom python client failed to establish socket to get messages from the server. This feature is optional and can be disabled by initializing Visdom with `use_incoming_socket=False`, which will prevent waiting for this request to timeout.

Could not connect to Visdom server. 
 Trying to start a server....
Command: /usr/bin/python3 -m visdom.server -p 8097 &>/dev/null &
create web directory ./checkpoints/memory_seq_p2p/11/web...
Traceback (most recent call last):
  File "train.py", line 45, in <module>
    for i, data in enumerate(dataset):  # inner loop within one epoch
  File "/content/Audio-driven-TalkingFace-HeadPose/render-to-video/data/__init__.py", line 90, in __iter__
    for i, data in enumerate(self.dataloader):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 856, in _next_data
    return self._process_data(data)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 881, in _process_data
    data.reraise()
  File "/usr/local/lib/python3.6/dist-packages/torch/_utils.py", line 394, in reraise
    raise self.exc_type(msg)
FileNotFoundError: Caught FileNotFoundError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/content/Audio-driven-TalkingFace-HeadPose/render-to-video/data/aligned_feature_multi_dataset.py", line 54, in __getitem__
    A = Image.open(AB_path).convert('RGB')
  File "/usr/local/lib/python3.6/dist-packages/PIL/Image.py", line 2809, in open
    fp = builtins.open(filename, "rb")
FileNotFoundError: [Errno 2] No such file or directory: '/content/Audio-driven-TalkingFace-HeadPose/render-to-video/../Deep3DFaceReconstruction/output/render/19_news/11/bm/frame111_renderold_bm.png'

----------------- Options ---------------
                       Nw: 3                             
                    alpha: 0.3                           
             aspect_ratio: 1.0                           
                attention: 1                             
               batch_size: 1                             
             blinkframeid: 41                            
          checkpoints_dir: ./checkpoints                 
                crop_size: 256                           
                 dataroot: 11_bmold_win3                    [default: None]
             dataset_mode: aligned_feature_multi         
                direction: AtoB                          
          display_winsize: 256                           
         do_saturate_mask: False                         
                    epoch: 60                               [default: latest]
                     eval: False                         
                  gpu_ids: 0                             
            iden_feat_dim: 512                           
            iden_feat_dir: arcface/iden_feat/            
               iden_thres: 0.98                          
              imagefolder: images60                         [default: images]
                init_gain: 0.02                          
                init_type: normal                        
                 input_nc: 3                             
                  isTrain: False                            [default: None]
                load_iter: 0                                [default: 0]
                load_size: 256                           
         max_dataset_size: inf                           
                 mem_size: 30000                         
                    model: memory_seq                       [default: test]
                        n: 26                            
               n_layers_D: 3                             
                     name: memory_seq_p2p/11                [default: experiment_name]
                      ndf: 64                            
                     netD: basic                         
                     netG: unetac_adain_256              
                      ngf: 64                            
               no_dropout: False                         
                  no_flip: False                         
                     norm: batch                         
                    ntest: inf                           
                 num_test: 200                              [default: 50]
              num_threads: 4                             
                output_nc: 3                             
                    phase: test                          
               preprocess: resize_and_crop               
             resizemethod: lanczos                       
              results_dir: ./results/                    
           serial_batches: False                         
         spatial_feat_dim: 512                           
                   suffix:                               
          test_batch_list:                               
              test_use_gt: 0                             
                    top_k: 256                           
                  verbose: False                         
----------------- End -------------------
dataset [AlignedFeatureMultiDataset] was created
initialize network with normal
model [MemorySeqModel] was created
loading the model from ./checkpoints/memory_seq_p2p/60_net_G.pth
Traceback (most recent call last):
  File "test.py", line 47, in <module>
    model.setup(opt)               # regular setup: load and print networks; create schedulers
  File "/content/Audio-driven-TalkingFace-HeadPose/render-to-video/models/base_model.py", line 89, in setup
    self.load_networks(load_suffix)
  File "/content/Audio-driven-TalkingFace-HeadPose/render-to-video/models/base_model.py", line 202, in load_networks
    state_dict = torch.load(load_path, map_location=str(self.device))
  File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 525, in load
    with _open_file_like(f, 'rb') as opened_file:
  File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 212, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 193, in __init__
    super(_open_file, self).__init__(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: './checkpoints/memory_seq_p2p/60_net_G.pth'
kryzhikov commented 4 years ago

And can`t access to rendered images 

FileNotFoundError: [Errno 2] No such file or directory: '/content/Audio-driven-TalkingFace-HeadPose/render-to-video/../Deep3DFaceReconstruction/output/render/19_news/11/bm/frame133_renderold_bm.png'
kryzhikov commented 4 years ago

Running both with 

!python /content/Audio-driven-TalkingFace-HeadPose/render-to-video/train_19news_1.py 11 0

and 

%cd /content/Audio-driven-TalkingFace-HeadPose/render-to-video/
!python train_19news_1.py 11 0
kryzhikov commented 4 years ago

Still have problem with CheckPoints Managed to fix access to renders, but now it is trying to gain access to some render files, that do not exist. /content/Audio-driven-TalkingFace-HeadPose/Deep3DFaceReconstruction/output/render/19_news/11/bm/frame112_renderold_bm.png

19_news/11 11_bmold_win3
sh: 1: matlab: not found
loading models/model-r100-ii/model 0
[09:53:11] src/nnvm/legacy_json_util.cc:209: Loading symbol saved by previous version v1.2.0. Attempting to upgrade...
[09:53:11] src/nnvm/legacy_json_util.cc:217: Symbol successfully upgraded!

Segmentation fault: 11

Stack trace:
  [bt] (0) /usr/local/lib/python3.6/dist-packages/mxnet/libmxnet.so(+0x3c27360) [0x7fec0fc9b360]
loading models/model-r100-ii/model 0
[09:53:13] src/nnvm/legacy_json_util.cc:209: Loading symbol saved by previous version v1.2.0. Attempting to upgrade...
[09:53:13] src/nnvm/legacy_json_util.cc:217: Symbol successfully upgraded!

Segmentation fault: 11

Stack trace:
  [bt] (0) /usr/local/lib/python3.6/dist-packages/mxnet/libmxnet.so(+0x3c27360) [0x7f1a2ad88360]
----------------- Options ---------------
                       Nw: 3                             
                    alpha: 0.3                           
                attention: 1                             
               batch_size: 1                             
                    beta1: 0.5                           
          checkpoints_dir: ./checkpoints                 
           continue_train: True                             [default: False]
                crop_size: 256                           
                 dataroot: 11_bmold_win3                    [default: None]
             dataset_mode: aligned_feature_multi         
                direction: AtoB                          
              display_env: memory_seq_11                    [default: main]
             display_freq: 400                           
               display_id: 1                             
            display_ncols: 4                             
             display_port: 8097                          
           display_server: http://localhost              
          display_winsize: 256                           
         do_saturate_mask: False                         
                    epoch: 0                                [default: latest]
              epoch_count: 1                             
                 gan_mode: vanilla                       
                  gpu_ids: 0                             
            iden_feat_dim: 512                           
            iden_feat_dir: arcface/iden_feat/            
               iden_thres: 0.98                          
                init_gain: 0.02                          
                init_type: normal                        
                 input_nc: 3                             
                  isTrain: True                             [default: None]
                lambda_L1: 100.0                         
              lambda_mask: 2.0                              [default: 0.1]
       lambda_mask_smooth: 1e-05                         
                load_iter: 0                                [default: 0]
                load_size: 286                           
                       lr: 0.0001                           [default: 0.0002]
           lr_decay_iters: 50                            
                lr_policy: linear                        
         max_dataset_size: inf                           
                 mem_size: 30000                         
                    model: memory_seq                       [default: cycle_gan]
               n_layers_D: 3                             
                     name: memory_seq_p2p/11                [default: experiment_name]
                      ndf: 64                            
                     netD: basic                         
                     netG: unetac_adain_256              
                      ngf: 64                            
                    niter: 60                               [default: 100]
              niter_decay: 0                                [default: 100]
               no_dropout: False                         
                  no_flip: False                         
                  no_html: False                         
                     norm: batch                         
              num_threads: 4                             
                output_nc: 3                             
                    phase: train                         
                pool_size: 0                             
               preprocess: resize_and_crop               
               print_freq: 100                           
             resizemethod: lanczos                       
             save_by_iter: False                         
          save_epoch_freq: 5                             
         save_latest_freq: 5000                          
           serial_batches: False                         
         spatial_feat_dim: 512                           
                   suffix:                               
                    top_k: 256                           
         update_html_freq: 1000                          
                  verbose: False                         
----------------- End -------------------
dataset [AlignedFeatureMultiDataset] was created
The number of training images = 298
initialize network with normal
initialize network with normal
model [MemorySeqModel] was created
loading the model from ./checkpoints/memory_seq_p2p/0_net_G.pth
loading the model from ./checkpoints/memory_seq_p2p/0_net_D.pth
loading the model from ./checkpoints/memory_seq_p2p/0_net_mem.pth
---------- Networks initialized -------------
[Network G] Total number of parameters : 259.056 M
[Network D] Total number of parameters : 2.775 M
[Network mem] Total number of parameters : 11.952 M
-----------------------------------------------
Setting up a new session...
create web directory ./checkpoints/memory_seq_p2p/11/web...
Traceback (most recent call last):
  File "train.py", line 45, in <module>
    for i, data in enumerate(dataset):  # inner loop within one epoch
  File "/content/Audio-driven-TalkingFace-HeadPose/render-to-video/data/__init__.py", line 90, in __iter__
    for i, data in enumerate(self.dataloader):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 856, in _next_data
    return self._process_data(data)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 881, in _process_data
    data.reraise()
  File "/usr/local/lib/python3.6/dist-packages/torch/_utils.py", line 394, in reraise
    raise self.exc_type(msg)
FileNotFoundError: Caught FileNotFoundError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/content/Audio-driven-TalkingFace-HeadPose/render-to-video/data/aligned_feature_multi_dataset.py", line 54, in __getitem__
    A = Image.open(AB_path).convert('RGB')
  File "/usr/local/lib/python3.6/dist-packages/PIL/Image.py", line 2809, in open
    fp = builtins.open(filename, "rb")
FileNotFoundError: [Errno 2] No such file or directory: '/content/Audio-driven-TalkingFace-HeadPose/Deep3DFaceReconstruction/output/render/19_news/11/bm/frame112_renderold_bm.png'

----------------- Options ---------------
                       Nw: 3                             
                    alpha: 0.3                           
             aspect_ratio: 1.0                           
                attention: 1                             
               batch_size: 1                             
             blinkframeid: 41                            
          checkpoints_dir: ./checkpoints                 
                crop_size: 256                           
                 dataroot: 11_bmold_win3                    [default: None]
             dataset_mode: aligned_feature_multi         
                direction: AtoB                          
          display_winsize: 256                           
         do_saturate_mask: False                         
                    epoch: 60                               [default: latest]
                     eval: False                         
                  gpu_ids: 0                             
            iden_feat_dim: 512                           
            iden_feat_dir: arcface/iden_feat/            
               iden_thres: 0.98                          
              imagefolder: images60                         [default: images]
                init_gain: 0.02                          
                init_type: normal                        
                 input_nc: 3                             
                  isTrain: False                            [default: None]
                load_iter: 0                                [default: 0]
                load_size: 256                           
         max_dataset_size: inf                           
                 mem_size: 30000                         
                    model: memory_seq                       [default: test]
                        n: 26                            
               n_layers_D: 3                             
                     name: memory_seq_p2p/11                [default: experiment_name]
                      ndf: 64                            
                     netD: basic                         
                     netG: unetac_adain_256              
                      ngf: 64                            
               no_dropout: False                         
                  no_flip: False                         
                     norm: batch                         
                    ntest: inf                           
                 num_test: 200                              [default: 50]
              num_threads: 4                             
                output_nc: 3                             
                    phase: test                          
               preprocess: resize_and_crop               
             resizemethod: lanczos                       
              results_dir: ./results/                    
           serial_batches: False                         
         spatial_feat_dim: 512                           
                   suffix:                               
          test_batch_list:                               
              test_use_gt: 0                             
                    top_k: 256                           
                  verbose: False                         
----------------- End -------------------
dataset [AlignedFeatureMultiDataset] was created
initialize network with normal
model [MemorySeqModel] was created
loading the model from ./checkpoints/memory_seq_p2p/60_net_G.pth
Traceback (most recent call last):
  File "test.py", line 47, in <module>
    model.setup(opt)               # regular setup: load and print networks; create schedulers
  File "/content/Audio-driven-TalkingFace-HeadPose/render-to-video/models/base_model.py", line 89, in setup
    self.load_networks(load_suffix)
  File "/content/Audio-driven-TalkingFace-HeadPose/render-to-video/models/base_model.py", line 202, in load_networks
    state_dict = torch.load(load_path, map_location=str(self.device))
  File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 525, in load
    with _open_file_like(f, 'rb') as opened_file:
  File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 212, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 193, in __init__
    super(_open_file, self).__init__(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: './checkpoints/memory_seq_p2p/60_net_G.pth'
kryzhikov commented 4 years ago

Ok, I found out, that I need to install Matlab to perform training.