Testing the model on my data (source image and driving video) is so bad

[mohamedabdallah1996]

I tested the model on the data provided in the Colab notebook. it was really interesting. but when I tried to test on my own data, the result was so weird (especially when testing with a face turning in different direction (up, down, right or left))

so, what should I do to produce good results like what were produced from the original data (in the Colab notebook)?

Note: I run the same code on my data. and also I cropped the source image and the driving video using ffmpeg

[AliaksandrSiarohin]

I have no idea, since you did not provide your input.

[mohamedabdallah1996]

Thanks @AliaksandrSiarohin for your reply. this is an example of my input and output. what I mean that when the movement begins to be extreme towards any direction (up, down, right, left), the output video becomes distorted. and for me I want to apply dlib face points predictor on the generated video. and so such bad input to the predictor model will absolutely produce bad output!

[AliaksandrSiarohin]

The model is trained on interview video sequences, I doubd such a movement exist in a interview videos. So you have to train on video sequence that is similar to what you provided.

[mohamedabdallah1996]

yes, I need to train on my own dataset. but how can I use vox-256 as pretrained model and retrain on top of it instead of retraining from scratch?

[AliaksandrSiarohin]

Increase the number of ecpochs in the config and add --checkpoint vox-256.pth.tar to the train command.

[mohamedabdallah1996]

Hello @AliaksandrSiarohin I retrained on my dataset but unintentionally the training is interrupted. I want to resume training from the last saved checkpoint during the training. so, where can I find this checkpoint?

[AliaksandrSiarohin]

In log/..../00000##.pth.tar

[mohamedabdallah1996]

I trained additional 26 epochs on my dataset. this is the log.txt file

00000099) perceptual - 93.26425; equivariance_value - 0.15666; equivariance_jacobian - 0.49131
00000100) perceptual - 87.63938; equivariance_value - 0.13238; equivariance_jacobian - 0.47112
00000101) perceptual - 83.79310; equivariance_value - 0.12046; equivariance_jacobian - 0.46049
00000101) perceptual - 81.54766; equivariance_value - 0.11524; equivariance_jacobian - 0.45491
00000102) perceptual - 79.53074; equivariance_value - 0.11127; equivariance_jacobian - 0.44806
00000101) perceptual - 81.23047; equivariance_value - 0.11526; equivariance_jacobian - 0.45443
00000102) perceptual - 79.55930; equivariance_value - 0.11233; equivariance_jacobian - 0.45147
00000103) perceptual - 78.38695; equivariance_value - 0.11002; equivariance_jacobian - 0.44648
00000104) perceptual - 77.15685; equivariance_value - 0.10771; equivariance_jacobian - 0.44183
00000105) perceptual - 76.09751; equivariance_value - 0.10611; equivariance_jacobian - 0.43849
00000106) perceptual - 74.99386; equivariance_value - 0.10477; equivariance_jacobian - 0.43427
00000107) perceptual - 74.39798; equivariance_value - 0.10417; equivariance_jacobian - 0.43185
00000108) perceptual - 73.81180; equivariance_value - 0.10372; equivariance_jacobian - 0.42826
00000109) perceptual - 73.18208; equivariance_value - 0.10255; equivariance_jacobian - 0.42414
00000109) perceptual - 72.75357; equivariance_value - 0.10167; equivariance_jacobian - 0.42294
00000110) perceptual - 72.07644; equivariance_value - 0.10038; equivariance_jacobian - 0.41799
00000110) perceptual - 71.80728; equivariance_value - 0.10074; equivariance_jacobian - 0.41722
00000111) perceptual - 71.58586; equivariance_value - 0.09954; equivariance_jacobian - 0.41357
00000112) perceptual - 71.09346; equivariance_value - 0.09893; equivariance_jacobian - 0.41139
00000112) perceptual - 70.58865; equivariance_value - 0.09886; equivariance_jacobian - 0.41136
00000113) perceptual - 70.21467; equivariance_value - 0.09838; equivariance_jacobian - 0.40746
00000114) perceptual - 69.88168; equivariance_value - 0.09804; equivariance_jacobian - 0.40631
00000114) perceptual - 69.47350; equivariance_value - 0.09791; equivariance_jacobian - 0.40541
00000115) perceptual - 69.11671; equivariance_value - 0.09772; equivariance_jacobian - 0.40404
00000116) perceptual - 69.01225; equivariance_value - 0.09660; equivariance_jacobian - 0.40055
00000117) perceptual - 68.74825; equivariance_value - 0.09717; equivariance_jacobian - 0.40142
00000118) perceptual - 68.41641; equivariance_value - 0.09686; equivariance_jacobian - 0.39855
00000119) perceptual - 68.22659; equivariance_value - 0.09572; equivariance_jacobian - 0.39530
00000114) perceptual - 69.54724; equivariance_value - 0.09784; equivariance_jacobian - 0.40410
00000115) perceptual - 69.31381; equivariance_value - 0.09763; equivariance_jacobian - 0.40371
00000116) perceptual - 68.94456; equivariance_value - 0.09721; equivariance_jacobian - 0.40122
00000119) perceptual - 68.02745; equivariance_value - 0.09649; equivariance_jacobian - 0.39521
00000120) perceptual - 67.89172; equivariance_value - 0.09579; equivariance_jacobian - 0.39486
00000121) perceptual - 67.62985; equivariance_value - 0.09573; equivariance_jacobian - 0.39322
00000122) perceptual - 67.35661; equivariance_value - 0.09500; equivariance_jacobian - 0.39155
00000122) perceptual - 67.18710; equivariance_value - 0.09509; equivariance_jacobian - 0.39134
00000123) perceptual - 66.93566; equivariance_value - 0.09507; equivariance_jacobian - 0.38965
00000124) perceptual - 66.66692; equivariance_value - 0.09477; equivariance_jacobian - 0.38931
00000124) perceptual - 66.70306; equivariance_value - 0.09507; equivariance_jacobian - 0.38843
00000125) perceptual - 66.32366; equivariance_value - 0.09424; equivariance_jacobian - 0.38589
00000126) perceptual - 66.27177; equivariance_value - 0.09482; equivariance_jacobian - 0.38721

But I don't know exactly what does these values mean? so, please how can I know at some point that this number of epochs is sufficient for training? and how can I track the progress of the model from the generated images in train_vis folder?

[mohamedabdallah1996]

@AliaksandrSiarohin please any help would be appreciated.

[AliaksandrSiarohin]

I guess you just have to run the demo with your ckp and see if you get satisfactory quality. Usually I stop training when first value perceptual no longer decrease.

[chloejihye]

Increase the number of ecpochs in the config and add --checkpoint vox-256.pth.tar to the train command.

Hello! @AliaksandrSiarohin I want to train my own dataset on top of vox checkpoint as well. In this case, do you mean to increase the number of ecpochs of vox datatset ( at vox-256.yaml ) ?

I created new directory in dataset, saved all my 256x256 size mp4 videos, and created config file. After that I ran command python run.py --checkpoint checkpoints/vox-cpk.pth.tar --config config/mydataset-256.yaml --device_ids 0

It was working when I ran the command without adding checkpoint argument, but after adding checkpoints, I encountered errors as below. Could you let me know if something is wrong with my way?

`(firstmodel-animation) C:\Users\Administrator\Desktop\animation\first-order-model>python run.py --checkpoint checkpoints/vox-cpk.pth.tar --config config/idol-256.yaml --device_ids 0 run.py:40: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. config = yaml.load(f) Use predefined train-test split. Training... 0%| | 0/1 [00:00<?, ?it/s]C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 16-45-18-536_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 18-41-46-185_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-35-14-009_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-59-27-185_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-57-21-904_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 15-44-30-583.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-23-42-233_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-35-14-009_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-35-14-009_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-30-36-408_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-47-59-449_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 16-08-27-415_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 19-04-43-354_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 18-15-33-105_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 15-44-30-583.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 19-04-43-354_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-52-29-440_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-59-27-185_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 19-04-43-354_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 18-15-33-105_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 18-55-22-034_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 18-15-33-105_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-23-42-233_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-35-14-009_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 18-54-20-281_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 19-06-25-441_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-47-59-449_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 15-44-30-583.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-35-14-009_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 15-44-30-583.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 16-43-49-107_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-57-21-904_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 18-15-33-105_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-48-37-873_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 18-55-22-034_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 18-15-33-105_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-48-37-873_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 16-45-18-536_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-47-59-449_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 18-41-46-185_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 19-06-25-441_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 16-08-27-415_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 18-54-20-281_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 18-41-46-185_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 19-06-25-441_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-23-42-233_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-30-36-408_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 19-04-43-354_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-59-27-185_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 16-08-27-415_Trim.mp4. warn('Could not read last frame of %s.' % uri)
((deleted some)) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 16-08-27-415_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\torch\nn\functional.py:2423: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details. "See the documentation of nn.Upsample for details.".format(mode)) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-35-14-009_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-35-14-009_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 18-15-33-105_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 18-54-20-281_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-57-21-904_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 16-45-18-536_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-35-14-009_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 15-51-22-554_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 19-04-43-354_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-59-27-185_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-52-29-440_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 16-43-49-107_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 18-55-22-034_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 15-51-22-554_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-59-27-185_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 19-05-51-611_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-23-42-233_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 18-55-22-034_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-48-37-873_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 18-54-20-281_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-57-21-904_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 18-41-46-185_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 15-44-30-583.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-52-29-440_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 18-55-22-034_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-30-36-408_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 16-08-27-415_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-59-27-185_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 19-06-25-441_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 18-54-20-281_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 18-41-46-185_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-59-27-185_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-48-37-873_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 16-43-49-107_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 15-51-22-554_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 19-06-25-441_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 19-04-43-354_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-23-42-233_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 18-54-20-281_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 16-43-49-107_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 18-41-46-185_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 19-06-25-441_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 18-41-46-185_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 19-04-43-354_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 16-43-49-107_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 18-41-46-185_Trim.mp4. warn('Could not read last frame of %s.' % uri) C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\imageio\core\format.py:403: UserWarning: Could not read last frame of C:\Users\Administrator\Desktop\animation\first-order-model\data\idol\train\bandicam 2021-01-08 17-23-42-233_Trim.mp4. warn('Could not read last frame of %s.' % uri) Traceback (most recent call last): File "run.py", line 81, in train(config, generator, discriminator, kp_detector, opt.checkpoint, log_dir, dataset, opt.device_ids) File "C:\Users\Administrator\Desktop\animation\first-order-model\train.py", line 51, in train losses_generator, generated = generator_full(x) File "C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\torch\nn\modules\module.py", line 489, in call result = self.forward(*input, kwargs) File "C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\torch\nn\parallel\data_parallel.py", line 141, in forward return self.module(*inputs[0], *kwargs[0]) File "C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\torch\nn\modules\module.py", line 489, in call result = self.forward(input, kwargs) File "C:\Users\Administrator\Desktop\animation\first-order-model\modules\model.py", line 155, in forward generated = self.generator(x['source'], kp_source=kp_source, kp_driving=kp_driving) File "C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\torch\nn\modules\module.py", line 489, in call result = self.forward(*input, kwargs) File "C:\Users\Administrator\Desktop\animation\first-order-model\modules\generator.py", line 89, in forward out = self.bottleneck(out) File "C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\torch\nn\modules\module.py", line 489, in call result = self.forward(*input, *kwargs) File "C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\torch\nn\modules\container.py", line 92, in forward input = module(input) File "C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\torch\nn\modules\module.py", line 489, in call result = self.forward(input, kwargs) File "C:\Users\Administrator\Desktop\animation\first-order-model\modules\util.py", line 72, in forward out = self.conv2(out) File "C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\torch\nn\modules\module.py", line 489, in call result = self.forward(*input, **kwargs) File "C:\ProgramData\Anaconda3\envs\firstmodel-animation\lib\site-packages\torch\nn\modules\conv.py", line 320, in forward self.padding, self.dilation, self.groups) RuntimeError: CUDA out of memory. Tried to allocate 160.00 MiB (GPU 0; 15.88 GiB total capacity; 14.03 GiB already allocated; 125.44 MiB free; 659.34 MiB cached)`

[AliaksandrSiarohin]

You out of memory. Did you resize all your videos to 256x256? How many gpus you are using?

[chloejihye]

You out of memory. Did you resize all your videos to 256x256? How many gpus you are using?

Thank you for your quick response. @AliaksandrSiarohin I am using one GPU - Tesla v100 and managed to run after saving all videos in png formats and reduce batch size. It took me 4 hours to finish training using vox checkpoints and my new dataset which is included with 50 videos in all png format + 150 epochs with 10 batch sizes.

Now, I encountered another problem. I can't find any log files in my log folder. I could see that the new directory has made but the folder is empty. I'm training in Window 10. It was working just fine when I trained it with Google colab.... What would be the potential problems you can guess...?

[AliaksandrSiarohin]

Log is in the same folder as the vox checkpoint.

[chloejihye]

@AliaksandrSiarohin Ah, I see. I found it. Thank you so much!

[ghost]

Hey @mohamedabdallah1996 , @chloejihye were you able to successfully train your datasets on top of the existing ones? Do you think you could share the resulting checkpoints? I want to animate artbreeder characters, but I'm encountering the same problem with face distortion when the head turns sideways

[Fishyy599]

Must all epoch training be completed before the checkpoints file is generated? The default epoch is 100. I still don’t see the checkpoints file after more than ten generations of training.