Motion Reference Frame

[DrexelRose]

Is there a way to have the motion vectors be zeroed out on a certain frame other than the first frame? by default the source video remains unwarped on the first frame, then gets warped by the driving video on subsequent frames.

Say in this example that there will inevitably be heavy distortion, but If I could apply the animation referencing the driving video at around halfway (where the face is facing camera) then the warping when looking up and down may not be so harsh. So the source video would remain most unwarped around the halfway point, but distorted the most at the beginning and end. I've made a horrible example here using a different vector warping program:

[AliaksandrSiarohin]

Don't entirely understand what you mean. You can specify --best_frame frame_number or - - find_best_frame in demo.py (it will search for the closest pose). Then it will generate the video forward and backward from this frame.

[DrexelRose]

I think thats exactly what I was looking for, thank you!

[DrexelRose]

I'm using the colab, on the "perform image animation" section I swapped out a bit of code as such:

predictions = make_animation(source_image, driving_video, generator, kp_detector, relative=True)

i = 100 driving_forward = driving_video[i:] driving_backward = driving_video[:(i+1)][::-1] predictions_forward = make_animation(source_image, driving_forward, generator, kp_detector, relative=True) predictions_backward = make_animation(source_image, driving_backward, generator, kp_detector, relative=True) predictions = predictions_backward[::-1] + predictions_forward[1:]

Horrifying, but a better result that I was looking for, thank you.

[ZeroCool940711]

@AliaksandrSiarohin I have a question that im not sure if it has been asked before and its kinda related to this issue. Can we use a video or multiple images as reference to improve the results? that way we could provide multiple images with different face angles or a video showing multiple angles and have a better result, as there will be more images or more frames that will match the driving video frames.

[AliaksandrSiarohin]

This is possible, you can take different parts from different images. Using occlusion map you can understand which parts you can take from each image. For example you can take weighted average of representations of different sources here https://github.com/AliaksandrSiarohin/first-order-model/blob/af781e012b723684657b25745a9fd1608dbf7327/modules/generator.py#L84 However this most likely would require retraining.

[ZeroCool940711]

@AliaksandrSiarohin is there any chance that you could implement that and add a parameter so we can use it instead of --source_image or make it so the parameter is called --source and it could take a folder with images, a single image or a video as source? I would do it myself but I cant train on Windows yet, im trying to figure out how to fix the problems im getting so I can start training and start improving things, I want to see if I can create a simple GUI later so its easier to work with the code but first I have to be able to train.

[AliaksandrSiarohin]

@ZeroCool940711 now I'm busy with another paper of mine. So probably I don't have time and gpu resources in the near future to implement this feature. Sorry about this.

[ZeroCool940711]

I understand, I will see if I can manage to make the code work on Windows properly so I can retrain and try to do those things. You said you're working on a new paper, is it something similar to this one or something else completely different? btw, I have some constructive feedback, would be nice if you give the paper a better name or even a code with numbers with some letters would work, something that could be used to reference it when creating issues the repository or on a blog somewhere instead of just saying First Order Model or the code from this paper :)

[AliaksandrSiarohin]

Yes it is similar. Basically unsupervised segmentation for applications like face-swap. I don't really get your second point, please note I can not change the name of the paper since it is already published in NeurIPS.

[ZeroCool940711]

For what you said about the new paper it sounds interesting, can't wait to see it, and what I mean is that you should try to give the paper (or at least the repository name on Github) a better name that could be easily recognized by other people, for example, the DeepFakeLab repository has a name which can be easily recognized, no matter where you mentioned it if you google DeepFakeLab it will point to the Github repository for that project, what I mean is you should give your papers some name that is unique or at least something that is easy to find with a quick google search, that will help others find the repository with the code for the paper easily and also make it easier for people to create issues on the repository as they will have a name they can use to reference the code, project or paper.

[AliaksandrSiarohin]

Ok I see, I will ask my co-autors about it.

[Jefersonwulf]

I'm using the colab, on the "perform image animation" section I swapped out a bit of code as such:

How exactly did your code look after editing? Can you send a copy?

[DrexelRose]

Jefersonwulf, I grabbed this section from demo.py:

and then commented out the "prediction" line in the animate section of the colab, and pasted that bit of code, manually redefinining the frame I thought was best (variable 'i') and taking out some variables being called:

[eSaniello]

@AliaksandrSiarohin is there any chance that you could implement that and add a parameter so we can use it instead of --source_image or make it so the parameter is called --source and it could take a folder with images, a single image or a video as source? I would do it myself but I cant train on Windows yet, im trying to figure out how to fix the problems im getting so I can start training and start improving things, I want to see if I can create a simple GUI later so its easier to work with the code but first I have to be able to train.

Hi, if you manage to do this then please share or maybe create a pull request. Thanks

[Ease78]

@eSaniello I think a folder of images is doable and I will attempt it, but I can't visualize what would happen if the '--source_image' is also video.