search

sign-language-processing / pose-to-video

Render pose sequences as photorealistic videos.
8 stars 6 forks source link

Avatar customization possible? #6

Closed Tortoise17 closed 3 months ago

Tortoise17 commented 3 months ago

Dear @AmitMY Is it possible to replace the avatar with any other? as I have used in sign translation and I couldn't figure out if any other still avatar can be included on fly? or do I have to retrain engine completely with changed desired avatar? Please if you could guide.

AmitMY commented 3 months ago

For the pix2pix model, for example, you have to retrain. For the controlnet model, you can just give a text prompt (for example, "a black man"), but performance not gurenteed. For the animatediff post-processor, you can also specify the prompt.

In my opinion, for good quality, you should use controlnet as is, then, in the animatediff, specify a different person. If it doesn't change enough, you can give the animatediff more diffusion steps.

Tortoise17 commented 3 months ago

@AmitMY I tried to use this command pose_to_video --type=controlnet --model=sign/sd-controlnet-mediapipe --pose=original.pose --video=original-cnad.mp4 --processors animatediff and give text prompt to change the avatar. Still it didnt accept. Can you guide if the command or pipeline is different for that?

Tortoise17 commented 3 months ago

Is it possible to export video with only pose without any human ? using controlnet? if you can guide that way also.?

AmitMY commented 3 months ago

From https://github.com/sign-language-processing/pose, the following command will visualize the pose:

visualize_pose -i file.pose -o file.mp4

If you wanted to edit the prompt, you have to change it in the code, locally, then run this command. It was not designed to make the prompt changeable. One can add that if you find it useful with different prompts

Tortoise17 commented 3 months ago

Great help. Thank you. I will try to implement the prompt stuff if successful.