search

MooreThreads / Moore-AnimateAnyone

Character Animation (AnimateAnyone, Face Reenactment)
Apache License 2.0
3.14k stars 245 forks source link

Enhancing the Fidelity of Generated Animations in Moore-AnimateAnyone #29

Open yihong1120 opened 9 months ago

yihong1120 commented 9 months ago

Dear Moore-AnimateAnyone Contributors,

I hope this message finds you well. I have been thoroughly exploring the capabilities of the Moore-AnimateAnyone repository and am deeply impressed by the strides made in animating still images with such remarkable results. The demo hosted on HuggingFace Spaces is particularly indicative of the potential this technology holds.

However, upon delving into the examples provided and running my own tests, I have observed certain limitations that I believe, if addressed, could significantly elevate the quality of the animations produced. I would like to propose a few enhancements that could potentially mitigate these issues and refine the overall animation process.

  1. Background Artifacts: The presence of artifacts in animations, especially when the reference image has a clean background, can be quite distracting. Could we consider implementing a more robust background detection and preservation algorithm to maintain the integrity of the original image?

  2. Scale Mismatch: The suboptimal results due to scale mismatch between the reference image and keypoints are noticeable. While the paper suggests preprocessing techniques, their implementation is not yet apparent in the current version. Could we prioritise the integration of these preprocessing techniques to improve the handling of scale variations?

  3. Motion Subtleties: The flickering and jittering in animations with subtle motions or static scenes detract from the fluidity of the animation. Would it be possible to introduce a smoothing mechanism or a motion threshold to ensure that only significant movements are translated into the animation sequence?

I understand that these enhancements may involve considerable research and development efforts, but I believe they could be instrumental in pushing the boundaries of what Moore-AnimateAnyone can achieve. Additionally, these improvements could be pivotal in the deployment of this technology on the MoBi MaLiang AIGC platform, ensuring a more polished and professional output for end-users.

I am keen to follow the progress of this project and am more than willing to contribute to discussions or testing, should you find my feedback of value.

Thank you for your dedication to this innovative project, and I look forward to your thoughts on the potential for these enhancements.

Best regards, yihong1120

G-force78 commented 9 months ago

The flickering could be down to poor interpolation and or a frame rate out of sync with the pose video. I think it would be best to lock the FPS to the pose video and then allow the user to choose the length in seconds, so long as it doesnt exceed the length of the pose video.

songtao-liu-mt commented 9 months ago

Hi, @yihong1120 , thanks a lot for your valuable suggestions. During the reproduction process, we indeed encountered the issues you mentioned. Through filtering and cleaning of the training data, as well as employing some training tricks (we will soon release the training code), we observed a mitigation of such issues.

Certainly, we are currently exploring some of our own optimization measures to further alleviate these issues. We will also consider the valuable suggestions you provided, and you are welcome to experiment with improvements once we release the training code. We look forward to further collaboration and discussions in the future.