search

fuqifan / Region-Aware-Cycle-Loss

This repo is the official implementation of "Adaptive Multi-Modal Control of Digital Human Hand Synthesis using a Region-Aware Cycle Loss"
MIT License
1 stars 0 forks source link

About Region-Aware Cycle Loss #1

Open resindraburiza opened 1 month ago

resindraburiza commented 1 month ago

Hi,

I have read your paper titled Adaptive Multi-Modal Control of Digital Human Hand Synthesis Using a Region-Aware Cycle Loss and I have one question regarding the region-aware cycle loss.

In my understanding, the RACL is calculated based on keypoints difference between detected keypoint in input real image and its generated image counterpart. Do you use clean fully denoised generated image to take the keypoints or is it an image which still has some noise in it (intermediate results)?
I think it takes quite some time to generate a fully denoised image. If fully denoised image is used during training to compare the keypoint, do you have any tricks to shorten the generation time during training? If intermediate imagew were used, which intermediate timestep did you use? How well DWPose detector works on intermediate images?

Thank you very much. Looking forward to your reply.

fuqifan commented 1 month ago

Hi,

Thanks for your questions.

We tried to add RACL at each step of denoising. But as you are concerned, in the early stages of model training and in the first few steps of denoising, because of the poor quality of the generated images, the keypoints detected by DWPose are very inaccurate, which interferes with the model training. To determine which step's denoising result is worth calculating RACL, we set hand area thresholds to filter the denoising images at each step. Specifically, we put the denoising images of each step into DWPose and calculate the area of the hand region based on the keypoints predicted by DWPose. RACL is computed only for those denoising results whose hands area values are within the threshold values.

resindraburiza commented 1 month ago

Thank you very much for your reply!

I understand it now. I have an additional question regarding the RACL. In the paper, it is written that RACL is used as a weight to the original diffusion model loss. What happend if there is no keypoint detected at all? It will leads to RACL being zero, hence the total loss will be zero as well. Is the RACL not used when there is no detected keypoint?

Again, thank you very much!