Wrong face color

[zealYOU]

this is the code from train_body.py

def image_process(img_pth):
    img = cv2.imread(img_pth)
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
    img = cv2.resize(img, (512,512))
    return Image.fromarray(img)

while I found the code from train_head.py is different，cv2.cvtColor is missing

def image_process(img_pth):
    img = cv2.imread(img_pth)
    img = cv2.resize(img, (256, 256))
    return Image.fromarray(img)

I think It‘s a bug because I finally found it because my generated face is blue. However, It's so fatal that I have to suspect that the entire project is running on the basis of this bug 😃. For example, I am worried that the pre-trained model was trained under this situation. Could you check it ?

[bone-11]

I checked the previous code, and it did contain that line. I might have accidentally deleted it while organizing the release version. Since it didn't affect the run of the code, I didn't notice it. Thank you, and I apologize for any misunderstanding I may have caused.

[vibhu-raturi]

This might not be an issue and something I am not able to do, when I run inference using my own image as head and body it still gives me the video of default image. How can I resolve it? Do I need to train for my images. How can i create mesh for training?

[bone-11]

As a personalized framework, our model should be fine-tuned for specific person. We utilize SHOW to extract the human 3D mesh for training, and the face mesh is cropped from the body mesh image by face aligment code under the inference folder.

Additionally, since reproducing the SHOW code may be a trouble, you might want to try 4D-humans if your goal is solely to obtain body meshes. While it is not as accurate as SHOW, it offers quicker implementation.

[vibhu-raturi]

Thank you for this quick reply, okay I will check. So my goal is to try and create my anchor which is audio driven, if an audio is given as input the final code gives me back the anchor video speaking the audio with lip sync and hand movements, I guess that is possible with this source code. When I create my training data and fine tune the model, how will it be audio driven, do I have to give mesh synced with audio as input in seth1 folder and then it will create the final video?

[bone-11]

We implemented audio-driven animation based on the TalkSHOW. Specifically, for the fine-tuned model, we use the TalkSHOW model to generate 3D mesh sequences driven by audio as the input pose, similar to what is shown in seth1.

[zealYOU]

Hello, I have another little question. Are the parameters of the denoise module frozen during pretraining or fine-tuning like ControlNet?

[vibhu-raturi]

Hello, so to create our own custom training dataset, how can we generate head, mask and mesh for the video. Do we just crop the images we have from the body folder or are there some python scripts to get these required details.

[bone-11]

Hello, I have another little question. Are the parameters of the denoise module frozen during pretraining or fine-tuning like ControlNet?

All parameters are fine-tuned. In my experiments, I found that appearance improves when all parameters are fine-tuned.

[bone-11]

Hello, so to create our own custom training dataset, how can we generate head, mask and mesh for the video. Do we just crop the images we have from the body folder or are there some python scripts to get these required details.

Yes, i extract the head conditions from the body folder. If you can wait a bit longer, I'll organize and upload the relevant code this week.

[vibhu-raturi]

Yes, i extract the head conditions from the body folder. If you can wait a bit longer, I'll organize and upload the relevant code this week.

That would be great! So by the time you upload preprocess code I will try till then. Thank you but please do try and upload the code this week, it would be really helpful

[bone-11]

Yes, i extract the head conditions from the body folder. If you can wait a bit longer, I'll organize and upload the relevant code this week.

That would be great! So by the time you upload preprocess code I will try till then. Thank you but please do try and upload the code this week, it would be really helpful

The preprocess is uploaded. If any problem, please contract me.