Video flickers severely on my own data

[WyHy]

Hi，Thank you very much for your great work. it's really awesome！

I successfully ran the entire project on my own dataset, but the generated results seem to flicker much more severely compared to the example data. Is there a way to stabilize the results like the examples do? The tools I used are as follows:

0x1. To ensure the stability of the running results, I deliberately resized the ref_image and motion_data to the same dimensions. The ref_image was regenerated based on the pose from the motion data. 0x2. I obtained complete motion data based on the project at https://github.com/kijai/ComfyUI-champWrapper, and then imported the data into CHAMP to run the process: 1.1 DSINE Normal Map to obtain normal data 1.2 DWpose Estimator to obtain dwpose data 1.3 Depth Anything to obtain depth data 1.4 DensePose Estimator to obtain semantic_map data 0x3. The motion data and ref image I used are attached. Thank you very much.

data.zip

[Leoooo333]

Hi there, thanks for your detailed description. It's a great ComfyUI Wrapper for Champ. But for the Semantic_map, we actually use a different ColorMap and different part segmentation. This "gap" may introduce severe artifacts for diffusion models. Now we have released our SMPL & Rendering code, which can solve this problem. Let me know if it will fix the flickers👏

[WyHy]

Hi there, thanks for your detailed description. It's a great ComfyUI Wrapper for Champ. But for the Semantic_map, we actually use a different ColorMap and different part segmentation. This "gap" may introduce severe artifacts for diffusion models. Now we have released our SMPL & Rendering code, which can solve this problem. Let me know if it will fix the flickers👏

Thank you very much for your response, but I couldn't find the code related to SMPL & Rendering in this project. However, I found this link in another issue: Champ GitHub Repository. Following the instructions of this project, I completed the Fit SMPL and Transfer SMPL steps. But when I was processing Rendering with Blender, I encountered an error:

1. I executed the following command under the 4D-Humans directory:

   blender smpl_rendering.blend --background --python rendering.py --driving_path ../transfer_result/smpl_results --reference_path ../reference_imgs/images/ref.png --device 0

2. The version of Blender I used is the latest one: Blender 4.1.0 (hash 40a5e739e270 built 2024-03-26 00:31:19).

3. The error output is as follows:

   Blender 4.1.0 (hash 40a5e739e270 built 2024-03-26 00:31:19)
   Read blend: "/data/wangy/workspaces/future_dev/champv2/4D-Humans/smpl_rendering.blend"
   Traceback (most recent call last):
   File "/data/wangy/workspaces/future_dev/champv2/4D-Humans/rendering.py", line 3, in <module>
    import pandas as pd
   ModuleNotFoundError: No module named 'pandas'
   unknown argument, loading as file: --driving_path
   Error: Cannot read file "/data/wangy/workspaces/future_dev/champv2/4D-Humans/--driving_path": No such file or directory
   
   Blender quit

   The error message here indicates that pandas is not installed, but in fact, I already have pandas installed in my local virtual environment. Moreover, I have configured all the other parameter paths according to the documentation, but I have not been able to identify where the issue lies. Since I am not very familiar with Blender, could you please help me identify where the problem is?

[WyHy]

Hi there, thanks for your detailed description. It's a great ComfyUI Wrapper for Champ. But for the Semantic_map, we actually use a different ColorMap and different part segmentation. This "gap" may introduce severe artifacts for diffusion models. Now we have released our SMPL & Rendering code, which can solve this problem. Let me know if it will fix the flickers👏

Additionally, I found a potential error in the file transfer_smpl.py at line 60, where it states:

smooth_smpl_path = os.path.join(driving_folder, "smpl_results", "smpls_group.npz")

The actual path should be:

smooth_smpl_path = os.path.join(driving_folder, "smpls_group.npz")

After the Fit SMPL step, the smpls_group.npz file and the smpl_results folder are at the same level of hierarchy.

[Leoooo333]

@WyHy Thanks! I have just fix it in my fork/master. We tested with Blender 3.x. To install the pandas in blender, this may be helpful to you!

[WyHy]

@WyHy Thanks! I have just fix it in my fork/master. We tested with Blender 3.x. To install the pandas in blender, this may be helpful to you!

Thank you for your prompt response. The third step "Rendering" has been successfully executed. I downgraded Blender to version 3.5 and installed additional dependency packages within Blender, and now it runs successfully.

However, I feel it's a bit slow, and the GPU utilization is not high either. Are there any other acceleration plans or solutions? Is multi-GPU support available?

[lxwkobe24]

谢谢！我刚刚在我的叉子/主叉中修复了它。我们使用 Blender 3.x 进行了测试。要在搅拌机中安装熊猫，这可能对您有所帮助！

感谢您的及时回复。第三步“渲染”已成功执行。我将 Blender 降级到 3.5 版，并在 Blender 中安装了额外的依赖包，现在它成功运行。

但是，我觉得它有点慢，GPU利用率也不高。还有其他加速计划或解决方案吗？是否支持多 GPU？

please tell me how to change blender to 3.5 in linux 。thanks

[WyHy]

@lxwkobe24 I installed Blender using the Snap tool, and the official documentation can be found at: Installing on Linux - Blender 4.1 Manual. To install a specific version, you can refer to the following steps:

1. Uninstall the existing version.
2. Install the specified version using the following command:

   sudo snap install blender --channel=3.5/stable --classic

Please note that the command provided is an example and the channel name 3.5/stable should be replaced with the desired version's channel if it's available on Snap.

[lxwkobe24]

@lxwkobe24 I installed Blender using the Snap tool, and the official documentation can be found at: Installing on Linux - Blender 4.1 Manual. To install a specific version, you can refer to the following steps:

1. Uninstall the existing version.
2. Install the specified version using the following command:

   sudo snap install blender --channel=3.5/stable --classic

Please note that the command provided is an example and the channel name should be replaced with the desired version's channel if it's available on Snap.3.5/stable

i try , look this error . (error: cannot add authorization: open /home/AIlab/.snap/auth.json: permission denied) how to solve this problem.thanks you reply。

[WyHy]

@lxwkobe24 I installed Blender using the Snap tool, and the official documentation can be found at: Installing on Linux - Blender 4.1 Manual. To install a specific version, you can refer to the following steps:

1. Uninstall the existing version.
2. Install the specified version using the following command:

   sudo snap install blender --channel=3.5/stable --classic

Please note that the command provided is an example and the channel name should be replaced with the desired version's channel if it's available on Snap.3.5/stable

i try , look this error . (error: cannot add authorization: open /home/AIlab/.snap/auth.json: permission denied) how to solve this problem.thanks you reply。

This appears to be a permission error; ensure that you have sufficient privileges to access the auth.json file.

[lxwkobe24]

@lxwkobe24 I installed Blender using the Snap tool, and the official documentation can be found at: Installing on Linux - Blender 4.1 Manual. To install a specific version, you can refer to the following steps:

1. Uninstall the existing version.
2. Install the specified version using the following command:

   sudo snap install blender --channel=3.5/stable --classic

Please note that the command provided is an example and the channel name should be replaced with the desired version's channel if it's available on Snap.3.5/stable

i try , look this error . (error: cannot add authorization: open /home/AIlab/.snap/auth.json: permission denied) how to solve this problem.thanks you reply。

This appears to be a permission error; ensure that you have sufficient privileges to access the file.auth.json

it's very strange！ i use chmod change the permission, but i don't have this document with auth.json! could you tell me how to solve? thank you so much

[Leoooo333]

@WyHy Yeah this blender script is a little bit slow, even with GPU. I am wondering if that's caused by slow I/O stream. We met the same problems when we rendered through the whole dataset. What's the resolution of your video? I suggest to downscale it to around 500.

[WyHy]

@WyHy Yeah this blender script is a little bit slow, even with GPU. I am wondering if that's caused by slow I/O stream. We met the same problems when we rendered through the whole dataset. What's the resolution of your video? I suggest to downscale it to around 500.

I am using a landscape video with a resolution of 2084x1080. Once this run is finished, I will try to see if reducing the resolution can improve the processing speed. I am very much looking forward to seeing whether the use of SMPL and Blender effectively improves the flickering issue in the video. 😊

[WyHy]

it's very strange！ i use chmod change the permission, but i don't have this document with auth.json! could you tell me how to solve? thank you so much

I apologize, but I am unable to assist with this issue as I am also new to Blender. I suggest you visit the official forums to seek a solution.

[lxwkobe24]

这很奇怪！ 我使用 chmod 更改权限，但我没有这个文档auth.json！你能告诉我怎么解决吗？非常感谢

我很抱歉，但我无法协助解决这个问题，因为我也是 Blender 的新手。我建议您访问官方论坛以寻求解决方案。 i have download blender3.5 successfull. hello ,please tell me how to start blender .thanks

[WyHy]

@Leoooo333 I have completed all the processes on my own dataset. The flickering in the video is much better than before, but now two new issues have arisen:

1. I feel as though there might be a frame drop issue, as the characters appear to be jittery. I'm not sure what's causing this. I've inspected the data of the processed video frames individually, and it seems like when played in sequence, it should be very smooth, Where can this be adjusted?

2. The video clarity has decreased. It seems to be significantly different in quality compared to the frame images in the 'transfer_result' subfolder. I'm not sure what the reason for this is. Is it because my original resolution was 2084x1080 and the processing program compressed it? Where can this be adjusted?

https://github.com/fudan-generative-vision/champ/assets/13045015/ba3fbd4e-5141-4db3-bc84-19fdb7cf7c60

[WyHy]

这很奇怪！ 我使用 chmod 更改权限，但我没有这个文档auth.json！你能告诉我怎么解决吗？非常感谢

我很抱歉，但我无法协助解决这个问题，因为我也是 Blender 的新手。我建议您访问官方论坛以寻求解决方案。 i have download blender3.5 successfull. hello ,please tell me how to start blender .thanks

try this https://github.com/Leoooo333/champ/tree/master

[Leoooo333]

@Leoooo333 I have completed all the processes on my own dataset. The flickering in the video is much better than before, but now two new issues have arisen:

1. I feel as though there might be a frame drop issue, as the characters appear to be jittery. I'm not sure what's causing this. I've inspected the data of the processed video frames individually, and it seems like when played in sequence, it should be very smooth, Where can this be adjusted?

2. The video clarity has decreased. It seems to be significantly different in quality compared to the frame images in the 'transfer_result' subfolder. I'm not sure what the reason for this is. Is it because my original resolution was 2084x1080 and the processing program compressed it? Where can this be adjusted?

   animation.23.06.36.mp4

1. It's caused by SMPL inconsistency between frames. The SMPL Smooth tutorial is coming soon!
2. Actually we will resize the input video to align with our neural network, the output will then be interpolated to the original resolution. That may cause the blur.

For better results, I recommend using a half-body portrait with a front face as the reference image. We are going to solve the face problem(like the one shown in your full-body example) in the following version. Keep an eye on Champ🤸‍♂️

[lxwkobe24]

这很奇怪！ 我使用 chmod 更改权限，但我没有这个文档auth.json！你能告诉我怎么解决吗？非常感谢

我很抱歉，但我无法协助解决这个问题，因为我也是 Blender 的新手。我建议您访问官方论坛以寻求解决方案。我已经成功下载Blender3.5。您好，请告诉我如何启动Blender.thanks谢谢

试试这个 https://github.com/Leoooo333/champ/tree/master

please tell me how to download pandas in blender ? thanks.

[2502128021]

@Leoooo333 I have completed all the processes on my own dataset. The flickering in the video is much better than before, but now two new issues have arisen:

1. I feel as though there might be a frame drop issue, as the characters appear to be jittery. I'm not sure what's causing this. I've inspected the data of the processed video frames individually, and it seems like when played in sequence, it should be very smooth, Where can this be adjusted?

2. The video clarity has decreased. It seems to be significantly different in quality compared to the frame images in the 'transfer_result' subfolder. I'm not sure what the reason for this is. Is it because my original resolution was 2084x1080 and the processing program compressed it? Where can this be adjusted?

animation.23.06.36.mp4

Hello，I‘m still confused what caused the flickers in the background? Do you know the reasons and how to solve it?

[xusongxuliangwangsulong]

I used blender3.5, but I still got the same problem. @WyHy

[WyHy]

这很奇怪！ 我使用 chmod 更改权限，但我没有这个文档auth.json！你能告诉我怎么解决吗？非常感谢

我很抱歉，但我无法协助解决这个问题，因为我也是 Blender 的新手。我建议您访问官方论坛以寻求解决方案。我已经成功下载Blender3.5。您好，请告诉我如何启动Blender.thanks谢谢

试试这个 https://github.com/Leoooo333/champ/tree/master

please tell me how to download pandas in blender ? thanks.

My solution was to write an installation script called 'blender_python.py', the content of which is as follows:

import sys
import subprocess

# Get the path to the Blender's Python executable
blender_python_executable = sys.executable

# Construct the command to install pandas using the system's pip
install_command = [blender_python_executable, "-m", "pip", "install", "tqdm"]

# Call the system's pip to install the package
subprocess.check_call(install_command)

Then, use the Blender command to execute this installation script:

blender --background --python blender_python.py

[WyHy]

I used blender3.5, but I still got the same problem. @WyHy

you need install tqdm in Blender, which operates in a separate Python environment. follow https://github.com/fudan-generative-vision/champ/issues/41#issuecomment-2049656596

[WyHy]

@Leoooo333 I have completed all the processes on my own dataset. The flickering in the video is much better than before, but now two new issues have arisen:

1. I feel as though there might be a frame drop issue, as the characters appear to be jittery. I'm not sure what's causing this. I've inspected the data of the processed video frames individually, and it seems like when played in sequence, it should be very smooth, Where can this be adjusted?

2. The video clarity has decreased. It seems to be significantly different in quality compared to the frame images in the 'transfer_result' subfolder. I'm not sure what the reason for this is. Is it because my original resolution was 2084x1080 and the processing program compressed it? Where can this be adjusted?

animation.23.06.36.mp4

Hello，I‘m still confused what caused the flickers in the background? Do you know the reasons and how to solve it?

see https://github.com/fudan-generative-vision/champ/issues/41#issuecomment-2045593731

[xusongxuliangwangsulong]

Love You! WyHy

[2502128021]

@Leoooo333 I have completed all the processes on my own dataset. The flickering in the video is much better than before, but now two new issues have arisen:

1. I feel as though there might be a frame drop issue, as the characters appear to be jittery. I'm not sure what's causing this. I've inspected the data of the processed video frames individually, and it seems like when played in sequence, it should be very smooth, Where can this be adjusted?

2. The video clarity has decreased. It seems to be significantly different in quality compared to the frame images in the 'transfer_result' subfolder. I'm not sure what the reason for this is. Is it because my original resolution was 2084x1080 and the processing program compressed it? Where can this be adjusted?

animation.23.06.36.mp4

Hello，I‘m still confused what caused the flickers in the background? Do you know the reasons and how to solve it?

see #41 (comment)

I'm confused about the flickers in the background, it is reasonable that bugs in smpl module cause the jitter of foreground(human), but the flickers in the background are also caused by this ?