fail to inference and cannot find where to input the driven image

AWangji commented 1 year ago

hi, thanks for your excellent project! But when I try to inference your model with my video, it failed: I dont know why. And as you says in README: only input one video? So where is the driven images?

Wei-Chen-hub commented 1 year ago

As in README.md

Place the video for inference under SMPLer-X/demo/videos
Prepare the pretrained models to be used for inference under SMPLer-X/pretrained_models
Prepare the mmdet pretrained model and config under SMPLer-X/pretrained_models
Inference output will be saved in SMPLer-X/demo/results

Please check these above path, see if problem still exists.

HeptagramV commented 1 year ago

Hey, I'm encountering the same problem. I have a directory of images, but I'm not sure what content should be placed here. Based on my reading of the homepage, I only know that I need to provide video files

AWangji commented 1 year ago

As in README.md

Place the video for inference under SMPLer-X/demo/videos

Prepare the pretrained models to be used for inference under SMPLer-X/pretrained_models

Prepare the mmdet pretrained model and config under SMPLer-X/pretrained_models

Inference output will be saved in SMPLer-X/demo/results

Please check these above path, see if problem still exists.

Hi, thanks for your suggestion. I have resolved all questions above. Now it can inference successfully.

But I still doubt about how to transfer the action in the video to another target image. Could you give some advice?

AWangji commented 1 year ago

Hey, I'm encountering the same problem. I have a directory of images, but I'm not sure what content should be placed here. Based on my reading of the homepage, I only know that I need to provide video files

yeah same question. Have you solved it?

HeptagramV commented 1 year ago

Hey, I'm encountering the same problem. I have a directory of images, but I'm not sure what content should be placed here. Based on my reading of the homepage, I only know that I need to provide video files

yeah same question. Have you solved it?

Yeah, I have resolved the previous issue, but since I am using a Windows system, I encountered some new issues while running shell commands. Are you using a Linux system, and should I try again on the same system?

HeptagramV commented 1 year ago

Hey, I'm encountering the same problem. I have a directory of images, but I'm not sure what content should be placed here. Based on my reading of the homepage, I only know that I need to provide video files

yeah same question. Have you solved it?

Yeah, I have resolved the previous issue, but since I am using a Windows system, I encountered some new issues while running shell commands. Are you using a Linux system, and should I try again on the same system?

I encountered a problem while running the srun command, it seems that I did not configure my slurm correctly

AWangji commented 1 year ago

Hey, I'm encountering the same problem. I have a directory of images, but I'm not sure what content should be placed here. Based on my reading of the homepage, I only know that I need to provide video files

yeah same question. Have you solved it?

Yeah, I have resolved the previous issue, but since I am using a Windows system, I encountered some new issues while running shell commands. Are you using a Linux system, and should I try again on the same system?

I encountered a problem while running the srun command, it seems that I did not configure my slurm correctly

yes, so do I. I directly use python to execute the command instead. It also work.

AWangji commented 1 year ago

Hey, I'm encountering the same problem. I have a directory of images, but I'm not sure what content should be placed here. Based on my reading of the homepage, I only know that I need to provide video files

yeah same question. Have you solved it?

Yeah, I have resolved the previous issue, but since I am using a Windows system, I encountered some new issues while running shell commands. Are you using a Linux system, and should I try again on the same system?

I encountered a problem while running the srun command, it seems that I did not configure my slurm correctly

do you have any idea that we can input a image to rebuild the detected body in the inferenced result video？

matlab2017 commented 1 year ago

Hey, I'm encountering the same problem. I have a directory of images, but I'm not sure what content should be placed here. Based on my reading of the homepage, I only know that I need to provide video files

yeah same question. Have you solved it?

Yeah, I have resolved the previous issue, but since I am using a Windows system, I encountered some new issues while running shell commands. Are you using a Linux system, and should I try again on the same system?

hello i meet the same problem, How did you solve it

Wei-Chen-hub commented 1 year ago

Hi, i have noticed that usage of "slum_inference" could be troublesome, i will push a more straight forward script for inference. Before that, you could try the command from my personal pipeline as below:

cmd_smplerx_inference = f'cd smplerx/main && python inference.py ' \
            f'--num_gpus 1 --pretrained_model {args.ckpt} ' \
            f'--agora_benchmark agora_model ' \
            f'--img_path ../../{frame_path} --start {start_count} --end {end_count} ' \
            f'--output_folder ../../{args.save_dir}/{vid_name} ' \
            f'--show_verts --show_bbox '

"args.ckpt" is one of the models, e.g.'smpler_x_h32'
"--img_path" is where you save the split frames with name"%06d.jpg"
"--start" & "--end" refers to the frame number you wish to process
"--output_folder" is where the results save

And as for OS, we have been developing on Linux, Ubuntu18.04 & 20.04, for windows users I recommend WSL2 & Ubuntu 20.04, which is my environment and inference pipeline is well tested.

For projection of results, please refer to "2D Smplx Overlay" in "readme.md".

AWangji commented 1 year ago

Hi, i have noticed that usage of "slum_inference" could be troublesome, i will push a more straight forward script for inference. Before that, you could try the command from my personal pipeline as below:
cmd_smplerx_inference = f'cd smplerx/main && python inference.py ' \
            f'--num_gpus 1 --pretrained_model {args.ckpt} ' \
            f'--agora_benchmark agora_model ' \
            f'--img_path ../../{frame_path} --start {start_count} --end {end_count} ' \
            f'--output_folder ../../{args.save_dir}/{vid_name} ' \
            f'--show_verts --show_bbox '
"args.ckpt" is one of the models, e.g.'smpler_x_h32'

"--img_path" is where you save the split frames with name"%06d.jpg"

"--start" & "--end" refers to the frame number you wish to process

"--output_folder" is where the results save

And as for OS, we have been developing on Linux, Ubuntu18.04 & 20.04, for windows users I recommend WSL2 & Ubuntu 20.04, which is my environment and inference pipeline is well tested.

For projection of results, please refer to "2D Smplx Overlay" in "readme.md".

That is to say, you do not support make one image retargeted by a driving video, like this

AWangji commented 1 year ago

Hey, I'm encountering the same problem. I have a directory of images, but I'm not sure what content should be placed here. Based on my reading of the homepage, I only know that I need to provide video files

yeah same question. Have you solved it?

Yeah, I have resolved the previous issue, but since I am using a Windows system, I encountered some new issues while running shell commands. Are you using a Linux system, and should I try again on the same system?

I encountered a problem while running the srun command, it seems that I did not configure my slurm correctly

my python command is: python inference.py \ --num_gpus ${GPUS_PER_NODE} \ --expname output/demo${JOB_NAME} \ --pretrained_model ${CKPT} \ --agora_benchmark agora_model \ --img_path ${IMG_PATH} \ --start 1 \ --end $end_count \ --output_folder ${SAVE_DIR} \ --show_verts \ --show_bbox \ --save_mesh \ and delete all about "srun". It makes sense. you can try that

HeptagramV commented 1 year ago

嘿，我遇到了同样的问题。我有一个图像目录，但我不确定应该在这里放置什么内容。根据我对主页的阅读，我只知道我需要提供视频文件

是的，同样的问题。你解决了吗？

是的，我已经解决了上一个问题，但是由于我使用的是 Windows 系统，因此在运行 shell 命令时遇到了一些新问题。您使用的是 Linux 系统，我应该在同一系统上重试吗？

我在运行 srun 命令时遇到了问题，似乎我没有正确配置我的 slurm

我的python命令是： python inference.py --num_gpus ${GPUS_PER_NODE} --expname output/demo${JOB_NAME} --pretrained_model ${CKPT} --agora_benchmark agora_model --img_path ${IMG_PATH} --start 1 --end $end_count --output_folder ${SAVE_DIR} --show_verts --show_bbox --save_mesh 并删除所有关于“srun”的内容。这是有道理的。你可以试试

嗨，我注意到使用“slum_inference”可能会很麻烦，我将推送一个更直接的脚本进行推理。在此之前，您可以尝试我的个人管道中的命令，如下所示：
cmd_smplerx_inference = f'cd smplerx/main && python inference.py ' \
            f'--num_gpus 1 --pretrained_model {args.ckpt} ' \
            f'--agora_benchmark agora_model ' \
            f'--img_path ../../{frame_path} --start {start_count} --end {end_count} ' \
            f'--output_folder ../../{args.save_dir}/{vid_name} ' \
            f'--show_verts --show_bbox '
“args.ckpt”是其中一个模型，例如“smpler_x_h32”

“--img_path”是保存名称为“%06d.jpg”的拆分帧的位置

“--start”和“--end”是指您要处理的帧号

“--output_folder”是结果保存的位置

至于操作系统，我们一直在Linux、Ubuntu18.04和20.04上开发，对于Windows用户，我推荐WSL2和Ubuntu 20.04，这是我的环境，推理管道经过了很好的测试。

有关结果的投影，请参阅“readme.md”中的“2D Smplx Overlay”。

Thank you for your help. It can work now. I have successfully run the inference and I am trying to execute '2D Smplx Overlay', but it seems that my input has not been processed by render.py. Has anyone successfully run this step?

99bits commented 1 year ago

Thanks for this great project, we are trying to get this working in google collab and running into similar problem with path to the videos. Tried your personal pipeline but still not working.

Wei-Chen-hub commented 1 year ago

Hi @99bits , seems your problem is with mmcv, please use "mmcv-full" and version stated in the repo. Besides, I suggest you to search this error message under mmcv repo, seems plenty of people are struggling with mmcv in collab.

As for @HeptagramV , please check if thes image directory exists:

{SMPLERX INFERENCE DIR}/{VIDEO NAME (no extension)}/orig_img/%06d.jpg

And if all the output from inference are placed under:

{SMPLERX INFERENCE DIR}/{VIDEO NAME (no extension)}

Please refer to commands in README.md, the relateive path should be strictly followed.

ffmpeg -i {VIDEO_FILE} -f image2 -vf fps=30 \
        {SMPLERX INFERENCE DIR}/{VIDEO NAME (no extension)}/orig_img/%06d.jpg \
        -hide_banner  -loglevel error

cd main && python render.py \
            --data_path {SMPLERX INFERENCE DIR} --seq {VIDEO NAME} \
            --image_path {SMPLERX INFERENCE DIR}/{VIDEO NAME} \
            --render_biggest_person False

nasifuddin commented 1 year ago

I've been trying to run this pipeline on google colab. So far tried:

using the exact version of mmcv-full as repo
running the inference script which got me the same error as this post (check photo attachment)
tried running the inference.py which gave me "No module named 'mmcv._ext'" error
looked into the mmcv-full repo with this error message. Suggested solutions didn't work in colab.
Here is the link to the notepad: https://colab.research.google.com/drive/1nRcZOvKSJAUqictnILWtC1y23_xJ8uX_?usp=sharing

@Wei-Chen-hub can you please help me with this or suggest me a possible solution?

littleemo commented 1 year ago

hi, thanks for your excellent project! But when I try to inference your model with my video, it failed: I dont know why. And as you says in README: only input one video? So where is the driven images?

Hey, I'm encountering the same problem.it failed: I have checked README.md but it doesn`t work. I noticed that you have solved the problem, can you help me with this?

g-ch commented 1 year ago

Hi, i have noticed that usage of "slum_inference" could be troublesome, i will push a more straight forward script for inference. Before that, you could try the command from my personal pipeline as below:
cmd_smplerx_inference = f'cd smplerx/main && python inference.py ' \
            f'--num_gpus 1 --pretrained_model {args.ckpt} ' \
            f'--agora_benchmark agora_model ' \
            f'--img_path ../../{frame_path} --start {start_count} --end {end_count} ' \
            f'--output_folder ../../{args.save_dir}/{vid_name} ' \
            f'--show_verts --show_bbox '
"args.ckpt" is one of the models, e.g.'smpler_x_h32'

"--img_path" is where you save the split frames with name"%06d.jpg"

"--start" & "--end" refers to the frame number you wish to process

"--output_folder" is where the results save

And as for OS, we have been developing on Linux, Ubuntu18.04 & 20.04, for windows users I recommend WSL2 & Ubuntu 20.04, which is my environment and inference pipeline is well tested.

For projection of results, please refer to "2D Smplx Overlay" in "readme.md".

I tried to run the code in ubuntu 20.04 but I got an error with human model. Could anyone help me with this? Thank you in advance!

Command:

python inference.py --num_gpus 1 --exp_name output/demo --pretrained_model smpler_x_h32 --agora_benchmark agora_model --img_path "/home/cc/git/SMPLer-X/demo/images/video2" --start 1 --end 10 --output_folder "/home/cc/git/SMPLer-X" --show_verts --show_bbox

The error message is:

Traceback (most recent call last):
  File "inference.py", line 188, in <module>
    main()
  File "inference.py", line 55, in main
    from base import Demoer
  File "/home/cc/git/SMPLer-X/main/../common/base.py", line 11, in <module>
    from SMPLer_X import get_model
  File "/home/cc/git/SMPLer-X/main/../main/SMPLer_X.py", line 4, in <module>
    from nets.smpler_x import PositionNet, HandRotationNet, FaceRegressor, BoxNet, HandRoI, BodyRotationNet
  File "/home/cc/git/SMPLer-X/main/../common/nets/smpler_x.py", line 6, in <module>
    from utils.human_models import smpl_x
  File "/home/cc/git/SMPLer-X/main/../common/utils/human_models.py", line 175, in <module>
    smpl_x = SMPLX()
  File "/home/cc/git/SMPLer-X/main/../common/utils/human_models.py", line 11, in __init__
    self.layer = {'neutral': smplx.create(cfg.human_model_path, 'smplx', gender='NEUTRAL', use_pca=False, use_face_contour=True, **self.layer_arg),
  File "/home/cc/git/SMPLer-X/main/../common/utils/smplx/smplx/body_models.py", line 2331, in create
    raise ValueError(f'Unknown model type {model_type}, exiting!')
ValueError: Unknown model type human, exiting!

ziyuanding commented 1 year ago

Hi, i have noticed that usage of "slum_inference" could be troublesome, i will push a more straight forward script for inference. Before that, you could try the command from my personal pipeline as below:
cmd_smplerx_inference = f'cd smplerx/main && python inference.py ' \
            f'--num_gpus 1 --pretrained_model {args.ckpt} ' \
            f'--agora_benchmark agora_model ' \
            f'--img_path ../../{frame_path} --start {start_count} --end {end_count} ' \
            f'--output_folder ../../{args.save_dir}/{vid_name} ' \
            f'--show_verts --show_bbox '
"args.ckpt" is one of the models, e.g.'smpler_x_h32'

"--img_path" is where you save the split frames with name"%06d.jpg"

"--start" & "--end" refers to the frame number you wish to process

"--output_folder" is where the results save

And as for OS, we have been developing on Linux, Ubuntu18.04 & 20.04, for windows users I recommend WSL2 & Ubuntu 20.04, which is my environment and inference pipeline is well tested. For projection of results, please refer to "2D Smplx Overlay" in "readme.md".
I tried to run the code in ubuntu 20.04 but I got an error with human model. Could anyone help me with this? Thank you in advance!

Command:

python inference.py --num_gpus 1 --exp_name output/demo --pretrained_model smpler_x_h32 --agora_benchmark agora_model --img_path "/home/cc/git/SMPLer-X/demo/images/video2" --start 1 --end 10 --output_folder "/home/cc/git/SMPLer-X" --show_verts --show_bbox

The error message is:
Traceback (most recent call last):
  File "inference.py", line 188, in <module>
    main()
  File "inference.py", line 55, in main
    from base import Demoer
  File "/home/cc/git/SMPLer-X/main/../common/base.py", line 11, in <module>
    from SMPLer_X import get_model
  File "/home/cc/git/SMPLer-X/main/../main/SMPLer_X.py", line 4, in <module>
    from nets.smpler_x import PositionNet, HandRotationNet, FaceRegressor, BoxNet, HandRoI, BodyRotationNet
  File "/home/cc/git/SMPLer-X/main/../common/nets/smpler_x.py", line 6, in <module>
    from utils.human_models import smpl_x
  File "/home/cc/git/SMPLer-X/main/../common/utils/human_models.py", line 175, in <module>
    smpl_x = SMPLX()
  File "/home/cc/git/SMPLer-X/main/../common/utils/human_models.py", line 11, in __init__
    self.layer = {'neutral': smplx.create(cfg.human_model_path, 'smplx', gender='NEUTRAL', use_pca=False, use_face_contour=True, **self.layer_arg),
  File "/home/cc/git/SMPLer-X/main/../common/utils/smplx/smplx/body_models.py", line 2331, in create
    raise ValueError(f'Unknown model type {model_type}, exiting!')
ValueError: Unknown model type human, exiting!

Hi @g-ch ,

please check your file structure, especially this part:

if you don't set it correctly, this piece of code will try to guess your model type, then dir "human_model_files" will be evaluated as "human", boom! no such model type!

btw, all pretrained files and models could be here, I suggest you download them all and place them exactly like what is written in README.md.

ziyuanding commented 1 year ago

hi, thanks for your excellent project! But when I try to inference your model with my video, it failed: I dont know why. And as you says in README: only input one video? So where is the driven images?

Hey, I'm encountering the same problem.it failed: I have checked README.md but it doesn`t work. I noticed that you have solved the problem, can you help me with this?

Hi @littleemo ,

I have the same issue, but this could be ignored. Because even shell script failed, ffmpeg already generated a sequence of images for future inferernce. so you can start directly using python script from this step now, you can see how the python command works in this issue :)

if you are working with vscode, here is a launch.json that may help you https://github.com/zacida/SMPLer-X/blob/main/.vscode/launch.json

caizhongang / SMPLer-X

fail to inference and cannot find where to input the driven image #22