smplify-x sometimes fails and sometimes doesn't fail on certain images

[monacv]

So, when I run the fit code in batch among all test images of hico-det, it fails for HICO_test2015_00001000.jpg image however, if I run it separately on it, it doesn't. I am very confused by what is happening and how to fix it.

(smplifyx) [jalal@goku smplify-x]$ ./djrn_fit_fail_test.sh
Processing: ../../data/smplify-x/djrn_test_data_00001000/images/HICO_test2015_00001000.jpg
Found Trained Model: ../../data/smplify-x/vposer_v1_0/snapshots/TR00_E096.pt
/scratch3/research/code/smplify-x/smplifyx/optimizers/lbfgs_ls.py:238: UserWarning: This overload of add_ is deprecated:
    add_(Number alpha, Tensor other)
Consider using one of the following signatures instead:
    add_(Tensor other, *, Number alpha) (Triggered internally at  /pytorch/torch/csrc/utils/python_arg_parser.cpp:1005.)
  p.data.add_(step_size, update[offset:offset + numel].view_as(p.data))
Camera initialization done after 0.9238
Camera initialization final loss 575.2435
Stage 000 done after 2.8637 seconds                                                                                                
Stage 001 done after 0.4564 seconds                                                                                                
Stage 002 done after 6.1315 seconds                                                                                                
Stage 003 done after 18.8520 seconds                                                                                               
Stage 004 done after 25.0695 seconds                                                                                               
Stage: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:53<00:00, 10.68s/it]
Body fitting Orientation 0 done after 53.3851 seconds                                                                              
Body final loss val = 2082.48096                                                                                                   
Orientation:  50%|█████████████████████████████████████████▌                                         | 1/2 [00:53<00:53, 53.39s/it]/scratch3/venv/smplifyx/lib/python3.8/site-packages/smplx/body_models.py:271: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  param[:] = torch.tensor(params_dict[param_name])
Stage 000 done after 4.2677 seconds                                                                                                
Stage 001 done after 0.4241 seconds                                                                                                
Stage 002 done after 7.9165 seconds                                                                                                
Stage 003 done after 20.4528 seconds                                                                                               
Stage 004 done after 21.4049 seconds                                                                                               
Stage: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:54<00:00, 10.90s/it]
Body fitting Orientation 1 done after 54.4810 seconds                                                                              
Body final loss val = 2081.42847                                                                                                   
Orientation: 100%|███████████████████████████████████████████████████████████████████████████████████| 2/2 [01:47<00:00, 53.94s/it]
Found Trained Model: ../../data/smplify-x/vposer_v1_0/snapshots/TR00_E096.pt
Camera initialization done after 1.1827
Camera initialization final loss 3.5034
Stage 000 done after 1.4251 seconds                                                                                                
Stage 001 done after 0.2998 seconds                                                                                                
Stage 002 done after 0.7215 seconds                                                                                                
Stage 003 done after 5.4210 seconds                                                                                                
Stage 004 done after 4.5305 seconds                                                                                                
Stage: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:12<00:00,  2.48s/it]
Body fitting Orientation 0 done after 12.4083 seconds                                                                              
Body final loss val = 177.26837                                                                                                    
Orientation:  50%|█████████████████████████████████████████▌                                         | 1/2 [00:12<00:12, 12.41s/it]/scratch3/venv/smplifyx/lib/python3.8/site-packages/smplx/body_models.py:271: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  param[:] = torch.tensor(params_dict[param_name])
Stage 000 done after 4.9444 seconds                                                                                                
Stage 001 done after 0.4744 seconds                                                                                                
Stage 002 done after 1.0594 seconds                                                                                                
Stage 003 done after 6.2616 seconds                                                                                                
Stage 004 done after 0.9128 seconds                                                                                                
Stage: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:13<00:00,  2.73s/it]
Body fitting Orientation 1 done after 13.6635 seconds                                                                              
Body final loss val = 277.39865                                                                                                    
Orientation: 100%|███████████████████████████████████████████████████████████████████████████████████| 2/2 [00:26<00:00, 13.04s/it]
Found Trained Model: ../../data/smplify-x/vposer_v1_0/snapshots/TR00_E096.pt
Camera initialization done after 1.1597
Camera initialization final loss 4.8399
Stage 000 done after 0.4743 seconds                                                                                                
Stage 001 done after 0.4020 seconds                                                                                                
Stage 002 done after 1.4820 seconds                                                                                                
Stage 003 done after 8.3911 seconds                                                                                                
Stage 004 done after 5.7249 seconds                                                                                                
Stage: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:16<00:00,  3.30s/it]
Body fitting Orientation 0 done after 16.4828 seconds                                                                              
Body final loss val = 87.92973                                                                                                     
Stage 000 done after 1.3672 seconds                                                                                                
Stage 001 done after 0.3005 seconds                                                                                                
Stage 002 done after 1.9238 seconds                                                                                                
Stage 003 done after 4.8648 seconds                                                                                                
Stage 004 done after 8.3106 seconds                                                                                                
Stage: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:16<00:00,  3.36s/it]
Body fitting Orientation 1 done after 16.7777 seconds                                                                              
Body final loss val = 89.72486                                                                                                     
Orientation: 100%|███████████████████████████████████████████████████████████████████████████████████| 2/2 [00:33<00:00, 16.63s/it]
Processing the data took: 00 hours, 02 minutes, 55 seconds

Additionally, the results look like the following where in three folders, namely 000, 001, and 002 are created for meshes folder of results for this specific input image. I am not sure what they correspond to?

(smplifyx) [jalal@goku results_djrn_test_data_00001000]$ pwd
/scratch3/research/data/smplify-x/results_djrn_test_data_00001000
(smplifyx) [jalal@goku results_djrn_test_data_00001000]$ tree .
.
├── conf.yaml
├── images
│   └── HICO_test2015_00001000
│       ├── 000
│       ├── 001
│       └── 002
├── meshes
│   └── HICO_test2015_00001000
│       ├── 000.obj
│       ├── 001.obj
│       └── 002.obj
└── results
    └── HICO_test2015_00001000
        ├── 000.pkl
        ├── 001.pkl
        └── 002.pkl

9 directories, 7 files

Here's the image:

The script is:

(smplifyx) [jalal@goku smplify-x]$ pwd
/scratch3/research/code/smplify-x
(smplifyx) [jalal@goku smplify-x]$ cat djrn_fit_fail_test.sh 
python smplifyx/main.py --config cfg_files/fit_smplx.yaml --data_folder ../../data/smplify-x/djrn_test_data_00001000/ --output_folder ../../data/smplify-x/results_djrn_test_data_00001000 --visualize="False" --model_folder ../../data/smplify-x/models_smplx_v1_1/models/smplx/SMPLX_NEUTRAL.npz --vposer_ckpt ../../data/smplify-x/vposer_v1_0 --part_segm_fn ../../data/smplify-x/smplx_parts_segm.pkl

Previously, the same image failed with this error: "NaN loss value"

I get that one of the obj files shown in MeshLab is the lady holding a kite but what about the other two obj files? https://github.com/DirtyHarryLYL/DJ-RN/issues/50

[Anirudh257]

@monacv I faced a similar problem due to the presence of multiple persons in the image. The different object files that you have posted correspond to different people in the image. It depends on Openpose/2D pose estimates that you get from the image. You can try to use a bounding box to get a better estimate or try to ignore lesser confident pose estimates.

On running openpose on your image, I was able to plot 3 people. The person with the kite is having the highest confidence while the 2 other people in the background are poorly estimated.

[monacv]

@Anirudh257 thank you so much, do you know why the following folders have no output?

├── images
│   └── HICO_test2015_00001000
│       ├── 000
│       ├── 001
│       └── 002

[Anirudh257]

@monacv I also get these empty folders on running SMPL-X, I think that we are not giving the visualize variable and these folders might contain the overlayed mesh on the RGB image.

[itsanjukumar]

@monacv , @Anirudh257 Can you tell me from where you got 'smplx_parts_segm.pkl' file? For now I am using smplh/parts_segm.pkl from human_body_prior repository. And all I get is a black screen when I fit on an image. Have yoy come across such issue?

[Anirudh257]

Hi @itsanjukumar I think that I had gotten it from https://smpl-x.is.tue.mpg.de/