Inaccurate Results When Running Demo on Images

[MSutharsan]

I have successfully set up and run the demo for image as per the provided documentation. However, the output I'm getting is not accurate. Could you please assist in resolving this issue?

[saidwivedi]

I think there might be some issues with the renderer. Can you paste the output of pip list?

[MSutharsan]

absl-py 2.1.0 addict 2.4.0 aiohappyeyeballs 2.3.5 aiohttp 3.10.3 aiosignal 1.3.1 antlr4-python3-runtime 4.9.3 asttokens 2.4.1 async-timeout 4.0.3 attrs 24.2.0 black 24.8.0 blinker 1.8.2 body_visualizer 1.1.0 braceexpand 0.1.7 certifi 2022.12.7 charset-normalizer 2.1.1 chumpy 0.70 click 8.1.7 cloudpickle 3.0.0 colorlog 6.8.2 comm 0.2.2 ConfigArgParse 1.7 contourpy 1.2.1 cycler 0.12.1 dash 2.17.1 dash-core-components 2.0.0 dash-html-components 2.0.0 dash-table 5.0.0 decorator 5.1.1 detectron2 0.6 dill 0.3.8 einops 0.8.0 exceptiongroup 1.2.2 executing 2.0.1 fastjsonschema 2.20.0 filelock 3.13.1 fire 0.6.0 Flask 3.0.3 flatten-dict 0.4.2 fonttools 4.53.1 freetype-py 2.4.0 frozenlist 1.4.1 fsspec 2024.2.0 fvcore 0.1.5.post20221221 grpcio 1.65.4 huggingface-hub 0.24.5 hydra-colorlog 1.2.0 hydra-core 1.3.2 hydra-submitit-launcher 1.2.0 idna 3.4 imageio 2.35.0 importlib_metadata 8.2.0 iopath 0.1.9 ipython 8.26.0 ipywidgets 8.1.3 itsdangerous 2.2.0 jedi 0.19.1 Jinja2 3.1.3 joblib 1.4.2 jsonschema 4.23.0 jsonschema-specifications 2023.12.1 jupyter_core 5.7.2 jupyterlab_widgets 3.0.11 kiwisolver 1.4.5 lazy_loader 0.4 lightning-utilities 0.3.0 Markdown 3.6 markdown-it-py 3.0.0 MarkupSafe 2.1.5 matplotlib 3.9.2 matplotlib-inline 0.1.7 mdurl 0.1.2 mpmath 1.3.0 multidict 6.0.5 mypy-extensions 1.0.0 nbformat 5.10.4 nest-asyncio 1.6.0 networkx 3.2.1 numpy 1.23.1 omegaconf 2.3.0 open3d 0.18.0 opencv-python 4.10.0.84 packaging 24.1 pandas 2.2.2 parso 0.8.4 pathspec 0.12.1 pexpect 4.9.0 pillow 10.2.0 pip 24.0 platformdirs 4.2.2 plotly 5.23.0 portalocker 2.10.1 prompt_toolkit 3.0.47 protobuf 4.25.4 ptyprocess 0.7.0 pure_eval 0.2.3 pycocotools 2.0.8 pyglet 1.5.29 Pygments 2.18.0 PyOpenGL 3.1.0 pyparsing 3.1.2 pyquaternion 0.9.9 pyrender 0.1.45 pyrootutils 1.0.4 python-dateutil 2.9.0.post0 python-dotenv 1.0.1 pytorch-lightning 1.8.1 pytz 2024.1 PyYAML 6.0.2 referencing 0.35.1 regex 2024.7.24 requests 2.28.1 retrying 1.3.4 rich 13.7.1 rpds-py 0.20.0 safetensors 0.4.4 scikit-image 0.24.0 scikit-learn 1.5.1 scipy 1.13.1 setuptools 65.0.0 six 1.16.0 smplx 0.1.28 stack-data 0.6.3 submitit 1.5.1 sympy 1.12 tabulate 0.9.0 tenacity 9.0.0 tensorboard 2.17.0 tensorboard-data-server 0.7.2 termcolor 2.4.0 threadpoolctl 3.5.0 tifffile 2024.8.10 timm 1.0.8 tokenizers 0.19.1 tomli 2.0.1 torch 2.1.0+cu118 torchaudio 2.1.0+cu118 torchmetrics 0.11.4 torchvision 0.16.0+cu118 tqdm 4.66.5 traitlets 5.14.3 transformers 4.44.0 trimesh 4.4.4 triton 2.1.0 typing_extensions 4.9.0 tzdata 2024.1 urllib3 1.26.13 wcwidth 0.2.13 webdataset 0.2.86 Werkzeug 3.0.3 wheel 0.43.0 widgetsnbextension 4.0.11 yacs 0.1.8 yarl 1.9.4 zipp 3.20.0

[JongBeomBaek]

I encountered the same problem but was able to solve it. The owner's explanation is correct—it is an issue with the rendering part of the code.

Although the model correctly predicts the SMPL parameters, the problem arises during the rotation of the vertices and faces. You can check the relevant code here: https://github.com/saidwivedi/TokenHMR/blob/main/tokenhmr/lib/utils/renderer.py#L198C9-L198C13

We apply a 180-degree rotation matrix along the x-axis to the mesh [[1 0 0 0], [0 -1 0 0], [0 0 -1 0], [0 0 0 1]] However, many of the parameters end up with incorrect values. After simplifying the problem, we realized it was a reversal of the y and z axes.

I modified the code accordingly, and this resolved the issue.

However, I still don't fully understand why the problem occurred in the first place.