Wrong visualization results

[grtzsohalf]

Hi this is an amazing work! Thanks for sharing this technology to help more people check their teeth!

But I had a problem after the visualization. I could successfully run python main.py followed by python visualization.py for the demo case 1. However, I got the attached results. It seems that the angles and scales are incorrect. Could you help me with this issue?

[grtzsohalf]

If I directly plot the reconstructed mesh along with the ref mesh, it seems the result if fine. So I guess the problem is only about visualization, not the reconstruction part.

[SJTUzhou]

Could you please take a screenshot of the end part of the log file of the reconstruction process in directory ./demo/_temp/tag=1.log ? like the following one

[SJTUzhou]

It is weird. I just cloned my repo and ran the case 1. The visualization is fine on my side. Do you modify the code in visualization.py ? Further check the package vesion of your environment. The following is mine. I am using anaconda but tf2.6 is installed via pip. Name Version Build Channel absl-py 0.15.0 pypi_0 pypi aiosignal 1.2.0 pypi_0 pypi anyio 3.5.0 py38haa95532_0
argon2-cffi 21.3.0 pyhd3eb1b0_0
argon2-cffi-bindings 21.2.0 py38h2bbff1b_0
asttokens 2.0.5 pyhd3eb1b0_0
astunparse 1.6.3 pypi_0 pypi attrs 21.4.0 pyhd3eb1b0_0
babel 2.9.1 pyhd3eb1b0_0
backcall 0.2.0 pyhd3eb1b0_0
beautifulsoup4 4.11.1 py38haa95532_0
bleach 4.1.0 pyhd3eb1b0_0
brotli 1.0.9 pypi_0 pypi brotlipy 0.7.0 py38h2bbff1b_1003
ca-certificates 2022.10.11 haa95532_0
cachetools 5.2.0 pypi_0 pypi certifi 2022.9.24 py38haa95532_0
cffi 1.15.0 py38h2bbff1b_1
charset-normalizer 2.1.0 pypi_0 pypi clang 5.0 pypi_0 pypi click 8.0.4 pypi_0 pypi colorama 0.4.5 py38haa95532_0
configargparse 1.5.3 pypi_0 pypi cryptography 37.0.1 py38h21b164f_0
cuda-nvcc 11.8.89 0 nvidia cudatoolkit 11.3.1 h59b6b97_2
cudnn 8.2.1 cuda11.3_0
cupy-cuda113 10.6.0 pypi_0 pypi cycler 0.11.0 pypi_0 pypi cycpd 0.25 pypi_0 pypi dash 2.6.2 pypi_0 pypi dash-core-components 2.0.0 pypi_0 pypi dash-html-components 2.0.0 pypi_0 pypi dash-table 5.0.0 pypi_0 pypi debugpy 1.6.2 pypi_0 pypi decorator 5.1.1 pyhd3eb1b0_0 defusedxml 0.7.1 pyhd3eb1b0_0 distlib 0.3.6 pypi_0 pypi dlib 19.19.0 pypi_0 pypi entrypoints 0.4 py38haa95532_0 executing 0.8.3 pyhd3eb1b0_0 fastrlock 0.8.1 pypi_0 pypi filelock 3.8.0 pypi_0 pypi flask 2.2.2 pypi_0 pypi flask-compress 1.13 pypi_0 pypi flatbuffers 1.12 pypi_0 pypi fonttools 4.34.4 pypi_0 pypi frozenlist 1.3.1 pypi_0 pypi gast 0.4.0 pypi_0 pypi google-auth 2.9.0 pypi_0 pypi google-auth-oauthlib 0.4.6 pypi_0 pypi google-pasta 0.2.0 pypi_0 pypi grpcio 1.43.0 pypi_0 pypi h5py 3.1.0 pypi_0 pypi idna 3.3 pyhd3eb1b0_0 imageio 2.19.3 pypi_0 pypi importlib-metadata 4.12.0 pypi_0 pypi importlib_resources 5.2.0 pyhd3eb1b0_1 imutils 0.5.4 pypi_0 pypi ipykernel 6.15.1 pypi_0 pypi ipython 8.4.0 py38haa95532_0 ipython_genutils 0.2.0 pyhd3eb1b0_1 ipywidgets 8.0.2 pypi_0 pypi itsdangerous 2.1.2 pypi_0 pypi jedi 0.18.1 py38haa95532_1 jinja2 3.0.3 pyhd3eb1b0_0 joblib 1.1.0 pypi_0 pypi json5 0.9.6 pyhd3eb1b0_0 jsonschema 4.4.0 py38haa95532_0 jupyter-client 7.3.4 pypi_0 pypi jupyter-core 4.11.1 pypi_0 pypi jupyter_client 7.2.2 py38haa95532_0 jupyter_core 4.10.0 py38haa95532_0 jupyter_server 1.17.1 py38haa95532_0 jupyterlab 3.3.2 pyhd3eb1b0_0 jupyterlab-widgets 3.0.3 pypi_0 pypi jupyterlab_pygments 0.1.2 py_0 jupyterlab_server 2.12.0 py38haa95532_0 keras 2.6.0 pypi_0 pypi keras-preprocessing 1.1.2 pypi_0 pypi kiwisolver 1.4.3 pypi_0 pypi libsodium 1.0.18 h62dcd97_0 llvmlite 0.39.1 pypi_0 pypi markdown 3.3.7 pypi_0 pypi markupsafe 2.1.1 py38h2bbff1b_0 matplotlib 3.5.2 pypi_0 pypi matplotlib-inline 0.1.3 pypi_0 pypi mistune 0.8.4 py38he774522_1000 msgpack 1.0.4 pypi_0 pypi nb_conda 2.2.1 py38_1 nb_conda_kernels 2.3.1 py38haa95532_0 nbclassic 0.3.5 pyhd3eb1b0_0 nbclient 0.5.13 py38haa95532_0 nbconvert 6.4.4 py38haa95532_0 nbformat 5.5.0 py38haa95532_0 nest-asyncio 1.5.5 py38haa95532_0 networkx 2.8.4 pypi_0 pypi nodejs 16.13.1 haa95532_0 notebook 6.4.11 py38haa95532_0 numba 0.56.4 pypi_0 pypi numpy 1.19.5 pypi_0 pypi oauthlib 3.2.0 pypi_0 pypi open3d 0.16.0 pypi_0 pypi opencv-contrib-python 4.6.0.66 pypi_0 pypi opencv-python 4.6.0.66 pypi_0 pypi openssl 1.1.1s h2bbff1b_0 opt-einsum 3.3.0 pypi_0 pypi packaging 21.3 pyhd3eb1b0_0 pandas 1.4.3 pypi_0 pypi pandocfilters 1.5.0 pyhd3eb1b0_0 parso 0.8.3 pyhd3eb1b0_0 pickleshare 0.7.5 pyhd3eb1b0_1003 pillow 9.2.0 pypi_0 pypi pip 22.1.2 py38haa95532_0 platformdirs 2.5.2 pypi_0 pypi plotly 5.10.0 pypi_0 pypi prometheus_client 0.13.1 pyhd3eb1b0_0 prompt-toolkit 3.0.30 pypi_0 pypi protobuf 3.19.0 pypi_0 pypi psutil 5.9.1 pypi_0 pypi pure_eval 0.2.2 pyhd3eb1b0_0 pyaml 21.10.1 pypi_0 pypi pyasn1 0.4.8 pypi_0 pypi pyasn1-modules 0.2.8 pypi_0 pypi pycocotools 2.0 pypi_0 pypi pycparser 2.21 pyhd3eb1b0_0 pygments 2.12.0 pypi_0 pypi pyopenssl 22.0.0 pyhd3eb1b0_0 pyparsing 3.0.9 pypi_0 pypi pyrsistent 0.18.0 py38h196d8e1_0 pysocks 1.7.1 py38haa95532_0 python 3.8.13 h6244533_0 python-dateutil 2.8.2 pyhd3eb1b0_0 python-fastjsonschema 2.15.1 pyhd3eb1b0_0 pytz 2022.1 py38haa95532_0 pywavelets 1.3.0 pypi_0 pypi pywin32 304 pypi_0 pypi pywinpty 2.0.2 py38h5da7b33_0 pyzmq 23.2.0 py38hd77b12b_0 ray 2.0.1 pypi_0 pypi requests 2.28.1 py38haa95532_0 requests-oauthlib 1.3.1 pypi_0 pypi rsa 4.8 pypi_0 pypi rtree 1.0.1 pypi_0 pypi scikit-image 0.19.3 pypi_0 pypi scikit-learn 1.1.1 pypi_0 pypi scikit-optimize 0.9.0 pypi_0 pypi scipy 1.8.1 pypi_0 pypi seaborn 0.12.2 pypi_0 pypi send2trash 1.8.0 pyhd3eb1b0_1 setuptools 61.2.0 py38haa95532_0 shapely 1.8.5.post1 pypi_0 pypi six 1.15.0 pypi_0 pypi sniffio 1.2.0 py38haa95532_1 soupsieve 2.3.1 pyhd3eb1b0_0 sqlite 3.38.5 h2bbff1b_0 stack-data 0.3.0 pypi_0 pypi stack_data 0.2.0 pyhd3eb1b0_0 tenacity 8.1.0 pypi_0 pypi tensorboard 2.10.0 pypi_0 pypi tensorboard-data-server 0.6.1 pypi_0 pypi tensorboard-plugin-wit 1.8.1 pypi_0 pypi tensorflow-addons 0.16.1 pypi_0 pypi tensorflow-estimator 2.9.0 pypi_0 pypi tensorflow-gpu 2.6.0 pypi_0 pypi termcolor 1.1.0 pypi_0 pypi terminado 0.13.1 py38haa95532_0 testpath 0.6.0 py38haa95532_0 threadpoolctl 3.1.0 pypi_0 pypi tifffile 2022.5.4 pypi_0 pypi tornado 6.2 pypi_0 pypi traitlets 5.3.0 pypi_0 pypi trimesh 3.15.5 pypi_0 pypi typing-extensions 3.7.4.3 pypi_0 pypi typing_extensions 4.1.1 pyh06a4308_0 urllib3 1.26.10 pypi_0 pypi vc 14.2 h21ff451_1 virtualenv 20.16.5 pypi_0 pypi vs2015_runtime 14.27.29016 h5e58377_2 wcwidth 0.2.5 pyhd3eb1b0_0 webencodings 0.5.1 py38_1 websocket-client 0.58.0 py38haa95532_4 werkzeug 2.2.2 pypi_0 pypi wheel 0.37.1 pyhd3eb1b0_0 widgetsnbextension 4.0.3 pypi_0 pypi win_inet_pton 1.1.0 py38haa95532_0 wincertstore 0.2 py38haa95532_2 winpty 0.4.3 4 wrapt 1.12.1 pypi_0 pypi zeromq 4.3.4 hd77b12b_0 zipp 3.8.0 py38haa95532_0

[grtzsohalf]

I think I fixed the bug. It is because the version of open3d I used was 0.17.0 . There is a related bug report here: https://github.com/isl-org/Open3D/pull/6116 So after I rolled back the version to 0.16.0 , the result looked great!

[SJTUzhou]

Glad to see the bug fixed. I am going to close this issue since it is solved.

Moreover, at the current stage, the teeth reconstruction may fail sometimes, which is mainly because the point correspondences are not established correctly.

One way to do that is trying to label each tooth in the image with its corresponding tooth number, such as an index in FDI system in dentistry, and build contour point correspondence for each tooth.

If you are interested in this work, you could try to further increase the recontruction accuracy via this way.

[grtzsohalf]

Thanks for the instruction and the future work guidance! This work is really meaningful! We will try to further improve it!