Open DHuiTnut opened 5 months ago
roym899
commented
4 months ago I can't really reproduce this issue. The mask is auto-generated from the depth image and the ground-truth mesh, so my best guess is that something with the dataset is not correct.
Could you try to redownload it by deleting the redwood and redwood75 directories and rerun the script with this fix: https://github.com/roym899/pose_and_shape_evaluation/commit/8f32355471cf996e3671d68e43a6eabe7b86c9f4?
DHuiTnut
commented
4 months ago I can't really reproduce this issue. The mask is auto-generated from the depth image and the ground-truth mesh, so my best guess is that something with the dataset is not correct.
Could you try to redownload it by deleting the redwood and redwood75 directories and rerun the script with this fix: 8f32355?
Sorry for the late reply to your comment. I redownloaded the redwood dataset following your latest code. Unfortunately, the generated mask still does not match the picture. Is it convenient for you to provide the image or npy file of the mask you generated? Thanks for your help!
roym899
commented
4 months ago I have created a file containing the masks: redwood75_masks.zip I think it should be relatively straightforward to modify redwood_dataset.py to use these instead of the generated ones.
Could you maybe share the output of pip freeze? This might help to debug this issue. I suspect that open3d behaves differently on your platform when generating the depth image.
DHuiTnut
commented
3 months ago Thanks for your sharing. I have done the evaluation for my model on redwood75 dataset by the masks you provided.
And here is the output of pip freeze:
absl-py @ file:///C:/b/abs_d3cv5rzljl/croot/absl-py_1686852506854/work aiohttp @ file:///C:/b/abs_bc6tmjiy12/croot/aiohttp_1701112585940/work aiosignal @ file:///tmp/build/80754af9/aiosignal_1637843061372/work ansi2html==1.8.0 asttokens==2.4.1 async-timeout @ file:///C:/b/abs_43ozhz2a8g/croots/recipe/async-timeout_1664876362767/work attrs @ file:///C:/b/abs_35n0jusce8/croot/attrs_1695717880170/work Babel==2.14.0 backcall==0.2.0 beautifulsoup4==4.12.3 black==24.1.1 blinker @ file:///C:/b/abs_d9y2dm7cw2/croot/blinker_1696539752170/work Brotli @ file:///C:/Windows/Temp/abs_63l7912z0e/croots/recipe/brotli-split_1659616056886/work cachetools @ file:///tmp/build/80754af9/cachetools_1619597386817/work certifi @ file:///C:/b/abs_91u83siphd/croot/certifi_1700501720658/work/certifi cffi @ file:///C:/b/abs_924gv1kxzj/croot/cffi_1700254355075/work chardet==5.2.0 charset-normalizer @ file:///tmp/build/80754af9/charset-normalizer_1630003229654/work click @ file:///C:/b/abs_f9ihnt72pu/croot/click_1698129847492/work cloudpickle @ file:///C:/b/abs_3796yxesic/croot/cloudpickle_1683040098851/work colorama @ file:///C:/b/abs_a9ozq0l032/croot/colorama_1672387194846/work comm==0.2.0 ConfigArgParse==1.7 contourpy @ file:///C:/b/abs_d5rpy288vc/croots/recipe/contourpy_1663827418189/work coverage==7.4.1 -e git+https://github.com/roym899/pose_and_shape_evaluation.git@0632e3214274439a63cf8b4ae027c05ede49e330#egg=cpas_toolbox cryptography @ file:///C:/b/abs_f4do8t8jfs/croot/cryptography_1694444424531/work cycler @ file:///tmp/build/80754af9/cycler_1637851556182/work cytoolz @ file:///C:/b/abs_d43s8lnb60/croot/cytoolz_1701723636699/work dash==2.14.1 dash-core-components==2.0.0 dash-html-components==2.0.0 dash-table==5.0.0 dask @ file:///C:/b/abs_59fkcimley/croot/dask-core_1683065223377/work decorator==5.1.1 easydict==1.11 exceptiongroup==1.2.0 executing==2.0.1 fastjsonschema==2.18.1 filelock==3.13.1 filterpy==1.4.5 flake8==7.0.0 flake8-annotations==3.0.1 flake8-docstrings==1.7.0 Flask==3.0.0 fonttools==4.25.0 frozenlist @ file:///C:/b/abs_d8e__s1ys3/croot/frozenlist_1698702612014/work fsspec @ file:///C:/b/abs_97mpfsesn0/croot/fsspec_1701286534629/work gdown==5.1.0 ghp-import==2.1.0 google-auth @ file:///C:/b/abs_defnokp9xd/croot/google-auth_1694152741394/work google-auth-oauthlib @ file:///tmp/build/80754af9/google-auth-oauthlib_1617120569401/work griffe==0.40.0 grpcio @ file:///C:/b/abs_654jr7av_l/croot/grpc-suite_1698063171901/work idna @ file:///C:/b/abs_bdhbebrioa/croot/idna_1666125572046/work imagecodecs @ file:///C:/b/abs_e2g5zbs1q0/croot/imagecodecs_1695065012000/work imageio @ file:///C:/b/abs_3eijmwdodc/croot/imageio_1695996500830/work importlib-metadata @ file:///C:/b/abs_20ndzb2j6v/croot/importlib-metadata_1678997085534/work importlib-resources @ file:///C:/b/abs_9d6rmtszbl/croot/importlib_resources_1698254653090/work iniconfig==2.0.0 ipython==8.12.3 ipywidgets==8.1.1 isort==5.13.2 itsdangerous==2.1.2 jedi==0.19.1 Jinja2==3.1.2 joblib==1.3.2 jsonschema==4.19.2 jsonschema-specifications==2023.7.1 jupyter_core==5.5.0 jupyterlab-widgets==3.0.9 kiwisolver @ file:///C:/b/abs_88mdhvtahm/croot/kiwisolver_1672387921783/work llvmlite==0.41.1 locket @ file:///C:/ci/locket_1652904093019/work Markdown==3.5.1 MarkupSafe @ file:///C:/ci/markupsafe_1654489871526/work matplotlib==3.5.3 matplotlib-inline==0.1.6 mccabe==0.7.0 mergedeep==1.3.4 mkdocs==1.5.3 mkdocs-autorefs==0.5.0 mkdocs-material==9.5.7 mkdocs-material-extensions==1.3.1 mkdocstrings==0.24.0 mkdocstrings-python==1.8.0 mkl-fft @ file:///C:/b/abs_19i1y8ykas/croot/mkl_fft_1695058226480/work mkl-random @ file:///C:/b/abs_edwkj1_o69/croot/mkl_random_1695059866750/work mkl-service==2.4.0 multidict @ file:///C:/b/abs_44ido987fv/croot/multidict_1701097803486/work munkres==1.1.4 mypy-extensions==1.0.0 nbformat==5.7.0 nest-asyncio==1.5.8 networkx @ file:///C:/b/abs_e6gi1go5op/croot/networkx_1690562046966/work ninja==1.11.1.1 numba==0.58.1 numpy @ file:///C:/Users/dev-admin/mkl/numpy_and_numpy_base_1682982345978/work oauthlib @ file:///C:/b/abs_44riqpe5gg/croot/oauthlib_1679489644705/work open3d==0.17.0 opencv-python==4.8.1.78 opencv-python-headless==4.9.0.80 packaging @ file:///C:/b/abs_28t5mcoltc/croot/packaging_1693575224052/work paginate==0.5.6 parso==0.8.3 partd @ file:///C:/b/abs_46awex0fd7/croot/partd_1698702622970/work pathspec==0.12.1 pickleshare==0.7.5 Pillow @ file:///C:/b/abs_20ztcm8lgk/croot/pillow_1696580089746/work pkgutil_resolve_name==1.3.10 platformdirs @ file:///C:/b/abs_b6z_yqw_ii/croot/platformdirs_1692205479426/work plotly==5.18.0 pluggy==1.4.0 ply==3.11 plyfile==1.0.3 pooch @ file:///C:/b/abs_a8nmng7d_x/croot/pooch_1695850149827/work prompt-toolkit==3.0.39 protobuf==3.20.3 pure-eval==0.2.2 pyasn1 @ file:///Users/ktietz/demo/mc3/conda-bld/pyasn1_1629708007385/work pyasn1-modules==0.2.8 pybind11==2.11.1 pybind11-global==2.11.1 pycodestyle==2.11.1 pycparser @ file:///tmp/build/80754af9/pycparser_1636541352034/work pydocstyle==6.3.0 pyflakes==3.2.0 Pygments==2.16.1 PyJWT @ file:///C:/ci/pyjwt_1657529430378/work pymdown-extensions==10.7 pyOpenSSL @ file:///C:/b/abs_08f38zyck4/croot/pyopenssl_1690225407403/work pyparsing @ file:///C:/Users/BUILDE~1/AppData/Local/Temp/abs_7f_7lba6rl/croots/recipe/pyparsing_1661452540662/work PyQt5==5.15.10 PyQt5-sip @ file:///C:/b/abs_c0pi2mimq3/croot/pyqt-split_1698769125270/work/pyqt_sip PySocks @ file:///C:/ci/pysocks_1605287845585/work pytest==8.0.0 pytest-cov==4.1.0 python-dateutil @ file:///tmp/build/80754af9/python-dateutil_1626374649649/work pytz==2024.1 PyWavelets @ file:///C:/b/abs_a8r4b1511a/croot/pywavelets_1670425185881/work pywin32==306 PyYAML @ file:///C:/b/abs_782o3mbw7z/croot/pyyaml_1698096085010/work pyyaml_env_tag==0.1 referencing==0.30.2 regex==2023.12.25 requests @ file:///C:/b/abs_316c2inijk/croot/requests_1690400295842/work requests-oauthlib==1.3.0 retrying==1.3.4 rpds-py==0.12.0 rsa @ file:///tmp/build/80754af9/rsa_1614366226499/work ruamel.yaml==0.18.5 ruamel.yaml.clib==0.2.8 scikit-image @ file:///C:/b/abs_63r0vmx78u/croot/scikit-image_1669241746873/work scikit-learn==1.3.2 scipy==1.10.1 sip @ file:///C:/b/abs_edevan3fce/croot/sip_1698675983372/work six @ file:///tmp/build/80754af9/six_1644875935023/work snowballstemmer==2.2.0 soupsieve==2.5 stack-data==0.6.3 tenacity==8.2.3 tensorboard @ file:///C:/Users/builder/adipietro/mc3/tf210/conda-bld/tensorboard_1669760968711/work/tensorboard-2.10.0-py3-none-any.whl tensorboard-data-server @ file:///C:/b/abs_2fhvpo862s/croot/tensorboard-data-server_1670853600144/work/tensorboard_data_server-0.6.1-py3-none-any.whl tensorboard-plugin-wit @ file:///C:/tf/b/tensorboard-plugin-wit_1660162132996/work/tensorboard_plugin_wit-1.8.1-py3-none-any.whl tensorboardX==2.6.2.2 threadpoolctl==3.2.0 tifffile @ file:///C:/b/abs_45o5chuqwt/croot/tifffile_1695107511025/work tikzplotlib==0.10.1 tomli @ file:///C:/Windows/TEMP/abs_ac109f85-a7b3-4b4d-bcfd-52622eceddf0hy332ojo/croots/recipe/tomli_1657175513137/work toolz @ file:///C:/b/abs_cfvk6rc40d/croot/toolz_1667464080130/work torch==1.12.1 torchaudio==0.12.1 torchvision==0.13.1 tornado @ file:///C:/b/abs_0cbrstidzg/croot/tornado_1696937003724/work tqdm @ file:///C:/b/abs_f76j9hg7pv/croot/tqdm_1679561871187/work traitlets==5.13.0 transforms3d==0.4.1 trimesh==4.1.3 typing_extensions @ file:///C:/b/abs_213vg2cd1l/croot/typing_extensions_1690297804941/work urllib3 @ file:///C:/b/abs_9cmlsrm3ys/croot/urllib3_1698257595508/work watchdog==3.0.0 wcwidth==0.2.9 webcolors==1.13 Werkzeug==3.0.1 widgetsnbextension==4.0.9 win-inet-pton @ file:///C:/ci/win_inet_pton_1605306167264/work yarl @ file:///C:/b/abs_8bxwdyhjvp/croot/yarl_1701105248152/work yoco==1.1.1 zipp @ file:///C:/b/abs_b9jfdr908q/croot/zipp_1672387552360/work
Hi, thanks for your excellent work! With the help of your tool, I have done most of the performance evaluation of the model on the REAL dataset. But when I do the performance evaluation on the RedWood dataset, the obtained metrics look very different from those in the paper (take dpdn as an example, see the snapshot below).
I visualized the input of the dataset and found that the angle of the generated mask is not consistent with the angle in the RGB image,. I am not sure if this is the reason for this problem. Could you please tell me how to solve this problem?