Rotation Error in Predictions

madhanuman commented 4 months ago

Dear @ybkscht

I am trying to use a custom dataset and run into troubles regarding the predictions of the model. I used the debug.py file and it gave me following outputs:

This looks correct to me and I then trained my model and got these values of the training: mAP: 0.9906 ADD: 0.9900 ADD-S: 0.9900 5cm_5degree: 0.0000 TranslationErrorMean_in_mm: 37.0934 TranslationErrorStd_in_mm: 23.4644 RotationErrorMean_in_degree: 99.3070 RotationErrorStd_in_degree: 41.9650 2D-Projection: 0.2131 Summed_Translation_Rotation_Error: 201.8299 ADD(-S): 0.9900 AveragePointDistanceMean_in_mm: 37.0991 AveragePointDistanceStd_in_mm: 23.4630 AverageSymmetricPointDistanceMean_in_mm: 36.5519 AverageSymmetricPointDistanceStd_in_mm: 23.4492 MixedAveragePointDistanceMean_in_mm: 37.0991 MixedAveragePointDistanceStd_in_mm: 23.4630

The ADD was really high and the TranslationErrorMean_in_mm is low enough but unfortnuately the Rotation error was quite high... And the inference.py file showed that too: I checked the intrinsic camera matrix, the scale of the GT and the model itself... Regarding GT and model i had the feeling it should be true since also the debug.py file showed... I also tried using different translation loss scale factor but it did not improve the results...

Do you have any idea where my problem is?

Thank you very much for your help and contributions!

monajalal commented 4 months ago

@madhanuman could you please share your environment.yaml / ubuntu version/ other reqs please?

madhanuman commented 4 months ago

This would be the environment.yml `name: EfficientPose channels:

defaults dependencies:
_tflow_select=2.1.0=gpu
absl-py=0.15.0=pyhd3eb1b0_0
aiohttp=3.8.3=py37h2bbff1b_0
aiosignal=1.2.0=pyhd3eb1b0_0
astor=0.8.1=py37haa95532_0
async-timeout=4.0.2=py37haa95532_0
asynctest=0.13.0=py_0
attrs=22.1.0=py37haa95532_0
blas=1.0=mkl
blinker=1.4=py37haa95532_0
brotlipy=0.7.0=py37h2bbff1b_1003
ca-certificates=2023.12.12=haa95532_0
cachetools=4.2.2=pyhd3eb1b0_0
certifi=2022.12.7=py37haa95532_0
cffi=1.15.1=py37h2bbff1b_3
charset-normalizer=2.0.4=pyhd3eb1b0_0
click=8.0.4=py37haa95532_0
colorama=0.4.6=py37haa95532_0
cryptography=39.0.1=py37h21b164f_0
cudatoolkit=10.0.130=0
cudnn=7.6.5=cuda10.0_0
fftw=3.3.9=h2bbff1b_1
flit-core=3.6.0=pyhd3eb1b0_0
frozenlist=1.3.3=py37h2bbff1b_0
gast=0.2.2=py37_0
google-auth=2.6.0=pyhd3eb1b0_0
google-auth-oauthlib=0.4.4=pyhd3eb1b0_0
google-pasta=0.2.0=pyhd3eb1b0_0
grpcio=1.42.0=py37hc60d5dd_0
hdf5=1.10.6=h1756f20_1
icc_rt=2022.1.0=h6049295_2
idna=3.4=py37haa95532_0
importlib-metadata=4.11.3=py37haa95532_0
intel-openmp=2021.4.0=haa95532_3556
keras-applications=1.0.8=py_1
keras-preprocessing=1.1.2=pyhd3eb1b0_0
libprotobuf=3.20.3=h23ce68f_0
markdown=3.4.1=py37haa95532_0
mkl=2021.4.0=haa95532_640
mkl-service=2.4.0=py37h2bbff1b_0
mkl_fft=1.3.1=py37h277e83a_0
mkl_random=1.2.2=py37hf11a4ad_0
multidict=6.0.2=py37h2bbff1b_0
oauthlib=3.2.1=py37haa95532_0
openssl=1.1.1w=h2bbff1b_0
opt_einsum=3.3.0=pyhd3eb1b0_1
pip=22.3.1=py37haa95532_0
protobuf=3.20.3=py37hd77b12b_0
pyasn1=0.4.8=pyhd3eb1b0_0
pyasn1-modules=0.2.8=py_0
pycparser=2.21=pyhd3eb1b0_0
pyjwt=2.4.0=py37haa95532_0
pyopenssl=23.0.0=py37haa95532_0
pysocks=1.7.1=py37_1
python=3.7.16=h6244533_0
requests=2.28.1=py37haa95532_0
requests-oauthlib=1.3.0=py_0
rsa=4.7.2=pyhd3eb1b0_1
scipy=1.7.3=py37h7a0a035_2
setuptools=65.6.3=py37haa95532_0
six=1.16.0=pyhd3eb1b0_1
sqlite=3.41.2=h2bbff1b_0
tensorboard-data-server=0.6.1=py37haa95532_0
tensorboard-plugin-wit=1.8.1=py37haa95532_0
tensorflow=1.15.0=gpu_py37hc3743a6_0
tensorflow-base=1.15.0=gpu_py37h1afeea4_0
tensorflow-gpu=1.15.0=h0d30ee6_0
termcolor=1.1.0=py37haa95532_1
typing-extensions=4.4.0=py37haa95532_0
typing_extensions=4.4.0=py37haa95532_0
urllib3=1.26.14=py37haa95532_0
vc=14.2=h21ff451_1
vs2015_runtime=14.27.29016=h5e58377_2
werkzeug=0.16.1=py_0
wheel=0.38.4=py37haa95532_0
win_inet_pton=1.1.0=py37haa95532_0
wincertstore=0.2=py37haa95532_2
wrapt=1.14.1=py37h2bbff1b_0
yarl=1.8.1=py37h2bbff1b_0
zipp=3.11.0=py37haa95532_0
zlib=1.2.13=h8cc25b3_0
pip:
- cycler==0.11.0
- cython==0.29.23
- fonttools==4.38.0
- h5py==2.10.0
- imageio==2.31.2
- imgaug==0.4.0
- kiwisolver==1.4.5
- matplotlib==3.5.3
- networkx==2.6.3
- numpy==1.18.5
- opencv-python==4.3.0.36
- packaging==23.2
- pandas==1.3.5
- pillow==9.5.0
- plydata==0.4.3
- plyfile==0.7.2
- progressbar2==3.53.1
- pyparsing==3.1.1
- python-dateutil==2.8.2
- python-utils==3.5.2
- pytz==2023.4
- pywavelets==1.3.0
- pyyaml==5.3.1
- scikit-image==0.19.3
- shapely==2.0.2
- tensorboard==1.15.0
- tensorflow-estimator==1.15.1
- tifffile==2021.11.2
- tqdm==4.48.0
- typeguard==2.9.1 prefix: C:\Users\admin\anaconda3\envs\EfficientPose `

madhanuman commented 4 months ago

I was able to solve this issue... I realized it was indeed the scaling which was wrong... I had to change the gt.yml i created for my custom dataset and then also modify the models_info.yml now the rotation error is >10 degrees

monajalal commented 4 months ago

@madhanuman would you please share the .yaml file here (the original file) or email it to mona.jalal.tmh@gmail.com? The current format is damaged by github.

Also, what is your Ubuntu version? Is your Python python=3.7.16? Are you using a Docker btw?

Thanks a lot for your help.

madhanuman commented 4 months ago

I am using anaconda for managing the project so i dont use ubuntu... and yes i use python 3.7.16

SMhang commented 3 months ago

May I ask if your problem has been resolved? I have encountered the same problem as you and would like to ask for your advice

ybkscht / EfficientPose

Rotation Error in Predictions #78