search

CVLAB-Unibo / Real-time-self-adaptive-deep-stereo

Code for "Real-time self-adaptive deep stereo" - CVPR 2019 (ORAL)
Apache License 2.0
420 stars 73 forks source link

measurement of depth variation is very large for every frame. #21

Closed vigneshgig closed 5 years ago

vigneshgig commented 5 years ago

Hi, I used live demo python code it's working good sometime and sometime it showing bad depth for same environment. it's continuously varying for same environment and also object are stationary.

any suggestion regarding to solve this problem. because for calibration I am using linear regression. so that if I get 4.08 m i can map to 4m.but due to too much variation I unable to use the linear regression. I also tried to train the model in mad mode then I tried but I am getting same variation. basline : 0.110m focal_length:792.8456

cmd : python3.7 Live_Adaptation_Demo.py --weights /home/giuser/MyWorkspace/Real-time-self-adaptive-deep-stereo/pretrained_nets/MADNet/synthetic/weights.ckpt

I used diy camera ,so I modified the grabber for my camera. and also I am sending the rectified image.

Step 27: 0.09816490113735199 max : 340.11823 min: 24.168932 shape_disp: (320, 512, 1) depth_max : 3.6084762 depth_min: 0.2564197 Step 28: 0.09606088697910309 max : 341.95813 min: 22.652584 shape_disp: (320, 512, 1) depth_max : 3.850025 depth_min: 0.25504002 Step 29: 0.09736260026693344 max : 328.382 min: 29.71734 shape_disp: (320, 512, 1) depth_max : 2.9347515 depth_min: 0.26558402 Step 30: 0.09974691271781921 max : 339.23883 min: 9.043276 shape_disp: (320, 512, 1) depth_max : 9.643963 depth_min: 0.2570844 Step 31: 0.09843119233846664 max : 336.58228 min: 25.447483 shape_disp: (320, 512, 1) depth_max : 3.4271765 depth_min: 0.2591135 Step 32: 0.09997092932462692 max : 317.42358 min: 24.566387 shape_disp: (320, 512, 1) depth_max : 3.550095 depth_min: 0.2747528 Step 33: 0.09671687334775925 max : 327.99982 min: 24.208145 shape_disp: (320, 512, 1) depth_max : 3.6026309 depth_min: 0.2658935 Step 34: 0.09664341807365417 max : 342.8994 min: 29.254269 shape_disp: (320, 512, 1) depth_max : 2.9812064 depth_min: 0.25433993 Step 35: 0.09987105429172516 max : 329.02496 min: 21.65923 shape_disp: (320, 512, 1) depth_max : 4.026598 depth_min: 0.26506504 Step 36: 0.09744423627853394 max : 342.30637 min: 22.168484 shape_disp: (320, 512, 1) depth_max : 3.9340992 depth_min: 0.25478056 Step 37: 0.09748262912034988 max : 340.14005 min: 16.909391 shape_disp: (320, 512, 1) depth_max : 5.157667 depth_min: 0.25640324 Step 38: 0.10002128779888153 max : 342.14606 min: 20.058685 shape_disp: (320, 512, 1) depth_max : 4.3478928 depth_min: 0.25489995 Step 39: 0.09692289680242538 max : 347.4746 min: 24.163637 shape_disp: (320, 512, 1) depth_max : 3.6092668 depth_min: 0.25099105 Step 40: 0.0976732149720192 max : 340.15384 min: 26.022049 shape_disp: (320, 512, 1) depth_max : 3.3515043 depth_min: 0.25639284 Step 41: 0.09611301124095917 max : 332.27756 min: 31.099731 shape_disp: (320, 512, 1) depth_max : 2.8043013 depth_min: 0.26247036 Step 42: 0.09700638800859451 max : 327.2958 min: 19.325695 shape_disp: (320, 512, 1) depth_max : 4.5128007 depth_min: 0.26646543 Step 43: 0.09772888571023941 max : 334.38843 min: 23.648607 shape_disp: (320, 512, 1) depth_max : 3.687871 depth_min: 0.26081347 Step 44: 0.10100344568490982 max : 335.46814 min: 14.885302 shape_disp: (320, 512, 1) depth_max : 5.859002 depth_min: 0.25997406 Step 45: 0.09700295329093933 max : 341.30365 min: 25.402328 shape_disp: (320, 512, 1) depth_max : 3.4332685 depth_min: 0.2555291 Step 46: 0.10043686628341675 max : 342.11014 min: 17.508682 shape_disp: (320, 512, 1) depth_max : 4.9811296 depth_min: 0.2549267 Step 47: 0.09660971164703369 max : 342.87683 min: 23.009247 shape_disp: (320, 512, 1) depth_max : 3.7903461 depth_min: 0.25435668 Step 48: 0.09604205936193466 max : 331.56754 min: 21.01507 shape_disp: (320, 512, 1) depth_max : 4.1500225 depth_min: 0.26303244 Step 49: 0.10040544718503952 max : 336.10278 min: 9.170251 shape_disp: (320, 512, 1) depth_max : 9.510428 depth_min: 0.25948316 Step 50: 0.09776761382818222 max : 335.4831 min: 19.53571 shape_disp: (320, 512, 1) depth_max : 4.464287 depth_min: 0.25996247 Step 51: 0.09593342244625092 max : 322.93915 min: 25.418804 shape_disp: (320, 512, 1) depth_max : 3.4310431 depth_min: 0.2700602 Step 52: 0.09986594319343567 max : 328.32758 min: 19.760132 shape_disp: (320, 512, 1) depth_max : 4.4135847 depth_min: 0.26562804 Step 53: 0.09683622419834137 max : 328.09158 min: 22.77004 shape_disp: (320, 512, 1) depth_max : 3.830165 depth_min: 0.2658191 Step 54: 0.09858302772045135 max : 343.56897 min: 22.25194 shape_disp: (320, 512, 1) depth_max : 3.9193442 depth_min: 0.25384426 Step 55: 0.09800785779953003 max : 331.22284 min: 27.093431 shape_disp: (320, 512, 1) depth_max : 3.2189727 depth_min: 0.26330617 Step 56: 0.09877340495586395 max : 342.5056 min: 22.059685 shape_disp: (320, 512, 1) depth_max : 3.9535022 depth_min: 0.25463235 Step 57: 0.09741590917110443 max : 333.68042 min: 19.966022 shape_disp: (320, 512, 1) depth_max : 4.3680716 depth_min: 0.26136687 Step 58: 0.09748111665248871 max : 336.98834 min: 20.162483 shape_disp: (320, 512, 1) depth_max : 4.3255095 depth_min: 0.25880128 Step 59: 0.09806456416845322 max : 334.3465 min: 26.305145 shape_disp: (320, 512, 1) depth_max : 3.3154354 depth_min: 0.2608462 Step 60: 0.09643004834651947 max : 332.56482 min: 24.784477 shape_disp: (320, 512, 1) depth_max : 3.5188563 depth_min: 0.26224366 Step 61: 0.09784868359565735 max : 331.90765 min: 22.224346 shape_disp: (320, 512, 1) depth_max : 3.9242105 depth_min: 0.26276287 Step 62: 0.10155725479125977 max : 340.5157 min: 14.950236 shape_disp: (320, 512, 1) depth_max : 5.8335543 depth_min: 0.25612038 Step 63: 0.10066811740398407 max : 330.7791 min: 11.245592 shape_disp: (320, 512, 1) depth_max : 7.7553062 depth_min: 0.26365936 Step 64: 0.09890112280845642 max : 332.29175 min: 21.645153 shape_disp: (320, 512, 1) depth_max : 4.029217 depth_min: 0.26245916 Step 65: 0.09773185104131699 max : 339.99353 min: 27.148638 shape_disp: (320, 512, 1) depth_max : 3.212427 depth_min: 0.25651374 Step 66: 0.09856224805116653 max : 345.18665 min: 27.773724 shape_disp: (320, 512, 1) depth_max : 3.1401267 depth_min: 0.25265464 Step 67: 0.10059231519699097 max : 341.5364 min: 22.40047 shape_disp: (320, 512, 1) depth_max : 3.8933563 depth_min: 0.25535494 Step 68: 0.10034771263599396 max : 334.68796 min: 13.772877 shape_disp: (320, 512, 1) depth_max : 6.332229 depth_min: 0.26058006 Step 69: 0.09811978787183762 max : 327.84946 min: 26.435816 shape_disp: (320, 512, 1) depth_max : 3.2990475 depth_min: 0.26601544 Step 70: 0.09709776937961578 max : 336.1238 min: 29.418407 shape_disp: (320, 512, 1) depth_max : 2.964573 depth_min: 0.25946692 Step 71: 0.09988223016262054 max : 334.7636 min: 22.078262 shape_disp: (320, 512, 1) depth_max : 3.9501755 depth_min: 0.26052117 Step 72: 0.09658467024564743 max : 329.68552 min: 26.856031 shape_disp: (320, 512, 1) depth_max : 3.2474275 depth_min: 0.26453397 Step 73: 0.09921842068433762 max : 328.62988 min: 16.477346 shape_disp: (320, 512, 1) depth_max : 5.292904 depth_min: 0.2653837 Step 74: 0.10068050026893616 max : 336.04236 min: 16.554989 shape_disp: (320, 512, 1) depth_max : 5.26808 depth_min: 0.25952983 Step 75: 0.10103623569011688 max : 340.24564 min: 18.45589 shape_disp: (320, 512, 1) depth_max : 4.725484 depth_min: 0.25632367 Step 76: 0.09824462980031967 max : 330.18896 min: 27.530098 shape_disp: (320, 512, 1) depth_max : 3.167915 depth_min: 0.26413062 Step 77: 0.0993424579501152 max : 337.9097 min: 15.524535 shape_disp: (320, 512, 1) depth_max : 5.6177535 depth_min: 0.25809562 Step 78: 0.09849735349416733 max : 333.8875 min: 26.08546 shape_disp: (320, 512, 1) depth_max : 3.343357 depth_min: 0.26120478 Step 79: 0.10081659257411957 max : 331.29367 min: 16.75264 shape_disp: (320, 512, 1) depth_max : 5.2059264 depth_min: 0.26324987 Step 80: 0.0999860018491745 max : 325.02478 min: 23.194695 shape_disp: (320, 512, 1) depth_max : 3.7600415 depth_min: 0.26832727 Step 81: 0.09867776930332184 max : 332.46118 min: 29.91875 shape_disp: (320, 512, 1) depth_max : 2.9149952 depth_min: 0.2623254 Step 82: 0.1008116751909256 max : 334.99695 min: 24.041918 shape_disp: (320, 512, 1) depth_max : 3.6275399 depth_min: 0.26033974 Step 83: 0.0999288558959961 max : 331.01315 min: 20.677118 shape_disp: (320, 512, 1) depth_max : 4.2178516 depth_min: 0.26347294 Step 84: 0.0983358770608902 max : 337.17862 min: 27.256302 shape_disp: (320, 512, 1) depth_max : 3.1997375 depth_min: 0.25865522 Step 85: 0.10136320441961288 max : 333.1576 min: 15.041556 shape_disp: (320, 512, 1) depth_max : 5.7981377 depth_min: 0.26177704 Step 86: 0.09810695797204971 max : 330.13412 min: 24.183144 shape_disp: (320, 512, 1) depth_max : 3.6063554 depth_min: 0.2641745 Step 87: 0.09889024496078491 max : 336.52222 min: 21.0831 shape_disp: (320, 512, 1) depth_max : 4.1366315 depth_min: 0.25915974 Step 88: 0.09982095658779144 max : 332.24713 min: 11.930489 shape_disp: (320, 512, 1) depth_max : 7.310096 depth_min: 0.26249442 Step 89: 0.09888845682144165 max : 330.85913 min: 28.925856 shape_disp: (320, 512, 1) depth_max : 3.0150537 depth_min: 0.2635956 Step 90: 0.10014571249485016 max : 332.61096 min: 16.913315 shape_disp: (320, 512, 1) depth_max : 5.156471 depth_min: 0.26220727 Step 91: 0.10243596881628036 max : 339.77344 min: 24.393148 shape_disp: (320, 512, 1) depth_max : 3.5753078 depth_min: 0.2566799 Step 92: 0.10243068635463715 max : 325.43582 min: 15.383979 shape_disp: (320, 512, 1) depth_max : 5.6690803 depth_min: 0.26798835 Step 93: 0.10007596760988235 max : 332.2273 min: 25.080872 shape_disp: (320, 512, 1) depth_max : 3.477272 depth_min: 0.2625101 Step 94: 0.09953679144382477 max : 331.67255 min: 23.796257 shape_disp: (320, 512, 1) depth_max : 3.6649888 depth_min: 0.26294914 Step 95: 0.09818486869335175 max : 329.05035 min: 29.062828 shape_disp: (320, 512, 1) depth_max : 3.000844 depth_min: 0.26504457 Step 96: 0.10310391336679459 max : 325.41806 min: 13.885269 shape_disp: (320, 512, 1) depth_max : 6.280974 depth_min: 0.268003 Step 97: 0.09774727374315262 max : 325.9686 min: 26.708965 shape_disp: (320, 512, 1) depth_max : 3.2653086 depth_min: 0.26755035 Step 98: 0.10923965275287628 max : 325.73315 min: 23.08259 shape_disp: (320, 512, 1) depth_max : 3.7783027 depth_min: 0.26774374 Step 99: 0.10516759008169174 max : 345.40063 min: 25.015917 shape_disp: (320, 512, 1) depth_max : 3.486301 depth_min: 0.25249812

any suggestion

thanks

AlessioTonioni commented 5 years ago

because for calibration I am using linear regression. so that if I get 4.08 m i can map to 4m.but due to too much variation I unable to use the linear regression.

What do you mean? Is your stereo system properly calibrated with something like opencv?

I used diy camera

Check that the cameras are properly synchronized and that the rectification is properly executed.

With the command you are using: python3.7 Live_Adaptation_Demo.py --weights /home/giuser/MyWorkspace/Real-time-self-adaptive-deep-stereo/pretrained_nets/MADNet/synthetic/weights.ckpt you are performing only inference, so if the cameras are properly calibrated and synchronized the output for a static scene should be more or less the same always.

vigneshgig commented 5 years ago

cameras are properly synchronized

yes I calibrated the camera with opencv and can you tell me some reference for cameras synchronise check

and also I checked the dispnet but it's working good without much variation only 0.001 variation only there. and i am using usb camera so I can't synchronise the frame perfect but surely some ms delay will be there . In stationary depth detection I think synchronisation of camera does not matter. my question is dispnet working perflectly without much variation but in madnet variation is high

AlessioTonioni commented 5 years ago

Dispnet is definitely more robust than MADNet, but I think it is still strange that you get important variation in the estimated depth on a static scene. Can you share a sequence of stereo couples that cause this behavior?

vigneshgig commented 5 years ago

Real-time-self-adaptive-deep-stereo.zip here I attached the file. I have attached madnet kitti and madnet synthetic .I found that madnet kitti variation less but structure of the image is not good but in synthetic model variation is high but image structure is good which means it preserve the original left image structure. please the check the uploaded file. @AlessioTonioni @mattpoggi
thanks

vigneshgig commented 5 years ago

and can you tell purpose of cropShape.

vigneshgig commented 5 years ago

I think variation is due to light or brightness flicker. I will try to normalise and check whether it will work or not

vigneshgig commented 5 years ago

Hi , I applied grayscale then histogram clahe equalisation using opencv and also histogram matching using sckit between the frame left and frame right.Now I am getting less variation only.so it's due to the brightness flicker.I am getting less variation in synthetic madnet model.If you have any suggestion regarding this problem please share. Thanks

AlessioTonioni commented 5 years ago

here I attached the file.

yeah as you said there is a lot of flickering in those images... Maybe you can try in an environment with better illumination and check if the quality improves?

and can you tell purpose of cropShape

Apply a central crop to the image before feeding them into the network