search

hulop / SfMLocalization

HULOP Localization using Structure from Motion
MIT License
9 stars 6 forks source link

after running for a while, the engine will fail to localize #5

Open colegleason opened 8 years ago

colegleason commented 8 years ago

I'm not sure why this happens yet, but the same engine submitted multiple times will stop working.

tatsuya-ishihara commented 8 years ago

Did you send multiple request at the same time? Current server implementation assumes same user ID does not send multiple request at the same time.

If you have any errors on node.js console, please add as comment of this issue.

colegleason commented 8 years ago

Usually it would just return an empty t and R matrix.

I did just get this, but I am unsure if it is related:

successed to create working folder : /tmp/vision-localize-server/1744B076-525E-4F38-BF83-5E3C4F76ADFB
calc bof, thread id=0
OpenCV Error: Assertion failed (ifs.is_open()) in readMatBin, file ../../VisionLocalizeCommon/src/FileUtils.cpp, line 61
terminate called after throwing an instance of 'cv::Exception'
  what():  ../../VisionLocalizeCommon/src/FileUtils.cpp:61: error: (-215) ifs.is_open() in function readMatBin
colegleason commented 8 years ago

Ok, it just happened again. I tried to localize like this:

curl http://hulop.qolt.cs.cmu.edu:3000/localize -F "map=cole-qolt" -F "user=1-shrink-0.75" -F "image=@/Users/cole/Desktop/test.jpg"
{"estimate":{"t":[],"R":[]}}

Server output:

successed to create working folder : /tmp/vision-localize-server/1C7E5314-E3A6-4853-A4DB-7E268969C750
calc bof, thread id=0
number of selected local views by bow : 20
Extract features from query image
Cannot read image describer /tmp/vision-localize-server/1C7E5314-E3A6-4853-A4DB-7E268969C750/image_describer.txt
image ID :3BAEF952-076C-4136-A381-771A039FEF38
image size : [1047 x 561]
Init AKAZE : 7.22e-07 s
Extract feature : 0.123998 s
Write feature : 0.0013943 s
Load regions : 0.000495413 s
Total AKAZE time : 0.125888 s
query image size : 1047 x 561
query image index : 3047
query image path : 3BAEF952-076C-4136-A381-771A039FEF38
size pairs: 20
Start putative matching
number of putative matches : 0
Not enough putative matches
Load SfM views : 0.00234212 s
Select views by iBeacon : 5.46e-07 s
Select views by BoW : 0.274666 s
Load image describer : 6.2581e-05 s
Generate UUID : 2.3352e-05 s
Extract feature from query image : 0.125938 s
Putative matching : 0.0185598 s
successed to delete working folder : /tmp/vision-localize-server/1C7E5314-E3A6-4853-A4DB-7E268969C750
localization result :
POST /localize 200 1240ms

Image: test

Note that this image normally works. Whenever the server gets into whatever this state is, it won't localize at all. After restarting it, localization works. After restart:

curl http://hulop.qolt.cs.cmu.edu:3000/localize -F "map=cole-qolt" -F "user=1-shrink-0.75" -F "image=@/Users/cole/Desktop/test.jpg"
{"estimate":{"t":[-0.6510856318774748,2.7725730464592004,1.636260310992164],"R":[[-4.556697967179266,-0.0741017045632142,-0.17781457251754965],[0.27322900253480975,1.0432259874850542,-4.027952789676068],[0.20161805505598154,-3.8942854113586405,-1.4521176407850531]]}}

Server log after restart:

Express server listening on port 3000
load new localize engine, userID : 1-shrink-0.75, mapID : cole-qolt
Reading sfm_data.json file : /home/cole/models/cole-qolt/Output/SfM/reconstruction/global/sfm_data.json

- Regions Loading -
0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Reading A mat file : /home/cole/models/cole-qolt/Ref/Amat.yml
Loaded A mat : [-4.55827143879, -0.020090816429, -0.149543633502, -0.276802070753;
 0.268931438382, 0.113303605003, -4.15959329733, 17.2513470723;
 0.117727647175, -4.12325416325, -0.547420911417, 1.67871031937]
find BOW model : /home/cole/models/cole-qolt/Output/matches/BOWfile.yml
loaded BOW model.
find PCA model for BOW : /home/cole/models/cole-qolt/Output/matches/PCAfile.yml
loaded PCA model.
successed to create working folder : /tmp/vision-localize-server/64CE54C3-5231-4990-93D1-AFB5E2D9A086
debug 0 0.75
intrinsicK[1446.7137018, 0, 788.688378214;
 0, 1455.30787024, 485.29516917;
 0, 0, 1]
intrinsicDist[0.231391014881;
 -0.37092594078;
 0.0207196766153;
 0.0203556409191;
 0.300448104293]
imageSize[1836 x 1377]
debug 1 0.75
calc bof, thread id=0
number of selected local views by bow : 20
Extract features from query image
Cannot read image describer /tmp/vision-localize-server/64CE54C3-5231-4990-93D1-AFB5E2D9A086/image_describer.txt
image ID :D96D0C5A-3A7A-4E21-BD2F-781F26FA7829
image size : [1748 x 1294]
Init AKAZE : 1.137e-06 s
Extract feature : 0.421999 s
Write feature : 0.00307005 s
Load regions : 0.00146673 s
Total AKAZE time : 0.426537 s
query image size : 1748 x 1294
query image index : 3047
query image path : D96D0C5A-3A7A-4E21-BD2F-781F26FA7829
size pairs: 20
Start putative matching
number of putative matches : 15

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
number of putative matches : 15
number of geometric matches : 12

number of geometric matches : 12
Match provider size 12
map to 3D size = 3042
mapFeatTo3DFeat size = 48
cpt = 48
  nfa=-15.8816 inliers=22/48 precisionNormalized=0.00471275 precision=91.6405 (iter=0,sample=5,21,23,)
  nfa=-69.5149 inliers=46/48 precisionNormalized=0.0046297 precision=90.8295 (iter=0,sample=5,21,23,)
  nfa=-82.6143 inliers=39/48 precisionNormalized=0.000601313 precision=32.7341 (iter=1,sample=21,3,34,)

-------------------------------
-- Robust Resection
-- Resection status: 1
-- #Points used for Resection: 48
-- #Points validated by robust Resection: 39
-- Threshold: 32.7341
-------------------------------
#inliers = 39
P =
  1333.33  -155.869   801.901  -2810.64
 -27.5323   1210.48   693.407  -1858.91
-0.00159495 -0.223006  0.974816  -3.47989
K =
    1334.91 1.19284e-13     814.339
-3.18425e-16     1334.91     406.043
1.82507e-19 1.22218e-17           1
R =
  0.999796   0.019277 0.00604576
-0.0201398   0.974627   0.222929
-0.00159495  -0.223006   0.974816
t =
0.0173575
-0.334053
-3.47989
-R't =
-0.029632
-0.450794
3.46662
global t =
[-0.6510856318774748;
 2.7725730464592;
 1.636260310992164]
global R =
[-4.556697967179266, -0.0741017045632142, -0.1778145725175496;
 0.2732290025348098, 1.043225987485054, -4.027952789676068;
 0.2016180550559815, -3.89428541135864, -1.452117640785053]
Load SfM views : 0.00224366 s
Select views by iBeacon : 3.59e-07 s
Select views by BoW : 0.262482 s
Load image describer : 6.6709e-05 s
Generate UUID : 2.1756e-05 s
Extract feature from query image : 0.426595 s
Putative matching : 0.024838 s
Geometric matching : 0.0241468 s
PnP : 0.0195074 s
Transform coordinate : 0.000311792 s
Total time: 0.760213 s
complete!
successed to delete working folder : /tmp/vision-localize-server/64CE54C3-5231-4990-93D1-AFB5E2D9A086
localization result : -0.6510856318774748,2.7725730464592004,1.636260310992164,-4.556697967179266,-0.0741017045632142,-0.17781457251754965,0.27322900253480975,1.0432259874850542,-4.027952789676068,0.20161805505598154,-3.8942854113586405,-1.4521176407850531
tatsuya-ishihara commented 8 years ago

Do you still see the same problem on CMU server? I tried to reproduce the same problem by using the same test image and same map on CMU server, but I could not for now. I will let you know if I can find more.

colegleason commented 8 years ago

I haven't seen it today. I will let you know when I do.

On Tue, Apr 26, 2016 at 1:22 AM tatsuya-ishihara notifications@github.com wrote:

Do you still see the same problem on CMU server? I tried to reproduce the same problem by using the same test image and same map on CMU server, but I could not for now. I will let you know if I can find more.

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/hulop/SfMLocalization/issues/5#issuecomment-214619187

colegleason commented 8 years ago

I just saw this again, but I don't have a good explanation of what is causing it.

colegleason commented 8 years ago 
successed to create working folder : /tmp/vision-localize-server/10CF2600-155B-4AD2-85B2-52C863634C2C
calc bof, thread id=0
number of selected local views by bow : 20
Extract features from query image
Cannot read image describer /tmp/vision-localize-server/10CF2600-155B-4AD2-85B2-52C863634C2C/image_describer.txt
image ID :C205DFC5-6CD3-4FF8-826E-98C799C171EA
image size : [1047 x 561]
Init AKAZE : 6.26e-07 s
Extract feature : 0.120047 s
Write feature : 0.000748502 s
Load regions : 0.0003723 s
Total AKAZE time : 0.121169 s
query image size : 1047 x 561
query image index : 3047
query image path : C205DFC5-6CD3-4FF8-826E-98C799C171EA
size pairs: 20
Start putative matching
number of putative matches : 4

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
number of putative matches : 4
number of geometric matches : 1

number of geometric matches : 1
Match provider size 1
map to 3D size = 3042
mapFeatTo3DFeat size = 10
cpt = 10
  nfa=-0.523855 inliers=7/10 precisionNormalized=0.0127141 precision=150.519 (iter=0,sample=1,3,9,)
  nfa=-15.5253 inliers=10/10 precisionNormalized=0.000604208 precision=32.8128 (iter=0,sample=1,3,9,)
  nfa=-18.1501 inliers=10/10 precisionNormalized=0.000254812 precision=21.3089 (iter=7,sample=5,2,4,)
  nfa=-18.183 inliers=10/10 precisionNormalized=0.000252069 precision=21.1938 (iter=76,sample=5,4,0,)

-------------------------------
-- Robust Resection
-- Resection status: 1
-- #Points used for Resection: 10
-- #Points validated by robust Resection: 10
-- Threshold: 21.1938
-------------------------------
Fail to estimate camera matrix
#inliers = 10
Load SfM views : 0.00240168 s
Select views by iBeacon : 6.36e-07 s
Select views by BoW : 0.276418 s
Load image describer : 6.0588e-05 s
Generate UUID : 2.2155e-05 s
Extract feature from query image : 0.121214 s
Putative matching : 0.0194548 s
Geometric matching : 0.00811709 s
PnP : 0.00489332 s
successed to delete working folder : /tmp/vision-localize-server/10CF2600-155B-4AD2-85B2-52C863634C2C
localization result :
POST /localize 200 1360ms
^Ccmaccess@cmaccess-server-01:~/SfMLocalization/VisionLocalizeServer$ node app.js
number of loaded maps : 12
map index : 0
map name : cmu-qolt
sfm data dir : /home/ishihara/test/CMU-video-data-undistort/NSH-4F-QOLT-merge/Output/SfM/reconstruction/global
match dir : /home/ishihara/test/CMU-video-data-undistort/NSH-4F-QOLT-merge/Output/matches
A mat file : /home/ishihara/test/CMU-video-data-undistort/NSH-4F-QOLT-merge/Ref/Amat.yml
map index : 1
map name : cole-qolt
sfm data dir : /home/cole/models/cole-qolt/Output/SfM/reconstruction/global
match dir : /home/cole/models/cole-qolt/Output/matches
A mat file : /home/cole/models/cole-qolt/Ref/Amat.yml
map index : 2
map name : cmu-entropy-chips
sfm data dir : /home/cmaccess/uc-chips/Output/final/Output/SfM/reconstruction/global
match dir : /home/cmaccess/uc-chips/Output/final/Output/matches
A mat file : /home/cmaccess/uc-chips/Ref/Amat.yml
map index : 3
map name : cmu-entropy-peanuts
sfm data dir : /home/cmaccess/SfMModels/uc-peanut/Output/final/Output/SfM/reconstruction/global
match dir : /home/cmaccess/SfMModels/uc-peanut/Output/final/Output/matches
A mat file : /home/cmaccess/SfMModels/uc-peanut/Ref/Amat.yml
map index : 4
map name : cmu-entropy-chips2
sfm data dir : /home/cmaccess/SfMModels/uc-chips/Output/final/Output/SfM/reconstruction/global
match dir : /home/cmaccess/SfMModels/uc-chips/Output/final/Output/matches
A mat file : /home/cmaccess/SfMModels/uc-chips/Ref/Amat.yml
map index : 5
map name : cmu-entropy-chips3
sfm data dir : /home/cmaccess/SfMModels/uc-chips2/Output/final/Output/SfM/reconstruction/global
match dir : /home/cmaccess/SfMModels/uc-chips2/Output/final/Output/matches
A mat file : /home/cmaccess/SfMModels/uc-chips2/Ref/Amat.yml
map index : 6
map name : cmu-shelf-office
sfm data dir : /home/cmaccess/SfMModels/shelf_office/Output/final/Output/SfM/reconstruction/global
match dir : /home/cmaccess/SfMModels/shelf_office/Output/final/Output/matches
A mat file : /home/cmaccess/SfMModels/shelf_office/Ref/Amat.yml
map index : 7
map name : cmu-shelf-office2
sfm data dir : /home/cmaccess/SfMModels/shelf_office2/Output/final/Output/SfM/reconstruction/global
match dir : /home/cmaccess/SfMModels/shelf_office2/Output/final/Output/matches
A mat file : /home/cmaccess/SfMModels/shelf_office2/Ref/Amat.yml
map index : 8
map name : thinklab-hakozaki
sfm data dir : /home/ishihara/test/ThinkLabHakozaki-undistort/Output/final/Output/SfM/reconstruction/global
match dir : /home/ishihara/test/ThinkLabHakozaki-undistort/Output/final/Output/matches
A mat file : /home/ishihara/test/ThinkLabHakozaki-undistort/Ref/Amat.yml
map index : 9
map name : thinklab-hakozaki-shrink-0.75
sfm data dir : /home/ishihara/test/ThinkLabHakozaki-shrink-0.75-undistort/Output/final/Output/SfM/reconstruction/global
match dir : /home/ishihara/test/ThinkLabHakozaki-shrink-0.75-undistort/Output/final/Output/matches
A mat file : /home/ishihara/test/ThinkLabHakozaki-shrink-0.75-undistort/Ref/Amat.yml
map index : 10
map name : thinklab-hakozaki-2-shrink-0.75
sfm data dir : /home/ishihara/test/ThinklabHakozaki2-shrink-0.75-undistort/Output/final/Output/SfM/reconstruction/global
match dir : /home/ishihara/test/ThinklabHakozaki2-shrink-0.75-undistort/Output/final/Output/matches
A mat file : /home/ishihara/test/ThinklabHakozaki2-shrink-0.75-undistort/Ref/Amat.yml
map index : 11
map name : hakozaki-office-3-shrink-0.75
sfm data dir : /home/ishihara/test/HakozakiOffice3-shrink-0.75-undistort/Output/final/Output/SfM/reconstruction/global
match dir : /home/ishihara/test/HakozakiOffice3-shrink-0.75-undistort/Output/final/Output/matches
A mat file : /home/ishihara/test/HakozakiOffice3-shrink-0.75-undistort/Ref/Amat.yml
number of loaded users : 3
user index : 0
user name : 1
K mat file : /home/cmaccess/SfMLocalization/VisionLocalizeServer/config/camera/iphone6-1920x1080/K.yml
Dist mat file : /home/cmaccess/SfMLocalization/VisionLocalizeServer/config/camera/iphone6-1920x1080/dist.yml
do not scale input image when server localizes
user index : 1
user name : 1-shrink-0.75
K mat file : /home/cmaccess/SfMLocalization/VisionLocalizeServer/config/camera/iphone6-1920x1080-shrink-0.75/K.yml
Dist mat file : /home/cmaccess/SfMLocalization/VisionLocalizeServer/config/camera/iphone6-1920x1080-shrink-0.75/dist.yml
scale image : 0.75
user index : 2
user name : 2-shrink-0.75
K mat file : /home/cmaccess/SfMLocalization/VisionLocalizeServer/config/camera/iphone5c-1280x720-shrink-0.75/K.yml
Dist mat file : /home/cmaccess/SfMLocalization/VisionLocalizeServer/config/camera/iphone5c-1280x720-shrink-0.75/dist.yml
scale image : 0.75
Express server listening on port 3000
load new localize engine, userID : 1-shrink-0.75, mapID : cole-qolt
Reading sfm_data.json file : /home/cole/models/cole-qolt/Output/SfM/reconstruction/global/sfm_data.json

- Regions Loading -
0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Reading A mat file : /home/cole/models/cole-qolt/Ref/Amat.yml
Loaded A mat : [-4.55827143879, -0.020090816429, -0.149543633502, -0.276802070753;
 0.268931438382, 0.113303605003, -4.15959329733, 17.2513470723;
 0.117727647175, -4.12325416325, -0.547420911417, 1.67871031937]
find BOW model : /home/cole/models/cole-qolt/Output/matches/BOWfile.yml
loaded BOW model.
find PCA model for BOW : /home/cole/models/cole-qolt/Output/matches/PCAfile.yml
loaded PCA model.
successed to create working folder : /tmp/vision-localize-server/1945F745-EE1D-48B0-A141-01790984428B
debug 0 0.75
intrinsicK[1446.7137018, 0, 788.688378214;
 0, 1455.30787024, 485.29516917;
 0, 0, 1]
intrinsicDist[0.231391014881;
 -0.37092594078;
 0.0207196766153;
 0.0203556409191;
 0.300448104293]
imageSize[2448 x 1836]
debug 1 0.75
calc bof, thread id=0
number of selected local views by bow : 20
Extract features from query image
Cannot read image describer /tmp/vision-localize-server/1945F745-EE1D-48B0-A141-01790984428B/image_describer.txt
image ID :EABBD0A3-D2FE-419F-BFE9-79A323F72637
image size : [1573 x 1044]
Init AKAZE : 6.38e-07 s
Extract feature : 0.279197 s
Write feature : 0.00525871 s
Load regions : 0.0022464 s
Total AKAZE time : 0.286703 s
query image size : 1573 x 1044
query image index : 3047
query image path : EABBD0A3-D2FE-419F-BFE9-79A323F72637
size pairs: 20
Start putative matching
number of putative matches : 17

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
number of putative matches : 17
number of geometric matches : 16

number of geometric matches : 16
Match provider size 16
map to 3D size = 3042
mapFeatTo3DFeat size = 135
cpt = 135
  nfa=-64.1114 inliers=59/135 precisionNormalized=0.00340537 precision=77.8991 (iter=0,sample=1,42,49,)
  nfa=-204.264 inliers=72/135 precisionNormalized=7.30473e-05 precision=11.4091 (iter=0,sample=1,42,49,)
  nfa=-205.429 inliers=128/135 precisionNormalized=0.00506106 precision=94.9667 (iter=9,sample=82,13,115,)
  nfa=-233.619 inliers=130/135 precisionNormalized=0.00339561 precision=77.7874 (iter=16,sample=78,75,129,)
  nfa=-241.538 inliers=131/135 precisionNormalized=0.00312941 precision=74.6761 (iter=24,sample=109,114,55,)
  nfa=-250.837 inliers=131/135 precisionNormalized=0.00264733 precision=68.6839 (iter=53,sample=43,88,92,)
  nfa=-254.872 inliers=128/135 precisionNormalized=0.00203564 precision=60.2283 (iter=116,sample=9,20,96,)
  nfa=-260.848 inliers=129/135 precisionNormalized=0.00194377 precision=58.8536 (iter=158,sample=83,132,98,)

-------------------------------
-- Robust Resection
-- Resection status: 1
-- #Points used for Resection: 135
-- #Points validated by robust Resection: 129
-- Threshold: 58.8536
-------------------------------
#inliers = 129
P =
 -1023.18  -72.6322  -1180.23   2193.91
  279.511   1229.21  -598.128   -608.02
 0.296733 -0.192306 -0.935397  -0.34729
K =
     1334.91 -8.10527e-14      814.339
-5.04319e-15      1334.91      406.043
-2.08392e-17 -1.00058e-17            1
R =
-0.947501 0.0629034 -0.313504
 0.119129  0.979317 -0.163545
 0.296733 -0.192306 -0.935397
t =
1.85536
-0.349841
-0.34729
-R't =
1.90268
0.159111
0.199594
global t =
[-8.982778752934813;
 16.95083633617174;
 1.137389691595662]
global R =
[4.272198752318703, -0.277647916828848, 1.572207016865635;
 -1.475602510626786, 0.9277921033054446, 3.788030790308539;
 -0.7651818676185087, -3.925294463264489, 1.149484420972046]
Load SfM views : 0.00218051 s
Select views by iBeacon : 3.78e-07 s
Select views by BoW : 0.262038 s
Load image describer : 6.4386e-05 s
Generate UUID : 2.1497e-05 s
Extract feature from query image : 0.290689 s
Putative matching : 0.0484586 s
Geometric matching : 0.0361338 s
PnP : 0.0569057 s
Transform coordinate : 0.000309396 s
Total time: 0.696802 s
complete!
successed to delete working folder : /tmp/vision-localize-server/1945F745-EE1D-48B0-A141-01790984428B
localization result : -8.982778752934813,16.95083633617174,1.1373896915956616,4.272198752318703,-0.277647916828848,1.5722070168656355,-1.475602510626786,0.9277921033054446,3.7880307903085386,-0.7651818676185087,-3.9252944632644886,1.149484420972046
POST /localize 200 7365ms
colegleason commented 8 years ago

Hmm, the image size seems off.