Closed fzh0917 closed 5 years ago
I guess your training checkpoint may be regressing image coordinates which are totally out of image bounds. In the next frame, GOTURN crops the image at the previous predicted box, which could be causing this issue.
I tested the PIL crop method for some cases when your box had coordinates out of image but it seemed to work fine and padded the out of box area with zeros.
Can you check bounding box values? Just print self.prev_rect
in the test
method of test.py
.
I'll look into this issue in detail and get back. Thanks!
Here is the stack trace.
$ python test.py -w ../saved_checkpoints/exp3/final_model.pth
Namespace(data_directory='../data/OTB/Man', model_weights='../saved_checkpoints/exp3/final_model.pth')
[69. 48. 95. 87.]
frame: 1 [69.91484090260097, 44.97729134559631, 94.00789887564522, 79.06419345310756]
[69.91484090260097, 44.97729134559631, 94.00789887564522, 79.06419345310756]
frame: 2 [70.72335313629264, 42.45427948015288, 93.20678849602302, 72.32945901253103]
[70.72335313629264, 42.45427948015288, 93.20678849602302, 72.32945901253103]
frame: 3 [71.44596422895413, 40.18087777316183, 92.52283250902349, 66.50209050907307]
[71.44596422895413, 40.18087777316183, 92.52283250902349, 66.50209050907307]
frame: 4 [72.0819581102108, 38.194360139916625, 91.982948297391, 61.4136520977397]
[72.0819581102108, 38.194360139916625, 91.982948297391, 61.4136520977397]
frame: 5 [72.70461372655708, 36.44987292098039, 91.53922051077595, 57.00444798650577]
[72.70461372655708, 36.44987292098039, 91.53922051077595, 57.00444798650577]
frame: 6 [73.25235647347078, 34.8670774288637, 91.09012486983683, 53.06942336212048]
[73.25235647347078, 34.8670774288637, 91.09012486983683, 53.06942336212048]
frame: 7 [73.7997159742809, 33.39063802152363, 90.64984497727374, 49.52346064112334]
[73.7997159742809, 33.39063802152363, 90.64984497727374, 49.52346064112334]
frame: 8 [74.33964891548749, 32.07947825142874, 90.19446878277232, 46.32401496608712]
[74.33964891548749, 32.07947825142874, 90.19446878277232, 46.32401496608712]
frame: 9 [74.80704545543891, 30.894863398402784, 89.74317647908586, 43.44246254765723]
[74.80704545543891, 30.894863398402784, 89.74317647908586, 43.44246254765723]
frame: 10 [75.23365582047252, 29.825739531553893, 89.33571490345336, 40.90703973116146]
[75.23365582047252, 29.825739531553893, 89.33571490345336, 40.90703973116146]
frame: 11 [75.63508291919808, 28.880748411465042, 88.93379260174657, 38.66743267108161]
[75.63508291919808, 28.880748411465042, 88.93379260174657, 38.66743267108161]
frame: 12 [76.02298592783066, 28.04869420987427, 88.55650949052102, 36.688339074899595]
[76.02298592783066, 28.04869420987427, 88.55650949052102, 36.688339074899595]
frame: 13 [76.39282059174934, 27.309084686451303, 88.18895100018156, 34.930228446314416]
[76.39282059174934, 27.309084686451303, 88.18895100018156, 34.930228446314416]
frame: 14 [76.74394839996826, 26.663149104813172, 87.85063656222403, 33.38392899976636]
[76.74394839996826, 26.663149104813172, 87.85063656222403, 33.38392899976636]
frame: 15 [77.07216133135084, 26.096040907341003, 87.53302146845527, 32.0215889038125]
[77.07216133135084, 26.096040907341003, 87.53302146845527, 32.0215889038125]
frame: 16 [77.386230541801, 25.592030039023115, 87.23948018134384, 30.823323999482568]
[77.386230541801, 25.592030039023115, 87.23948018134384, 30.823323999482568]
frame: 17 [77.68317988020497, 25.147977970279985, 86.96296215161996, 29.765176110268104]
[77.68317988020497, 25.147977970279985, 86.96296215161996, 29.765176110268104]
frame: 18 [77.96767286954294, 24.760622054184363, 86.70353015439692, 28.83135059953726]
[77.96767286954294, 24.760622054184363, 86.70353015439692, 28.83135059953726]
frame: 19 [78.2374775743712, 24.416588343537338, 86.45592561789685, 28.007621194959516]
[78.2374775743712, 24.416588343537338, 86.45592561789685, 28.007621194959516]
frame: 20 [78.48802745934606, 24.114571811008506, 86.22119925553922, 27.28143483651797]
[78.48802745934606, 24.114571811008506, 86.22119925553922, 27.28143483651797]
frame: 21 [78.72421819885221, 23.851223058567243, 86.00152334882918, 26.64284184874425]
[78.72421819885221, 23.851223058567243, 86.00152334882918, 26.64284184874425]
frame: 22 [78.94970312784183, 23.616645286118416, 85.79674246102009, 26.07970291158498]
[78.94970312784183, 23.616645286118416, 85.79674246102009, 26.07970291158498]
frame: 23 [79.15962786768203, 23.409738777281277, 85.60566052336529, 25.583585537900294]
[79.15962786768203, 23.409738777281277, 85.60566052336529, 25.583585537900294]
frame: 24 [79.35937892253877, 23.22762385592658, 85.42669871139493, 25.146010751028438]
[79.35937892253877, 23.22762385592658, 85.42669871139493, 25.146010751028438]
frame: 25 [79.54899423843463, 23.065746749931, 85.25849664262529, 24.758920644553815]
[79.54899423843463, 23.065746749931, 85.25849664262529, 24.758920644553815]
frame: 26 [79.7236723581475, 22.923202948105345, 85.1003472269544, 24.416502399705664]
[79.7236723581475, 22.923202948105345, 85.1003472269544, 24.416502399705664]
frame: 27 [79.89399317493114, 22.797559131379458, 84.94873224929889, 24.113369222208924]
[79.89399317493114, 22.797559131379458, 84.94873224929889, 24.113369222208924]
frame: 28 [80.06665600775062, 22.687128189394073, 84.81498058002798, 23.848191262701434]
[80.06665600775062, 22.687128189394073, 84.81498058002798, 23.848191262701434]
frame: 29 [80.23123729815418, 22.590855797908404, 84.68723050483973, 23.61456645982426]
[80.23123729815418, 22.590855797908404, 84.68723050483973, 23.61456645982426]
frame: 30 [80.38573255664596, 22.506789818911205, 84.57033950654784, 23.410034145371007]
[80.38573255664596, 22.506789818911205, 84.57033950654784, 23.410034145371007]
frame: 31 [80.5218327411521, 22.434465890459414, 84.4600807250996, 23.23291744222767]
[80.5218327411521, 22.434465890459414, 84.4600807250996, 23.23291744222767]
frame: 32 [80.64565831005498, 22.369563293879104, 84.35613235680012, 23.07597268367757]
[80.64565831005498, 22.369563293879104, 84.35613235680012, 23.07597268367757]
frame: 33 [80.75870367742401, 22.311938800636874, 84.25630334716838, 22.93665367210166]
[80.75870367742401, 22.311938800636874, 84.25630334716838, 22.93665367210166]
frame: 34 [80.8659812389454, 22.262185776192485, 84.15678325158495, 22.813818156579764]
[80.8659812389454, 22.262185776192485, 84.15678325158495, 22.813818156579764]
frame: 35 [80.95911590992418, 22.218244484276283, 84.05661732407158, 22.70510316150855]
[80.95911590992418, 22.218244484276283, 84.05661732407158, 22.70510316150855]
frame: 36 [81.05086601272814, 22.17861290148088, 83.97064220051529, 22.60939950009209]
[81.05086601272814, 22.17861290148088, 83.97064220051529, 22.60939950009209]
frame: 37 [81.13862519184306, 22.143507996368758, 83.89177994640234, 22.524893682217925]
[81.13862519184306, 22.143507996368758, 83.89177994640234, 22.524893682217925]
frame: 38 [81.21971696736883, 22.112401465900792, 83.81743946579138, 22.450173329689903]
[81.21971696736883, 22.112401465900792, 83.81743946579138, 22.450173329689903]
frame: 39 [81.29651923607176, 22.084809133792607, 83.74765092877834, 22.38400236137473]
[81.29651923607176, 22.084809133792607, 83.74765092877834, 22.38400236137473]
frame: 40 [81.36936904024294, 22.060016487057496, 83.68271358699286, 22.324960707705614]
[81.36936904024294, 22.060016487057496, 83.68271358699286, 22.324960707705614]
frame: 41 [81.44074029348299, 22.038069789251153, 83.62340303218613, 22.272738047157365]
[81.44074029348299, 22.038069789251153, 83.62340303218613, 22.272738047157365]
frame: 42 [81.50821557565716, 22.018658692662488, 83.56703705347559, 22.226560267051987]
[81.50821557565716, 22.018658692662488, 83.56703705347559, 22.226560267051987]
frame: 43 [81.57038371919289, 22.001486975539407, 83.5132089237723, 22.185696625997284]
[81.57038371919289, 22.001486975539407, 83.5132089237723, 22.185696625997284]
frame: 44 [81.6285503023113, 21.98621364654747, 83.46244238136993, 22.14940888847398]
[81.6285503023113, 21.98621364654747, 83.46244238136993, 22.14940888847398]
frame: 45 [81.68421155751, 21.97273800827213, 83.41474840309357, 22.11729447270698]
[81.68421155751, 21.97273800827213, 83.41474840309357, 22.11729447270698]
frame: 46 [81.73556295155772, 21.960748081612895, 83.36917465460701, 22.08882916410996]
[81.73556295155772, 21.960748081612895, 83.36917465460701, 22.08882916410996]
frame: 47 [81.78457982454557, 21.950148231665906, 83.32665152675034, 22.06361473307373]
[81.78457982454557, 21.950148231665906, 83.32665152675034, 22.06361473307373]
frame: 48 [81.83296773319475, 21.94068234164813, 83.28685553566382, 22.04111915426526]
[81.83296773319475, 21.94068234164813, 83.28685553566382, 22.04111915426526]
frame: 49 [81.87906151982254, 21.93231684342518, 83.2502234669888, 22.021251375213243]
[81.87906151982254, 21.93231684342518, 83.2502234669888, 22.021251375213243]
frame: 50 [81.92280018100665, 21.92490965608685, 83.2152249910795, 22.003589902347866]
[81.92280018100665, 21.92490965608685, 83.2152249910795, 22.003589902347866]
frame: 51 [81.96434503062042, 21.918329896427714, 83.18234014229698, 21.987950695836513]
[81.96434503062042, 21.918329896427714, 83.18234014229698, 21.987950695836513]
frame: 52 [82.00338853170223, 21.91253297287023, 83.15160232773344, 21.974120641239207]
[82.00338853170223, 21.91253297287023, 83.15160232773344, 21.974120641239207]
frame: 53 [82.0400352030239, 21.90740114404165, 83.12223352464179, 21.96188342626112]
[82.0400352030239, 21.90740114404165, 83.12223352464179, 21.96188342626112]
Traceback (most recent call last):
File "test.py", line 128, in <module>
tester.test()
File "test.py", line 114, in test
sample = self[i]
File "test.py", line 78, in __getitem__
sample = self._get_sample(idx)
File "test.py", line 87, in _get_sample
prev_img = self.transform_prev({'image': prev, 'bb': prevbb})['image']
File ....../tensorflow/lib/python3.6/site-packages/torchvision/transforms/transforms.py", line 49, in __call__
img = t(img)
File "....../src/helper.py", line 27, in __call__
h, w = image.shape[:2]
ValueError: not enough values to unpack (expected 2, got 0)
For some reasons, I replaced my truth path with some dots in the stack trace.
From the information in stack trace, you can see that the bounding box became smaller and smaller instead of out of the ranges.
Interesting. Two questions:
I'll try to reproduce the error.
Two answers:
num_batches
argument is 50.Here is the related code piece in train.py
:
parser.add_argument('-n', '--num-batches', default=50, type=int,
help='number of total batches to run')
Thanks! I feel crop operations in src/helper.py
need updates to handle such exceptions (just like original GOTURN). I didn't encounter this issue because I tested the final model. I'll work on it and get back as soon as I have some reasonable results.
OK. Thank you for your timely and detailed responses.
The code is latest. The problem description and exception infomation are same as the issue #17. However, the code worked finely when using the pretrained model file which was downloaded from Google Drive. So, I am thinking that whether the number of batch is too small when training the model manually. P.S. My
num_batch
is 50 when training the model manually by executing the commandpython train.py
.