search

MCG-NJU / MOTIP

Multiple Object Tracking as ID Prediction
https://arxiv.org/abs/2403.16848
Apache License 2.0
110 stars 10 forks source link

MOT17验证id不准确 #29

Closed ATun-cell closed 2 months ago

ATun-cell commented 2 months ago

你好，为什么我的submit出来的结果是这样的，用的是项目提供的权重r50_deformable_detr_motip_mot17.pth 看样子是id_decoder把所有的目标都当成了新出现的目标 提交

HELLORPG commented 2 months ago

可否方便上传一下 inference output dir 下的 config.yaml 文件，该文件应该是会被自动生成。

ATun-cell commented 2 months ago

ACCUMULATE_STEPS: 2 AREA_THRESH: 100 AUG_MAX_SIZE: 1536 AUG_OVERFLOW_BBOX: false AUG_RANDOM_CROP_MAX: 600 AUG_RANDOM_CROP_MIN: 384 AUG_RANDOM_RESIZE:

ATun-cell commented 2 months ago

配置 这是设置的配置

HELLORPG commented 2 months ago

有点奇怪，比较重要的参数我都没有看到有什么异常。 但是相同的代码我进行过验证并且上个月有另一位开发者也验证过这个 inference 流程。

我稍后有时间的话重新 download 一份代码从 0 验证一下。因为近期的服务器的网络服务有点问题，可能不能立即验证。

ATun-cell commented 2 months ago

好的 感谢抽空答疑！

HELLORPG commented 2 months ago

不用谢。

您有时间的话也可以试一下，在 debug 模式下，看看模型有没有进入到 ID Decoder 的 inference 推理，如下： https://github.com/MCG-NJU/MOTIP/blob/55e108c0beaaa67d1979cc6f8c6ccdb0c28a48f4/models/motip.py#L166-L172 我担心因为某些我现在不知道的原因，直接跳过了 ID Decoder 而选择直接累加 ID index。

ATun-cell commented 2 months ago

这个我试过，seq_decoder返回的pred_id_word经过softmax后都是id_word的最后一项值最大，所以被后面全部处理成新出现的目标

ATun-cell commented 2 months ago

模型预训练加载的时候id decoder的参数应该也是正确加载了

HELLORPG commented 2 months ago

您好，抱歉回复晚了，最近个人的问题比较多。

我重新 clone 了 GitHub code 并且下载了 MOT17 对应的 checkpoint，使用如下的指令进行推理：

python -m torch.distributed.run --nproc_per_node=2 main.py --mode submit --use-distributed Tru
e --use-wandb False --config-path ./configs/r50_deformable_detr_motip_mot17.yaml --data-root ./datasets/ --inference-model ./outputs/r50_deformable_detr_motip_
mot17/r50_deformable_detr_motip_mot17.pth --outputs-dir ./outputs/r50_deformable_detr_motip_mot17/ --inference-dataset MOT17 --inference-split train

得到的结果应该是正常的 ID 预测结果，我截取一段如下所示：

1,0,1412.5601806640625,428.670166015625,185.955322265625,337.4351806640625,1,-1,-1,-1
1,1,971.6073608398438,458.11712646484375,28.099853515625,72.24517822265625,1,-1,-1,-1
1,2,1099.1800537109375,439.2057800292969,37.634765625,107.20303344726562,1,-1,-1,-1
1,3,579.880126953125,431.81097412109375,19.0091552734375,41.77874755859375,1,-1,-1,-1
1,4,1485.362548828125,437.5739440917969,45.010009765625,105.64254760742188,1,-1,-1,-1
1,5,420.20428466796875,457.01171875,38.8935546875,89.07220458984375,1,-1,-1,-1
1,6,584.0565795898438,446.4095458984375,85.336669921875,263.5157470703125,1,-1,-1,-1
1,7,1054.74365234375,483.7979736328125,36.0771484375,109.787353515625,1,-1,-1,-1
1,8,581.8377075195312,457.2114562988281,36.58203125,127.94375610351562,1,-1,-1,-1
1,9,595.5172119140625,427.34832763671875,17.8084716796875,42.4259033203125,1,-1,-1,-1
1,10,442.60888671875,444.6185302734375,107.65435791015625,279.74267578125,1,-1,-1,-1
1,11,936.2731323242188,433.1972351074219,39.6700439453125,116.06289672851562,1,-1,-1,-1
1,12,1256.7777099609375,446.5997314453125,35.844482421875,101.49517822265625,1,-1,-1,-1
1,13,633.052978515625,457.9331359863281,67.783447265625,186.02877807617188,1,-1,-1,-1
1,14,1031.5513916015625,447.3587646484375,25.5400390625,73.8446044921875,1,-1,-1,-1
1,15,549.1532592773438,461.5870666503906,32.3531494140625,96.79916381835938,1,-1,-1,-1
1,16,665.3746337890625,453.5153503417969,32.8695068359375,84.91073608398438,1,-1,-1,-1
1,17,1089.371826171875,483.4592590332031,33.43212890625,114.36465454101562,1,-1,-1,-1
1,18,476.8069763183594,460.22833251953125,78.14419555664062,240.7255859375,1,-1,-1,-1
1,19,1015.5053100585938,430.1152038574219,42.49163818359375,122.78543090820312,1,-1,-1,-1
1,20,1361.1317138671875,434.85888671875,42.052490234375,120.24420166015625,1,-1,-1,-1
1,21,1338.0780029296875,416.2598571777344,164.052734375,369.5212707519531,1,-1,-1,-1
2,1,972.0623779296875,459.18206787109375,27.1136474609375,67.62750244140625,1,-1,-1,-1
2,0,1418.21142578125,428.408447265625,181.8203125,339.64404296875,1,-1,-1,-1
2,2,1100.1107177734375,439.2425842285156,37.00048828125,106.71224975585938,1,-1,-1,-1
2,3,580.4443969726562,431.5030212402344,18.99072265625,41.3995361328125,1,-1,-1,-1
2,4,1492.1558837890625,438.90185546875,46.429931640625,113.3199462890625,1,-1,-1,-1
2,5,421.7413330078125,457.28912353515625,37.234619140625,87.9195556640625,1,-1,-1,-1
2,6,585.148193359375,447.140380859375,84.16552734375,262.8447265625,1,-1,-1,-1
2,7,1055.1640625,484.11407470703125,36.018798828125,109.2440185546875,1,-1,-1,-1
2,8,582.0967407226562,457.4110107421875,36.116455078125,132.5318603515625,1,-1,-1,-1
2,9,595.8186645507812,427.1788635253906,17.83935546875,42.308349609375,1,-1,-1,-1
2,10,443.2992858886719,445.9121398925781,109.50656127929688,277.4842224121094,1,-1,-1,-1
2,11,936.298095703125,433.25634765625,39.745849609375,116.19970703125,1,-1,-1,-1
2,12,1257.2198486328125,446.3704833984375,35.06787109375,101.827392578125,1,-1,-1,-1
2,13,633.1276245117188,458.392333984375,68.583251953125,185.328369140625,1,-1,-1,-1
2,14,1032.1461181640625,448.584228515625,25.361083984375,73.21343994140625,1,-1,-1,-1
2,15,550.1680908203125,462.6790771484375,31.6500244140625,95.44097900390625,1,-1,-1,-1
2,16,666.2775268554688,452.970703125,33.16748046875,86.91522216796875,1,-1,-1,-1
2,17,1089.364013671875,483.2498779296875,33.98193359375,114.663330078125,1,-1,-1,-1
2,18,473.35015869140625,460.5018615722656,83.04833984375,245.53848266601562,1,-1,-1,-1
2,19,1015.4288330078125,430.6607971191406,42.861083984375,122.56338500976562,1,-1,-1,-1
2,20,1364.80419921875,434.2622985839844,45.36962890625,122.55545043945312,1,-1,-1,-1
2,21,1341.6326904296875,417.5187683105469,169.38623046875,373.1272277832031,1,-1,-1,-1
3,1,973.2008056640625,459.4092712402344,25.409423828125,65.28506469726562,1,-1,-1,-1
3,0,1424.7177734375,429.39324951171875,185.715087890625,340.572265625,1,-1,-1,-1
3,2,1099.9713134765625,439.4538269042969,37.347900390625,107.38485717773438,1,-1,-1,-1
3,3,580.8353881835938,430.91009521484375,18.4119873046875,41.70941162109375,1,-1,-1,-1
3,4,1490.152587890625,437.3462219238281,46.20654296875,110.31582641601562,1,-1,-1,-1
3,5,421.1836853027344,458.85748291015625,37.15985107421875,85.57757568359375,1,-1,-1,-1
3,6,585.4877319335938,446.84716796875,83.374755859375,262.1976318359375,1,-1,-1,-1
3,7,1055.6746826171875,484.51641845703125,35.586181640625,108.6893310546875,1,-1,-1,-1
3,8,581.4363403320312,456.3325500488281,36.575439453125,133.70748901367188,1,-1,-1,-1
3,9,595.90185546875,427.7950134277344,17.7264404296875,42.00048828125,1,-1,-1,-1
3,10,443.88079833984375,445.0120849609375,111.2088623046875,278.826171875,1,-1,-1,-1
3,11,936.4533081054688,433.1448669433594,39.84716796875,115.99380493164062,1,-1,-1,-1
3,12,1257.4488525390625,446.7004089355469,34.565185546875,101.43698120117188,1,-1,-1,-1
3,13,633.201416015625,457.98114013671875,69.3275146484375,186.240478515625,1,-1,-1,-1
3,14,1032.5606689453125,448.18212890625,25.28564453125,73.35504150390625,1,-1,-1,-1
3,15,550.1107177734375,462.2453918457031,32.25244140625,96.65145874023438,1,-1,-1,-1
3,16,666.8836059570312,451.6226501464844,33.7386474609375,87.22628784179688,1,-1,-1,-1
3,17,1089.4010009765625,484.1846923828125,33.2822265625,113.641845703125,1,-1,-1,-1
3,18,474.5164489746094,459.5653076171875,82.86459350585938,243.13525390625,1,-1,-1,-1
3,19,1014.8866577148438,430.78692626953125,43.13861083984375,121.84600830078125,1,-1,-1,-1
3,21,1344.7738037109375,418.7888488769531,173.486083984375,375.7941589355469,1,-1,-1,-1
4,1,974.2149047851562,460.5390625,25.522216796875,64.5203857421875,1,-1,-1,-1
4,0,1432.00439453125,428.7790222167969,178.14794921875,341.2529602050781,1,-1,-1,-1
4,10,447.5435791015625,449.1568298339844,110.59869384765625,274.0600891113281,1,-1,-1,-1
4,2,1100.378173828125,439.2999572753906,37.43603515625,107.62930297851562,1,-1,-1,-1
4,3,580.8155517578125,430.35626220703125,18.191650390625,41.09783935546875,1,-1,-1,-1
4,4,1490.8768310546875,436.5765380859375,46.1259765625,108.9727783203125,1,-1,-1,-1
4,5,422.1006774902344,458.01165771484375,36.4952392578125,85.5645751953125,1,-1,-1,-1
4,6,584.9765625,446.6094970703125,84.8699951171875,262.9403076171875,1,-1,-1,-1
4,7,1055.4815673828125,484.4241638183594,36.059814453125,109.21109008789062,1,-1,-1,-1
4,8,581.1541748046875,455.8271179199219,36.849609375,134.31619262695312,1,-1,-1,-1
4,9,595.6459350585938,428.1362609863281,17.795654296875,41.0274658203125,1,-1,-1,-1
4,11,935.6595458984375,433.4529113769531,40.0128173828125,115.75534057617188,1,-1,-1,-1
4,12,1257.7301025390625,447.0507507324219,34.48779296875,101.21273803710938,1,-1,-1,-1
4,13,637.2788696289062,456.55682373046875,64.1771240234375,187.75927734375,1,-1,-1,-1
4,14,1031.8870849609375,448.3743591308594,25.260498046875,73.00625610351562,1,-1,-1,-1
4,15,549.8311157226562,461.3452453613281,32.169921875,97.74008178710938,1,-1,-1,-1
4,16,668.008056640625,450.4544677734375,32.547119140625,85.95965576171875,1,-1,-1,-1
4,17,1089.7706298828125,483.99346923828125,33.696533203125,113.9298095703125,1,-1,-1,-1
4,18,476.4494323730469,458.8016357421875,82.34585571289062,243.5888671875,1,-1,-1,-1
4,20,1372.028076171875,433.53668212890625,48.6923828125,125.67041015625,1,-1,-1,-1
4,19,1015.1968383789062,430.2268981933594,42.02642822265625,121.42117309570312,1,-1,-1,-1
4,21,1348.90576171875,417.52899169921875,173.5478515625,379.29833984375,1,-1,-1,-1
5,1,974.8778686523438,461.5888977050781,27.239501953125,65.79849243164062,1,-1,-1,-1
5,0,1442.66650390625,427.4998779296875,171.45947265625,342.6715087890625,1,-1,-1,-1
5,2,1100.537841796875,439.55865478515625,37.0302734375,107.05670166015625,1,-1,-1,-1
5,3,580.5980224609375,430.3543701171875,18.16064453125,41.09588623046875,1,-1,-1,-1
5,4,1489.71044921875,436.4090576171875,47.883056640625,110.5401611328125,1,-1,-1,-1
5,5,421.9797058105469,458.3156433105469,37.5230712890625,86.51046752929688,1,-1,-1,-1
5,6,585.4022216796875,446.8351135253906,83.160400390625,262.7636413574219,1,-1,-1,-1
5,7,1055.7890625,484.357421875,35.650146484375,109.0517578125,1,-1,-1,-1
5,8,581.3610229492188,455.68756103515625,36.4989013671875,134.3551025390625,1,-1,-1,-1
5,9,595.3568115234375,427.43341064453125,18.3668212890625,41.84039306640625,1,-1,-1,-1
5,10,443.4902038574219,445.99908447265625,117.24679565429688,278.78173828125,1,-1,-1,-1
5,11,935.023681640625,433.49432373046875,40.783447265625,116.05255126953125,1,-1,-1,-1
5,12,1257.559326171875,447.03607177734375,34.2119140625,101.24798583984375,1,-1,-1,-1
5,13,640.3212890625,456.1402587890625,62.5423583984375,189.1439208984375,1,-1,-1,-1
5,14,1031.7208251953125,448.3823547363281,25.345703125,73.32400512695312,1,-1,-1,-1
5,15,550.4404296875,461.8767395019531,31.630859375,96.92587280273438,1,-1,-1,-1
5,16,668.427978515625,450.8019714355469,32.7225341796875,88.23318481445312,1,-1,-1,-1
5,17,1089.723388671875,483.990478515625,33.723876953125,113.7286376953125,1,-1,-1,-1
5,18,476.56329345703125,458.7918701171875,84.67236328125,244.7901611328125,1,-1,-1,-1
5,20,1373.749267578125,436.1348571777344,45.650634765625,118.05331420898438,1,-1,-1,-1
5,19,1015.3717041015625,429.95196533203125,42.161865234375,122.5643310546875,1,-1,-1,-1
5,21,1351.9085693359375,416.42974853515625,176.306396484375,383.00244140625,1,-1,-1,-1
6,21,1360.928466796875,415.47796630859375,169.205078125,381.2886962890625,1,-1,-1,-1
6,1,976.5912475585938,462.0542297363281,26.951904296875,64.80380249023438,1,-1,-1,-1
6,0,1454.64501953125,427.1568603515625,157.281982421875,343.23388671875,1,-1,-1,-1
6,2,1100.9703369140625,439.71221923828125,36.73779296875,106.89154052734375,1,-1,-1,-1
6,3,580.452880859375,429.7829284667969,19.2135009765625,41.744384765625,1,-1,-1,-1
6,4,1489.0194091796875,435.5610046386719,46.9326171875,112.61459350585938,1,-1,-1,-1
6,5,421.027587890625,458.2867431640625,36.878173828125,86.0555419921875,1,-1,-1,-1
6,6,585.461669921875,446.1603088378906,85.3018798828125,264.0313415527344,1,-1,-1,-1
6,7,1055.3265380859375,484.66912841796875,35.77685546875,108.5340576171875,1,-1,-1,-1
6,8,581.4738159179688,455.5904541015625,36.458251953125,134.4736328125,1,-1,-1,-1
6,9,595.5855102539062,427.1765441894531,18.5777587890625,41.8272705078125,1,-1,-1,-1
6,10,441.9161682128906,444.87957763671875,119.86697387695312,280.349609375,1,-1,-1,-1
6,11,934.81591796875,433.2556457519531,40.5274658203125,116.50320434570312,1,-1,-1,-1
6,12,1257.5504150390625,447.1675109863281,34.22802734375,101.04306030273438,1,-1,-1,-1
6,13,641.503173828125,457.0223693847656,61.5201416015625,188.58297729492188,1,-1,-1,-1
6,14,1031.731689453125,448.6620178222656,25.218017578125,73.37393188476562,1,-1,-1,-1
6,15,550.5706787109375,461.95147705078125,31.5054931640625,96.5924072265625,1,-1,-1,-1
6,16,670.0452270507812,450.58819580078125,32.9598388671875,91.5247802734375,1,-1,-1,-1
6,17,1089.5078125,484.6643371582031,33.42626953125,112.66549682617188,1,-1,-1,-1
6,18,480.7920227050781,459.7054748535156,80.42941284179688,245.63900756835938,1,-1,-1,-1
6,20,1373.912841796875,435.5864562988281,50.4560546875,126.59359741210938,1,-1,-1,-1
6,19,1015.1561889648438,430.3455505371094,41.88299560546875,121.78524780273438,1,-1,-1,-1

可能需要排查一下您那边是否有一些不一致的情况？

有问题的话欢迎回复讨论～

ATun-cell commented 2 months ago

您好，我重新clone了code并且下载对应权重，因为两处版本兼容问题修改了部分代码 1.因为我的python版本是3.8，在 Python 3.10 之前，| 不能用于类型提示，必须使用 typing.Union

2.这里修改了decoder_layer传进去的掩码类型

_unknown_tgts_flatten, _ = self.decoder_layers[_layer](
                query=_unknown_embeds_flatten[None, ...], key=_trajectory_embeds_flatten[None, ...], 
                value=_trajectory_embeds_flatten[None, ...],
                key_padding_mask=key_padding_mask.bool(), attn_mask=_decoder_mask.bool()
            )

我用您提供的指令进行推理，得到的结果仍然是错误的

1,0,1412.5599365234375,428.6690673828125,185.95654296875,337.4359130859375,1,-1,-1,-1
1,1,971.6083374023438,458.1192321777344,28.09912109375,72.24325561523438,1,-1,-1,-1
1,2,1099.1802978515625,439.2061462402344,37.63427734375,107.20187377929688,1,-1,-1,-1
1,3,579.880615234375,431.8118896484375,19.0086669921875,41.779296875,1,-1,-1,-1
1,4,1485.3671875,437.5726318359375,45.014404296875,105.64691162109375,1,-1,-1,-1
1,5,420.2038879394531,457.01214599609375,38.89361572265625,89.06732177734375,1,-1,-1,-1
1,6,584.0567626953125,446.40899658203125,85.33740234375,263.5159912109375,1,-1,-1,-1
1,7,1054.742919921875,483.798583984375,36.077392578125,109.7874755859375,1,-1,-1,-1
1,8,581.8377075195312,457.2105712890625,36.5823974609375,127.9473876953125,1,-1,-1,-1
1,9,595.5175170898438,427.3494873046875,17.807861328125,42.42437744140625,1,-1,-1,-1
1,10,442.6086730957031,444.6192626953125,107.65426635742188,279.7412109375,1,-1,-1,-1
1,11,936.273681640625,433.197021484375,39.66845703125,116.061279296875,1,-1,-1,-1
1,12,1256.7784423828125,446.59942626953125,35.845458984375,101.49578857421875,1,-1,-1,-1
1,13,633.05322265625,457.93292236328125,67.7847900390625,186.029052734375,1,-1,-1,-1
1,14,1031.5521240234375,447.3624267578125,25.5390625,73.84130859375,1,-1,-1,-1
1,15,549.1531982421875,461.58709716796875,32.353271484375,96.79925537109375,1,-1,-1,-1
1,16,665.3749389648438,453.514892578125,32.86962890625,84.90899658203125,1,-1,-1,-1
1,17,1089.3714599609375,483.4588317871094,33.432373046875,114.36557006835938,1,-1,-1,-1
1,18,476.8038330078125,460.2270202636719,78.14794921875,240.73263549804688,1,-1,-1,-1
1,19,1015.5050048828125,430.1148376464844,42.492431640625,122.78579711914062,1,-1,-1,-1
1,20,1361.131591796875,434.8583984375,42.0546875,120.24517822265625,1,-1,-1,-1
1,21,1338.077880859375,416.2593994140625,164.052978515625,369.522216796875,1,-1,-1,-1
2,22,972.0626220703125,459.1817932128906,27.1138916015625,67.62771606445312,1,-1,-1,-1
2,23,1418.2099609375,428.4075927734375,181.822021484375,339.64501953125,1,-1,-1,-1
2,24,1100.1102294921875,439.2418212890625,36.99951171875,106.7119140625,1,-1,-1,-1
2,25,580.444580078125,431.5041809082031,18.9908447265625,41.39886474609375,1,-1,-1,-1
2,26,1492.156982421875,438.9045104980469,46.43017578125,113.32199096679688,1,-1,-1,-1
2,27,421.7413330078125,457.29022216796875,37.234619140625,87.91888427734375,1,-1,-1,-1
2,28,585.1475219726562,447.1407470703125,84.1683349609375,262.8446044921875,1,-1,-1,-1
2,29,1055.164794921875,484.11370849609375,36.018798828125,109.2449951171875,1,-1,-1,-1
2,30,582.0966796875,457.41168212890625,36.11572265625,132.530517578125,1,-1,-1,-1
2,31,595.818603515625,427.1798400878906,17.8392333984375,42.30755615234375,1,-1,-1,-1
2,32,443.3006286621094,445.91351318359375,109.50466918945312,277.4818115234375,1,-1,-1,-1
2,33,936.298095703125,433.256103515625,39.745849609375,116.2001953125,1,-1,-1,-1
2,34,1257.21923828125,446.37042236328125,35.069091796875,101.8271484375,1,-1,-1,-1
2,35,633.1262817382812,458.3914794921875,68.5843505859375,185.3292236328125,1,-1,-1,-1
2,36,1032.1456298828125,448.5838317871094,25.361328125,73.21334838867188,1,-1,-1,-1
2,37,550.1682739257812,462.67919921875,31.649658203125,95.44049072265625,1,-1,-1,-1
2,38,666.2780151367188,452.9715576171875,33.16796875,86.91888427734375,1,-1,-1,-1
2,39,1089.3634033203125,483.2491149902344,33.982177734375,114.66427612304688,1,-1,-1,-1
2,40,473.3510437011719,460.5021057128906,83.04763793945312,245.53536987304688,1,-1,-1,-1
2,41,1015.4281005859375,430.65936279296875,42.8623046875,122.56561279296875,1,-1,-1,-1
2,42,1364.803466796875,434.2623596191406,45.3720703125,122.55905151367188,1,-1,-1,-1
2,43,1341.633056640625,417.51824951171875,169.38720703125,373.128662109375,1,-1,-1,-1
3,44,973.2009887695312,459.4090881347656,25.407470703125,65.28286743164062,1,-1,-1,-1
3,45,1424.716552734375,429.3929443359375,185.71533203125,340.5732421875,1,-1,-1,-1
3,46,1099.971923828125,439.45330810546875,37.348876953125,107.38507080078125,1,-1,-1,-1

我注意到我的结果和您的结果中坐标位置是相同的，只是id不同。我截取了调试过程中self.id_decoder部分的一些重要参数的value self.id_decoder输入 format结果 self.id_decoder输出，每一目标都是id字典201处对应位置的值明显最大 id_word输出 self.id_decoder部分层的权重 权重导入

供您参考分析

HELLORPG commented 2 months ago

这里的 attn_mask=_decoder_mask 是不能转换成 bool 类型的，因为此时的 _decoder_mask 中包括了相对位置编码，因此如果转换成为 bool 类型的话，就会丢掉相对位置编码中的信息，更严重的是会将所有的 mask 转换为 True，这样的话就无法看到历史轨迹了，因此推理的结果都是 newborn objects。

ATun-cell commented 2 months ago

耶！！！ 终于解决了 感谢大佬感谢

HELLORPG commented 2 months ago

不用谢。