Thank you for your prompt response. I have used the suggested code and made modifications based on the following suggestions. I successfully ran the evaluation code, but the MOTA value is still negative.

[LiawWun]

          Thank you for your prompt response. I have used the suggested code and made modifications based on the following suggestions. I successfully ran the evaluation code, but the MOTA value is still negative.

1. I removed objects with fewer than 5 radar points (both in the ground truth and model predictions).
2. I set the confidence threshold to 0.078483.
3. I merged the labels for bicycles and the person riding them together (following the suggested method).

Additionally, I made a video my video to show the ground truth and model detection results. In the video, the ground truth is displayed as bounding boxes (for visualization purposes only; in the evaluation code, the calculations are still based on radar points), and the model predictions are represented using different colors.

I compared this video with the one you provided RaTrack video and I noticed that in my video, there are a lot of false positives compares to yours, for example, between the 0:14 and 0:31 minute marks,, whereas in your video, between the 0:24 and 0:30 minute marks, there aren’t as many false positives.

Would it be possible to share the detection results used in your video? This could help me pinpoint where the discrepancies are arising.

Thank you for your continued assistance.

my email: jiawunliaw@gmail.com

Originally posted by @LiawWun in https://github.com/LJacksonPan/RaTrack/issues/11#issuecomment-2312876667

[MrTooOldDriver]

Hi, thank you for your continued effort in validating our work. Your efforts are greatly appreciated.

I have sent you the results we used for visualisation video. I will also push a copy to this repository later.

From the video you provided, I noticed that the model is not working as intended because many of the non-moving (background) objects have been segmented as moving objects, leading to poor clustering and tracking results. This might be a direction to look into during debugging.

May I ask if the visualisation results were from the checkpoint we provided, or were they trained by yourself?

[LiawWun]

Thanks for the prompt reply.

1. The model weight I used is model.last.t7 from the src/checkpoint/track4d_radar/models folder. I used this weight to evaluate the VOD validation dataset. The visualization results are based on this weight, with some clustered objects being filtered out (objects with fewer than 5 radar points and a confidence score less than 0.078483).

2. I will use the results provided by you for the validation. Thanks!

[MrTooOldDriver]

Thanks for the prompt reply.

1. The model weight I used is model.last.t7 from the src/checkpoint/track4d_radar/models folder. I used this weight to evaluate the VOD validation dataset. The visualization results are based on this weight, with some clustered objects being filtered out (objects with fewer than 5 radar points and a confidence score less than 0.078483).
2. I will use the results provided by you for the validation. Thanks!

Hi, I have observed the same noise segmentation issues you mentioned with the provided checkpoint. I believe some bugs might have been introduced when I was tidying up the code and tuning it. Many apologies for that, some fixes are on the way.

[MrTooOldDriver]

Hi, after some investigation the noise in segmentation might be due to outdated hyperparameters being used for training the provided ckpt. However this shouldn't lead to a negative MOTA as most of them will be counted as invalid objects during evaluation.

May I ask if you are able to share the TP, FP, TN, FN, IDS, and number of GT when evaluating with different threshold and recall?

[LiawWun]

Using the weights from the GitHub repository, I tested on the VOD dataset's Delft_1 sequence. The results were as follows: 413 ground truths (GT), 274 true positives (TP), 1336 false positives (FP), 139 false negatives (FN), and 4 identity switches (IDS).

I have a question about "invalid objects." Does this refer to occlusion and truncation, as mentioned in the KITTI format?

Since RaTrack is a class-agnostic detection network, I believe every object should be included in the evaluation.

[LiawWun]

Is an "invalid object" defined as one that contains fewer than 5 radar points? If so, I have already accounted for them in the evaluation code.

[MrTooOldDriver]

Is an "invalid object" defined as one that contains fewer than 5 radar points?

Yes. As we define in the paper (Section V-E), any objects with less 5 points are invalid objects. (For both predictions and ground truth)

[LiawWun]

I have already accounted for them in the evaluation code, as shown in the video, where all valid objects are included, and all invalid objects are filtered out.

[LiawWun]

Could you kindly let me know how many GT (Ground Truth) instances you have counted in the delft_1 sequence of the VOD dataset?

[MrTooOldDriver]

Could you kindly let me know how many GT (Ground Truth) instances you have counted in the delft_1 sequence of the VOD dataset?

For delft_1 seq our script count 433 gt. May I ask if you are still using the same evaluation code as #11? If yes I could have a side to side comparison on it.

[LiawWun]

I have used both methods(#11 and the code you suggested), and the results calculated from them are quite similar.

[LiawWun]

Could you kindly let me know how many GT (Ground Truth) instances you have counted in the delft_1 sequence of the VOD dataset?

For delft_1 seq our script count 433 gt. May I ask if you are still using the same evaluation code as #11? If yes I could have a side to side comparison on it.

I sincerely appreciate it

[LiawWun]

You may set the parameter vis_per_frame to True to view additional details, such as the mapping relationship and object information.