search

hustvl / MapTR

[ICLR'23 Spotlight] MapTR: Structured Modeling and Learning for Online Vectorized HD Map Construction
MIT License
984 stars 152 forks source link

original dataset overlap problem #109

Open wheelandwheat opened 9 months ago

wheelandwheat commented 9 months ago

Hello, author,

In the maptr_V2 paper, the central line exhibited excellent results. Did you split the dataset because the training and testing sets of the original dataset overlap geographically（both nuscense and argoverse）? We reproduced your paper lanegap and tested splitting dataset by location. The results on the original, non-split dataset were indeed impressive. However, after splitting based on geographic location, the performance declined. This was evident in both accuracy and recall. It seems that the model may have overfit the non-split dataset.

LegendBC commented 9 months ago

Thanks for your kind discussion! In MapTRv1,v2, and LaneGAP, we just follow the standard data split setting and have not considered this issue. But your suggestion is right. We will try out the non-overlap split setting and further benchmark.