search

prophesee-ai / prophesee-automotive-dataset-toolbox

A set of Python scripts to evaluate the Automotive Datasets provided by Prophesee
Apache License 2.0
152 stars 30 forks source link

training result on Gen1 #32

Closed shenqiwang27 closed 10 months ago

shenqiwang27 commented 1 year ago

Hi,

I used your example code train_detection.py to reproduce RED model on Gen1 dataset, using this command: python train_detection.py ./log/ ./data/Gen1_event_cube/ --classes pedestrian car

The model is Vanilla model in your metavision lib. After training 20 epochs, I used this command to evaluate: python train_detection.py ./log/ ./data/Gen1_event_cube/ --just_test --checkpoint ./log/checkpoints/epoch=14-step=483206.ckpt

Got 0.53 mAP, much higher than your paper reported. The dataset is downloaded directly from your website. . But I also tried to reproduce your result on Gen4 dataset and got a similar result as you reported. So what could be the problem on Gen1？Or did you retrain your model on Gen1?

lbristiel-psee commented 1 year ago

Hi,

Got 0.53 mAP, much higher than your paper reported.

could you share a link to the paper you are refering to as well as the value you expect?

shenqiwang27 commented 1 year ago

The paper is Learning to Detect Objects with a 1 Megapixel Event Camera

I expected to get a similar performance, 0.40 mAP on Gen1 dataset, as you reported

lbristiel-psee commented 1 year ago

Well, the paper is using the 1M pixel dataset (Gen4 dataset) but the "GEN1 Dataset" has a resolution of only 304×240 pixels so you won't get the same result with both datasets.

shenqiwang27 commented 1 year ago

Yes, but my point is that I reproduce RED model both on Gen1 and Gen4 dataset. For Gen4 I got 0.43 mAP which is consistent with what you have reported. However, for Gen1, I got 0.53 mAP, that is much higher than 0.40 mAP performance that you reported for gen1 dataset. Did you not retrain the model on gen1 and instead directly use the model trained on gen4 for evaluation on gen1？

lbristiel-psee commented 1 year ago

Thanks for your answer and sorry for my confusion, I did not check the table in the paper properly and did not notice it was advertising results for both 1M and Gen1 datasets.

So I circle around internally to try to understand what could be the source of the difference you observe. It's a bit complicated because part of the people who worked on this topic left the company in the meantime. But from what I could gather, one possible reason for this difference could be that Gen1 is not labelled with a fixed frequency (there are some gaps and missing labels in the GT). But in Metavision SDK, the scripts train_detection.py and eval_coco_kpi.py scripts expect only a fixed labeling frequency for the GT. Essentially, Gen1 dataset is small and presented some oddities that we didn't want to handle in Metavision SDK scripts because that would have added too much complexity for no real benefit. So when using the evaluation on Gen1, some detections of valid objects might be considered as false positives (due to missing labels for those frames). So that could explain the difference result.

Hope this helps, Laurent for Prophesee Support

JDYG commented 2 months ago

Hi, where do you find the RED code? It seems that this repository doesn't have any code about object detection. If possible, could you please share the link for that? Thanks.

lbristiel-psee commented 1 month ago

@JDYG the code "train_detection.py" code mentioned in the first message of that thread is part of the Metavision SDK and is documented here: https://docs.prophesee.ai/stable/samples/modules/ml/train_detection.html To see how to download/install the MV SDK, check https://docs.prophesee.ai/stable/installation/index.html