search

valeoai / RADIal

162 stars 52 forks source link

A solution to the invalid pre-training model #67

Open cuckoorain opened 9 months ago

cuckoorain commented 9 months ago

Brief description of bug

pre-training model downloaded in https://drive.google.com/drive/folders/1qh_ixfiDRUAiXg0d0SIBv0wj2L_DxTIS seems invalid in testing and evluation. The reason of this issue is the nomenclature.

Solutions

  1. I highly recommend to replace the pre-training model with following .pth file: https://drive.google.com/file/d/1ZmEL4Ees4vQGoGc0XpKkNhyIss-CtyKB/view?usp=sharing
  2. Or you can execute following code: `import torch from collections import OrderedDict state_dict = torch.load('FFTRadNet_RA_192_56_epoch78_loss_172.8239_AP_0.9813.pth', map_location=torch.device('cpu'))

temp_net_dir = state_dict['net_state_dict']

new_state_dict = OrderedDict() new_net_dict = OrderedDict()

for key, value in temp_net_dir.items(): if key.startswith('backbone.preproc'): new_key = key.replace('backbone.preproc', 'FPN.pre_enc') new_net_dict[new_key] = value elif key.startswith('backbone'): new_key = key.replace('backbone', 'FPN') new_net_dict[new_key] = value elif key.startswith('RAmap_header'): new_key = key.replace('RAmap_header', 'RA_decoder') new_net_dict[new_key] = value else: new_net_dict[key] = value

for key, value in state_dict.items(): new_state_dict[key] = value

new_state_dict['net_state_dict'] = new_net_dict

torch.save(new_state_dict, 'modifiedFFTRadNet_RA_192_56_epoch78_loss_172.8239_AP_0.9813.pth')`

Self-Diagnosis

Environment(for bug reports)

haomingHu-dd commented 9 months ago

Hello, have you trained using the source code, in my training results in hard mode mAR can reach amazingly more than 80%, but only 64% in the paper, what does overall stand for in the paper? How was the training done.

cuckoorain commented 9 months ago

Hello, have you trained using the source code, in my training results in hard mode mAR can reach amazingly more than 80%, but only 64% in the paper, what does overall stand for in the paper? How was the training done.

Hi, could you tell me your validation loss? I think the problem maybe caused by overfitting caused by data leakage.

haomingHu-dd commented 9 months ago

I cloned the code straight down and ran it without any modifications and the training loss image is shown below. 1704351006453 The training loss is basically around 1e5 and appears to decrease and then increase (I think it's overfitting)

Then, I validated it using the pre-trained model you uploaded (modifiedFFTRadNet_RA_192_56_epoch78_loss_172.8239_AP_0.9813.pth), and the results are consistent with those in the paper.

In the pre-training model, I noticed a loss of 172.8239, which I think should be a problem with the training step.

`(radial) dell@dell-Precision-7920-Tower:~/radial/RADIal$ python FFTRadNet/3-Evaluation.py True =========== Dataset ==================: Mode: sequence Training: 6231 Validation: 986 Test: 1035

=========== Loading the model ==================: =========== Running the evaluation ==================: Generating Predictions... 1034/1035 [==================>.] - ETA: 0s/home/dell/anaconda3/envs/radial/lib/python3.8/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection return lib.intersection(a, b, **kwargs) ------- Detection Scores ------------ mAP: 0.9684112584064783 mAR: 0.8217813051146385 F1 score: 0.8890912453079216 ------- Regression Errors------------ Range Error: 0.1176896432802109 m Angle Error: 0.104129341655343 degree ------- Freespace Scores ------------ mIoU 73.97902784541033 %`

haomingHu-dd commented 9 months ago

I have replied to you in Issue, I hope to check it out when you are free, and look forward to your reply！

胡浩明

@. | ---- Replied Message ---- | From | @.> | | Date | 1/4/2024 12:40 | | To | @.> | | Cc | @.> , @.***> | | Subject | Re: [valeoai/RADIal] A solution to the invalid pre-training model (Issue #67) |

Hello, have you trained using the source code, in my training results in hard mode mAR can reach amazingly more than 80%, but only 64% in the paper, what does overall stand for in the paper? How was the training done.

Hi, could you tell me your validation loss? I think the problem maybe caused by overfitting caused by data leakage.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

lcy0308 commented 5 months ago

I have replied to you in Issue, I hope to check it out when you are free, and look forward to your reply！ | | 胡浩明 | | @. | ---- Replied Message ---- | From | @.> | | Date | 1/4/2024 12:40 | | To | @.> | | Cc | @.> , @.> | | Subject | Re: [valeoai/RADIal] A solution to the invalid pre-training model (Issue #67) | Hello, have you trained using the source code, in my training results in hard mode mAR can reach amazingly more than 80%, but only 64% in the paper, what does overall stand for in the paper? How was the training done. Hi, could you tell me your validation loss? I think the problem maybe caused by overfitting caused by data leakage. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.>

Hello, since the data is not available yet, could you please provide ready to use dataset and raw dataset? This will be very helpful to me, thank you.

ArthurOuaknine commented 18 hours ago

Hello. We are sorry about the late reply, but also happy to announce that the dataset is back online! You can access it at this url: https://l.linklyhq.com/l/205Ok The link is also available in the README of the repository. Sorry again for the inconvenience.