Open blackmrb opened 3 months ago
补充一下7组训练失败的case,每个20s的场景分成前后两段10s,重新训练的结果。 怀疑是速度太大,自车移动距离过长,导致场景太大。
# - segment-1758724094753801109_1251_037_1271_037_with_camera_labels # 64km/h
# - segment-3490810581309970603_11125_000_11145_000_with_camera_labels # 71km/h
# - segment-3591015878717398163_1381_280_1401_280_with_camera_labels # 68km/h
# - segment-4468278022208380281_455_820_475_820_with_camera_labels # 70km/h, good case
# - segment-4537254579383578009_3820_000_3840_000_with_camera_labels # 68km/h, good case
# - segment-10072231702153043603_5725_000_5745_000_with_camera_labels # 40 -> 70km/h,宽阔,只有前方一辆车
# - segment-11454085070345530663_1905_000_1925_000_with_camera_labels #70km/h,good case
结论:
远处山突然丢了,紧接着一团云闪现出来向自车移动
https://github.com/PJLab-ADG/neuralsim/assets/165770555/2731ea0e-88fd-404f-bd47-61a9224bb6c5
地面没了,训练失败
https://github.com/PJLab-ADG/neuralsim/assets/165770555/7a77adca-7690-403f-acec-8f8d1cfd15c8
https://github.com/PJLab-ADG/neuralsim/assets/165770555/cfc43d25-65be-4760-840a-8a2f5d1b3a2e
立交桥突然消失
https://github.com/PJLab-ADG/neuralsim/assets/165770555/ec0f6022-b32f-4dbe-b70f-f05d6ae992c8
所有场景都是空中多一块,向自车冲来。 这个多一块尤为严重。
https://github.com/PJLab-ADG/neuralsim/assets/165770555/b7975d1c-6288-439b-a47d-9b77c478601e
立交桥下,非常糊。
https://github.com/PJLab-ADG/neuralsim/assets/165770555/93ab9728-c7fc-4492-809f-76e8ce38f057
首先感谢作者无私奉献开源了这个repo,我尝试了几个方法,目前Neuralsim是效果最好的。
我基于waymo做了18组实验(这些场景是在repo提供的81个动态场景里挑选的,有低速行驶和高速行驶的),有7组的loss不收敛。 使用的config: all_occ.with_normals.240201.yaml。 使用的segmentation模型:https://github.com/open-mmlab/mmsegmentation/tree/main/configs/mask2former
想请教的问题:
具体复现结果如下:
loss不收敛导致训到一半挂了(7组),自车速度在70km/h左右
训练完成的(11组)
效果好的(4组)
segment-9653249092275997647_980_000_1000_000_with_camera_labels, 0, 190 # 路口,很多行人
,这组效果最好,是code_multi/configs/exps/fg_neus=permuto/all_occ.with_normals.240201.yaml默认使用的scene,自车低速行驶。https://github.com/PJLab-ADG/neuralsim/assets/165770555/bbb374ed-8616-4135-be02-2fe7046373e1
119178,低速行驶,17km/h
https://github.com/PJLab-ADG/neuralsim/assets/165770555/eb5e982e-79af-48d0-a7d3-818711d8360b
365758,低速行驶,20km/h
https://github.com/PJLab-ADG/neuralsim/assets/165770555/db972f75-7284-40bd-aba7-4548ac32f769
189139,低速行驶,25km
https://github.com/PJLab-ADG/neuralsim/assets/165770555/9cf6572c-465a-4225-84c4-46eeff467286
车道线不清晰(2组)
segment-15053781258223091665_3192_117_3212_117_with_camera_labels # 20->75km/h
,问题:车道线不清晰https://github.com/PJLab-ADG/neuralsim/assets/165770555/cfbf155e-fbaf-4220-8e2a-5248b649ff35
seg188749
,整体可以,车道线不清晰, 36km/hhttps://github.com/PJLab-ADG/neuralsim/assets/165770555/7c1e9089-5d2f-40ce-afd0-1f7916edefde
空中多了一块东西(3组)
segment-14369250836076988112_7249_040_7269_040_with_camera_labels # 56km/h
,问题:空中多了一块 https://github.com/PJLab-ADG/neuralsim/assets/165770555/f4fb8402-2f6f-44e9-b3c5-a96bbb0ed5c0177369,空中多了一片东西,17km/h
https://github.com/PJLab-ADG/neuralsim/assets/165770555/c91dd1e7-ae29-49ec-9ec9-5fb140a99c14
391943,低速行驶,夜晚,效果可以,空中多了一片雨滴,15km/h
彻底糊了(1组)
364414,低速行驶,street彻底糊了,效果很差,0->50km/h
https://github.com/PJLab-ADG/neuralsim/assets/165770555/167e9e72-9352-4ae9-b21c-19eb4c6ce460
最开始几帧远处闪了几下(1组)
416406 ,0-25km https://github.com/PJLab-ADG/neuralsim/assets/165770555/ed86c126-da63-4b0c-b70b-a6cc0c9222d4
下面是7组失败的场景和一组成功的场景(seg965324,绿色)的loss对比
pixel loss
lidar loss
下面是各组实验的具体log记录
loss NAN
segment-1758724094753801109_1251_037_1271_037_with_camera_labels
segment-10072231702153043603_5725_000_5745_000_with_camera_labels
train-20240505234622385.log
segment-11454085070345530663_1905_000_1925_000_with_camera_labels
train-20240505234659815.log
segment-4537254579383578009_3820_000_3840_000_with_camera_labels
train-20240505234419432.log
train-20240505233823947.log
AssertionError: Occupancy grid becomes empty during training.
scenario_id=segment-3490810581309970603_11125_000_11145_000_with_camera_labels