search

tusen-ai / SST

Code for a series of work in LiDAR perception, including SST (CVPR 22), FSD (NeurIPS 22), FSD++ (TPAMI 23), FSDv2, and CTRL (ICCV 23, oral).
Apache License 2.0
765 stars 96 forks source link

Incremental/additional training of FSDv2 #188

Open ArseniuML opened 3 months ago

ArseniuML commented 3 months ago

Is there a way to apply additional training on our data to pre-trained checkpoint (say on Argoverse2)? If there is a simple CNN, I can fix conv layers weights and re-train fully-connected layers (on even the latest conv layers). Do I can apply similar technique to FSDv2?

Abyssaledge commented 3 months ago

It depends on the domain gap between your data and the pre-training data. I personally believe tuning all parameters is better.

ArseniuML commented 3 months ago

How do I can estimate this domain gap? We have 80-beam Robosense RS Ruby Lite LIDAR, Argo2 is created with two VLP-32C LIDARs.

Argo2 paper says that two VLP-32C effectively create 64 beams, but it's not fully clear for me, how are clouds from these tho LIDARs being merged and what the resulting beam distribution is...

Abyssaledge commented 3 months ago

It is hard to estimate the domain gap without experiments. My experience is only using xyz as input without intensity is good for narrowing the domain gap. Different sensors have quite different intensity values.

ArseniuML commented 3 months ago

Unfortunately, I am very limited in my ability to conduct experiments.

  1. We have 2xRTX3090 training server, and 24 epochs of FSDv2 training on Argo2 took about 5 days.
  2. We have very small amount of LIDAR data (2950 frames vs 100k in Argoverse2). Collecting of LIDAR data is not a fast process in my organization...

I guess the best possibility to train a detector in these conditions is to freeze some pre-trained layers and apply additional training. Do you can assume, which layers I have to freeze?