How to edit dynamic objects?

[xiao1874]

Hi, I have run this project and achieved good results and efficiency, but I don't know how to edit foreground objects, such as moving vehicles or replacing pedestrians. Can you provide some suggestions or example operations? thx!

[ziyc]

Hi @xiao1874, thank you for your interest.

The simulation part is interesting but also a bit involved. The scripts vary according to your simulation goals. My suggestion is if you want to do simulation, the best practice is building the script based on tools/eval.py.

If you want to move vehicles or pedestrians, try to modify the values of instances_quats and instances_trans. These store the box information of each dynamic actor. You can find this at https://github.com/ziyc/drivestudio/blob/a1fe5d29162e5c4da81bb281c8223b457964a693/models/nodes/rigid.py#L87-L88

If you want to replace or swap instances, it depends on the target category that you want to edit. For vehicles, if you want to replace A and B, I think the simplest way is to exchange object ids of gaussians. For example:

GS_B_mask = self.point_ids==B_id
self.point_ids[self.point_ids==A_id] = B_id
self.point_ids[GS_B_mask] = A_id

For other gs representations (deformable-gs, smpl) when editing, you need to pay attention to associate the network embedding (for deformable gs) and SMPL parameters (for SMPL-Gaussians) to the same individual while editing.

Being familiar with the scripts under 'https://github.com/ziyc/drivestudio/tree/main/models/nodes' that define the gaussian nodes would be helpful. Here are some helpful references that provide hints for manipulating objects:

https://github.com/ziyc/drivestudio/blob/a1fe5d29162e5c4da81bb281c8223b457964a693/models/nodes/deformable.py#L158-L190 https://github.com/ziyc/drivestudio/blob/a1fe5d29162e5c4da81bb281c8223b457964a693/models/nodes/rigid.py#L504-L535

We're considering developing general tools for easier and more efficient editing, but this will take some time.

[xiao1874]

Thank you so much for your quick reply and effective suggestion! I think I need to take some time to understand. Best!

[Debrove]

Awesome work!!!

If I want to add more vehicles, pedestrians or cyclists ( and so on...), do I just need to increase the value of num_instances?

If you want to move vehicles or pedestrians, try to modify the values of instances_quats and instances_trans. These store the box information of each dynamic actor. You can find this at

https://github.com/ziyc/drivestudio/blob/a1fe5d29162e5c4da81bb281c8223b457964a693/models/nodes/rigid.py#L87-L88

BTW, if possible, could you please release the ckpt weights (e.g., KITTI)? That would be more helpful for fast evaluation.

Thanks! Best regards!

[ziyc]

Hi @Debrove, thank you for your interest!

For adding objects, modifying just instances_quats and instances_trans isn't sufficient as these parameters only save the box information. You'll need to add Gaussian properties (opacities, means...) and per-gaussian object IDs as well like listed here (using rigid nodes as an example): https://github.com/ziyc/drivestudio/blob/388642a6a998833cb388e0f1a65dfc2071cc6a61/models/nodes/rigid.py#L72-L99

Regarding checkpoints, we're working on releasing those we've trained, but it takes time due to the number of scenes. We've made our results reproducible and the training process efficient. KITTI scenes typically take less than 1 hour to train on a single 4090 GPU. Feel free to start training if we can't release the checkpoints soon.

[zzz5y]

Hi @Debrove, thank you for your interest!

For adding objects, modifying just instances_quats and instances_trans isn't sufficient as these parameters only save the box information. You'll need to add Gaussian properties (opacities, means...) and per-gaussian object IDs as well like listed here (using rigid nodes as an example):

https://github.com/ziyc/drivestudio/blob/388642a6a998833cb388e0f1a65dfc2071cc6a61/models/nodes/rigid.py#L72-L99

Regarding checkpoints, we're working on releasing those we've trained, but it takes time due to the number of scenes. We've made our results reproducible and the training process efficient. KITTI scenes typically take less than 1 hour to train on a single 4090 GPU. Feel free to start training if we can't release the checkpoints soon.

Hello, thank you very much for your excellent work. I now have a question regarding adding and removing objects. If I previously trained a model with weights based on omnire/streetgs, do I need to retrain if I want to add/remove an object, or do I just need to modify parameters like instances_quats, instances_trans, point_ids, _means, distance, etc.?