georghess / neurad-studio

[CVPR2024] NeuRAD: Neural Rendering for Autonomous Driving
https://research.zenseact.com/publications/neurad/
Apache License 2.0
329 stars 23 forks source link

Render the original video with all actors moving and not just a shift #14

Closed amoghskanda closed 4 months ago

amoghskanda commented 4 months ago

Hey! Thanks for making this open source. I was trying to render the original sequence with all the actors moving. In the current render, only the camera moves and actor shifts are possible but I wanted the actors to move as per their original trajectory used while training(not just shift but the actors move every frame). I'm guessing it's possible as every actor has positions x,y and z in the cuboids file so was wondering if you could give any insights on this. I was trying to edit the trajectory of the dynamic actors and for that, they need to be moving in the render, not just the camera moving as per the keyframe. Thanks

atonderski commented 4 months ago

You need to make sure that to move time together with the camera position in your desired trajectory. This is manipulated using cameras.times

amoghskanda commented 4 months ago

You need to make sure that to move time together with the camera position in your desired trajectory. This is manipulated using cameras.times

I was actually looking to modify actors individually. The actor shift in render.py helps in editing all actors together.

atonderski commented 4 months ago

this is absolutely possible, but you need to modify the rendering script according to achieve your desired results. (instead of modifying the full actor position/rotation tensor, you need to modify individual elements). Perhaps you can also look in nerfstudio/scripts/closed_loop/server.py for inspiration.