search

sherwinbahmani / tc4d

TC4D: Trajectory-Conditioned Text-to-4D Generation
https://sherwinbahmani.github.io/tc4d
Apache License 2.0
173 stars 3 forks source link

About 3D representation. #9

Closed Colezwhy closed 1 week ago

Colezwhy commented 1 week ago

Hi, thank you for your great work. And I am wondering if this trajectory-conditioned pipeline can be performed on 3DGS-based 3D assets? Would you please give some suggestions on that?

sherwinbahmani commented 1 week ago

Hi,

Adjusting the method to 3DGS should be pretty simple. Since we apply rotations and translations for the global transformations to the sampled point of NeRFs, you could simply apply this to all Gaussians directly. It should be actually easier to do this task with Gaussians, but back then the quality of generative 3D with Gaussians wasn't good.

Here is the part where the scale and shift along the trajectory is done: https://github.com/sherwinbahmani/tc4d/blob/d1af1822a5fc4de8a78eb93f5bd774f74c12927e/threestudio/utils/bounding_boxes.py#L3

You could apply rotation and translation the same way to all Gaussians.

Colezwhy commented 1 week ago

Hi,

Adjusting the method to 3DGS should be pretty simple. Since we apply rotations and translations for the global transformations to the sampled point of NeRFs, you could simply apply this to all Gaussians directly. It should be actually easier to do this task with Gaussians, but back then the quality of generative 3D with Gaussians wasn't good.

Here is the part where the scale and shift along the trajectory is done:

https://github.com/sherwinbahmani/tc4d/blob/d1af1822a5fc4de8a78eb93f5bd774f74c12927e/threestudio/utils/bounding_boxes.py#L3

You could apply rotation and translation the same way to all Gaussians.

Thank you for your reply! So you have tried 3DGS as the representation, right? Actually, I wonder how bad the producing results are, if generated with 3DGS representation. Thanks!

sherwinbahmani commented 1 week ago

We tried in late 2023 dynamic 3D Gaussians without trajectory, just for text-to-4D and the results were not good. The results were saturated. But back then there were no good 3DGS text-to-3D pipelines. I think now with better pipelines it should work better. I don't have the visual results for it, but the geometry of the assets didn't look good.

I think doing this with Gaussians now is a good idea.

Colezwhy commented 1 week ago

We tried in late 2023 dynamic 3D Gaussians without trajectory, just for text-to-4D and the results were not good. The results were saturated. But back then there were no good 3DGS text-to-3D pipelines. I think now with better pipelines it should work better. I don't have the visual results for it, but the geometry of the assets didn't look good.

I think doing this with Gaussians now is a good idea.

Thank you for your prompt reply!