liuyuan-pal / SyncDreamer

[ICLR 2024 Spotlight] SyncDreamer: Generating Multiview-consistent Images from a Single-view Image
https://liuyuan-pal.github.io/SyncDreamer/
MIT License
906 stars 39 forks source link

Would it possible to support multiple views inputs? #26

Open hp-cv opened 1 year ago

hp-cv commented 1 year ago

Hi, the single shot image input provides reasonably good result. I wonder if there's a way to support multiple views(at least 2 or 3+ views,could be under the same angles) that can increase the quality?Thanks.

clinplayer commented 1 year ago

Yes! If you know the camera poses, a potential way is to unproject the multiview image features to the 3D feature volume, and then you can condition the denoising process on the correlated multiview features. To achieve this, you may need to finetune the network using a multiview setting.