search

facebookresearch / goliath

Goliath Dataset and Official PyTorch Implementation of RelightableHands, Relightable Gaussian Codec Avatars, and Driving-Signal Aware Full-Body Avatars.
Other
205 stars 15 forks source link

Problem about Custom Dataset #9

Open xi8zhu opened 3 months ago

xi8zhu commented 3 months ago

Hi! Thank you for your fantastic work! And I have a question about how to apply the model to the custom dataset. How can I get the UV images and the static assets?

una-dinosauria commented 3 months ago

Hi!,

All the assets are part of the release. You should email julietamartinez@meta.com to get access to the dataset.

Cheers,

xi8zhu commented 3 months ago

Thank you for answering my quesion! We got the dataset goliath uses and want to train the model on our own datasets. However, our dataset only consists of images captured on different cameras under diffenert light patterns. We are confused about how to get the uv images and static assets in goliath datasets! Could you give any suggestions? Thank you again!

una-dinosauria commented 3 months ago

Ah, for that you need some sort of registration of the canonical topology we provide wrt to your data -- then get average textures per frame by unwrapping the registration.

We don't have plans to release code to do those things in custom datasets at the moment.

jryebread commented 3 months ago

@una-dinosauria any future possibility of that code being released? strange to release code with no way to reproduce the results. Atleast some guide on how to get what is needed for training on a custom dataset.

una-dinosauria commented 3 months ago

@jryebread

@una-dinosauria any future possibility of that code being released?

Maybe in the future. Consider opening another issue with that request, so I can show this to the people in charge of those decisions.

strange to release code with no way to reproduce the results

Our results are reproducible with the data that we provide. We are not responsible for our code running on other people's data -- good data for avatar construction is hard and expensive to collect! This is not strange, but rather standard in academic settings. Thank your for your understanding.

Atleast some guide on how to get what is needed for training on a custom dataset.

Like I mentioned, you would need registration with respect to the canonical mesh that we provide. After that, texture unwrapping averaging all the cameras.

Non-rigid registration is a classical problem in computer vision that is studied separately from avatar reconstruction. Maybe see this recent paper from CVPR24 and references there to learn more about the state of the art in that area.

Cheers,

xi8zhu commented 3 months ago

Thank you once again for your impressive work and sincere responses.