search

taconite / arah-release

[ECCV 2022] ARAH: Animatable Volume Rendering of Articulated Human SDFs
https://neuralbodies.github.io/arah/
MIT License
182 stars 15 forks source link

Custom data preprocessing #6

Open ArmastusChen opened 1 year ago

ArmastusChen commented 1 year ago

Thanks for the great work. Is it possible to release pre-processing code (guideline) for my custom video so I could run it using your methods? Thanks again!

taconite commented 1 year ago

Hi, sorry for the late reply, I was traveling for the last few weeks. And thanks for your interest in our work!

For multi-view capture, I think you need to follow neural body and easymocap guidelines. After all, the datasets I used are created by their team. In principle, our preprocessing code should work with any easymocap output, still, you might need to do some tweaks yourself for custom datasets.

If your mean custom smartphone capture, then it's a bit complicated. The overall process is 1) extract person bounding boxes in each frame (I use openpose to get keypoints and compute bounding boxes), 2) get SMPL estimations using SPIN (with bounding boxes computed from stage 1), 3) get foreground masks with MODNet, 4) preprocess the raw SMPL estimations and raw masks so that ARAH dataloaders can load them, 5) train the model.

I could update a script for (4) with some guidelines, probably after the CVPR deadline. However, for other steps, there are too many dependencies which makes it really hard to have a unified codebase doing all of them. I would like to figure out a concise pipeline doing all the steps, but given the complexity of dependencies and other tasks at hand, I cannot promise any timeline for it...