Detailed human body recovery

[ouusan]

With Clothes 1.Learning to reconstruct people in clothing from a single rgb camera(2019) code:https://github.com/thmoa (no training code) (same link to 1,2,3) 2.Multi-garmentnet: Learning to dress 3d people from images(2019) code:No training code 3.Tex2shape: Detailed full human body geometry from a single image(2019) code: No

4. Learning body and cloth shape from a single image(2020) code:No 5.H4D: Human 4D Modeling by Learning Neural Compositional Representation(2022) code:https://github.com/BoyanJIANG/H4D With Hands 1.Reconstructing signing avatars from video using linguistic priors(2023) code:https://github.com/MPForte/SGNify
5. Interacting two-hand 3d pose and shape reconstruction from single color image(2021) code:https://github.com/iscas3dv/Two-Hand-Shape-Pose_v2 3.Joint hand object 3d reconstruction from a single image with cross-branch feature fusion(2021) code:No 4.Embodied hands: Modeling and capturing hands and bodies together() code: https://github.com/otaheri/MANO (no training code)

• Whole Body

1.Resolving 3d human pose ambiguities with 3d scene constraints(2019)

code:https://github.com/mohamedhassanmus/prox (optimization-based method) 2.Monocular expressive body regression through body-driven attention(2020)

code: https://github.com/vchoutas/expose (no training code) 3.A monocular 3d whole-body pose estimation system via regression and integration(2021)

code: https://github.com/facebookresearch/frankmocap.

4. Collaborative regression of expressive bodies using moderation(2021)

code: https://github.com/yfeng95/PIXIE(No training code)

5. Accurate 3d hand pose estimation for whole-body 3d human mesh estimation(2022) Why use Four hand MCP joint features:The hand MCP joint features are not only beneficial for 3D wrist rotations, but also for 3D elbow rotations, as 3D elbow rotations in the roll-axis are highly related to the hand MCP joint. Pose2Pose of Hands/body :Extract 3D joint features by interpolating joint coordinates on image features, then concatenate coordinates and features(3D joint coordinates+ Joint features) code:https://github.com/mks0601/Hand4Whole_RELEASE
6. PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Images(2023) grid-feature concat mesh align feature, and Spatial Alignment Attention, PyMAF for hand, face, body respectively then Integration. code: https://github.com/HongwenZhang/PyMAF-X 7.One-stage 3d whole-body mesh recovery with component aware transformer(2023) Upper Body Dataset for hand,face, perform RoIAlign on feature maps, then propose feature-level upsampling-crop strategy(multi scale) not image-level upsampling-crop(inspire by ViTDet) Keypoint-guided deformable attention decoder: use feature map Flr to regress each 2D keypoint positon as Tc (Q), deformable attention. code: https://github.com/IDEA-Research/OSX 8.Hybrik-x: Hybrid analytical-neural inverse kinematics for whole-body mesh recovery(2023) SMPL-x, Hybrik for hand, face, body respectively. code: https://github.com/Jeff-sjtu/HybrIK 9.PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop(2021) MAF: 3d joints-->2d joints-->point-wise features for each point(grid_sample)-->concat and dim reduction. Auxiliary Pixel-wise (IUV prediction)Supervision code: https://github.com/HongwenZhang/PyMAF 10.Hybrik: A hybrid analytical-neural inverse kinematics solution for 3d human pose and shape estimation(2021) rotation decomposition : twist(1-DoF),swing(2-DoF) see the code in comment, hasn't understand the details. code:https://github.com/Jeff-sjtu/HybrIK

[ouusan]

some other research: https://ps.is.mpg.de/person/otaheri : human-object interactions

[ouusan]

2. Spatial Transformer: https://arxiv.org/pdf/1506.02025 and code: 6.grid-feature and MAF fusion, BertAttention: https://github.com/HongwenZhang/PyMAF-X/blob/smplx/models/pymaf_net.py#L1433 Adaptive Integration with Elbow-Twist Compensation: Elbow-twist compensation adjusts the wrist and elbow poses by calculating the twist angle from the body expert’s predicted elbow-to-wrist alignment, applying a corrective rotation to ensure a natural wrist pose(not copy-paste in other methods) https://github.com/HongwenZhang/PyMAF-X/blob/smplx/models/pymaf_net.py#L421-L450 7.related works: Methods of Whole-body Mesh Recovery 7-34: Exploring Plain Vision Transformer Backbones for Object Detection https://arxiv.org/pdf/2203.16527 and code: https://github.com/facebookresearch/detectron2/tree/main/projects/ViTDet 7-39 Accurate 3D Hand Pose Estimation for Whole-Body 3D Human Mesh Estimation https://arxiv.org/pdf/2011.11534 and code: https://github.com/mks0601/Hand4Whole RELEASE 7-21: Mask-RCNN https://arxiv.org/abs/1703.06870 and code: https://github.com/facebookresearch/Detectron 7-75 deformable DETR https://arxiv.org/pdf/2010.04159 and code: https://github.com/fundamentalvision/Deformable-DETR
3. MAF (mesh alignment feature) Extractor: https://github.com/HongwenZhang/PyMAF/blob/smpl/models/maf_extractor.py#L136-L138 whole process：https://github.com/HongwenZhang/PyMAF/blob/smpl/models/pymaf_net.py#L260
4. hybrik code: https://github.com/Jeff-sjtu/HybrIK/blob/main/hybrik/models/layers/smpl/lbs.py#L291 (not understand??)

[ouusan]

7. (1)2d keypoint regression head: https://github.com/IDEA-Research/OSX/blob/main/main/transformer_utils/mmpose/models/heads/topdown_heatmap_simple_head.py#L266-L293 (2)task token (task_tokens are task-specific embeddings extracted from image features, where each token captures semantic information relevant to a particular task (e.g., shape, pose, expression) for multi-task learning) Especially, in this task the task_tokens_num=1+1+2+2+25 ( shape_token, cam_token, expr_token, jaw_pose_token, hand_token, body_pose_toke) : https://github.com/IDEA-Research/OSX/blob/main/main/transformer_utils/mmpose/models/backbones/vit.py#L207 https://github.com/IDEA-Research/OSX/blob/main/main/OSX.py#L197 Face and hand BBox estimation(all these network related func is in main/common/nets/module.py) https://github.com/IDEA-Research/OSX/blob/main/common/nets/module.py#L127