Issue with Object Rotation in ARAP Deformation When Attached to Moving Limbs

[gcccccccccccc12345]

I'm using the ARAP deformation method to manipulate a person holding an object. Specifically, I'm trying to rotate the model's elbow to simulate the motion of moving the arm. However, I've noticed that while the arm itself moves correctly, the object in the hand only experiences translation without the expected rotation—resulting in an unnatural pose where the object doesn't maintain its relative orientation to the hand.

This behavior leads me to question whether the ARAP deformation process in this implementation includes optimization for rotation, especially in scenarios where connected objects should rotate together with the primary subject. I have noticed in your demonstrations that the hand of a person can rotate normally in conjunction with the arm, which leads to a natural and expected outcome. However, in my tests, when the hand is holding a larger object, it seems that the object does not rotate accordingly with the hand and arm, resulting in an unnatural appearance where the object doesn't maintain its relative orientation to the hand.

This discrepancy between the behavior observed in your demos and the issue I'm encountering with larger objects in hand makes me wonder if there's a specific limitation or consideration within the ARAP process that affects how rotation is applied or optimized in such scenarios.

I appreciate any insights or suggestions you might have regarding this issue. Thank you for your support and for the great work on SC-GS!

[yihua7]

Greetings and thank you for your interest in our work!

1. The observed situation is a result of inaccurate initialization of ARAP deformation. To optimize both translation and rotation to a global optima, a good initialization of ARAP is required. Currently, our code uses Laplacian deformation for initialization, which only minimizes the error of Laplacian coordinate that changes related to rotation. Hence Laplacian deformation is not robust enough for rotation, resulting in inaccurate initialization for rotation. As a result, some areas fail in achieving correct rotations in subsequent ARAP deformation results.

2. To address this issue, I recommend following these steps, of which the core idea is to include as many control points as possible for large-scale deformation: (1) If the hand and the object remain relatively rigid, use more control points to include the hand-held object and manipulate these control points to deform. This allows for better Laplacian deformation results and better initialization of ARAP deformation. (2) If non-rigid deformation needs to be applied between the hand and object, first add control points on the object to control its deformation relative to the hand. After this step is completed, hold D and further add control points on the hand. Then you can manipulate both hand and object to perform further deformations as needed.

3. More tips: (1) To more efficiently add handle points, you can set the parameter n_rings to 3 or 4 on the GUI interface. (2) You can press Node button to visualize control points and check are there any points in the region of interest missed. Press RGB to switch back the Gaussian rendering.

These are some operational tricks for deformation, which require a sufficient understanding of ARAP deformation or more practice and attempts. This will allow for a clearer understanding of how to operate and achieve the desired deformation results.

[yihua7]

Thanks for your feedback. Please refer to the latest commit fed5b8e49e12a2e5b8a5132988071c875bea03aa for more detailed editing tips!

[yihua7]

Hi,

1. Considering the difficulty you faced, I added an iterative strategy to perform ARAP editing. Please refer to editing mode in readme file for more details.

2. The initialization of ARAP deformation is replaced with the current positions rather than the Laplacian deformed initial points in the strategy arap_iterative. This may help to deal with the deformation of the large rotation you desire.

3. Although arap_iterative shows better performance in cases of large rotations, it may perform worse than arap_from_init in reproducing previous poses. You can try both of them and pick the most suitable one.

[gcccccccccccc12345]

Thank you very much for your swift and thorough response! Your explanation has been incredibly helpful and has provided me with a clearer understanding of the challenges I encountered during the ARAP deformation process.

I plan to continue testing and exploring improvements based on your suggestions as soon as time allows. Your guidance is invaluable to me, and I greatly appreciate your help and support.

[gcccccccccccc12345]

I've conducted further testing with the iterative deformation method you suggested, and I'm pleased to report that the object in the hand now rotates together with it as intended. There's a bit of unexpected deformation, which likely means I'll need to refine my editing approach further, but this is already a significant improvement.

[yihua7]

I'm glad to hear that.

arap_from_init would benefit from more experience in selecting as many points as possible, while arap_iterative requires careful operations to avoid any inverse effects. Starting with arap_from_init and then switching to arap_iterative may be another helpful choice.

Skinning and animating general objects without prior templates, like SMPL, can be a challenging task. That's why industrial companies are willing to pay a lot to hire experienced artists to do the job. Regardless, I hope that SC-GS will bring you some convenience in reconstructing and animating, and at least meet the basic requirements of most people. :)