search

silviazuffi / smalr_online

Other
84 stars 8 forks source link

Align SMAL model to new 3D scan. #4

Open giles-hu opened 3 years ago

giles-hu commented 3 years ago

Hi, may I ask for how to align SMAL model to a new 3D scan? We already got the CT high-resolution scan. The code for the original SMAL model is not provided. Therefore, I tried to fit the scan with the SMAL model based on Flame (https://github.com/Rubikplayer/flame-fitting). I follow the code of FLAME and use a similar objective function ( scan-to-mesh distance, landmarks distance, pose penalty, shape penalty with similar weights). I defined 54 points (from head to tail) on both the SMAL model and our CT scan using MeshLab as FLAME. However, the result of fitting SMAL to a scan based on FLAME code is very poor. To eliminate the possibility that the SMAL model did not cover rodents, I also try to fit the Bear mesh provided in the result part of this SMALR project. It is also not working. Could you please give me some advice on it? Is it possible to share the code about how to build a SMAL model based on scan?

silviazuffi commented 3 years ago

Hi,

I do not think SMAL can work for rodents, they are too different from the animals in the SMAL training set. FLAME is for faces, I guess it is not directly applicable to an articulated animal. I can imagine you also have poor results with the bear, as the point of doing the refinement with SMALR is that you cannot get a bear with SMAL, as it is also not in the training set. If you want to fit the model to a 3D scan you can use the code from Benjamin Biggs: https://github.com/benjiebob/SMALify.

Best

On 28. May 2021, at 20:39, giles-hu @.***> wrote:

Hi, may I ask for how to align SMAL model to a new 3D scan? We already got the CT high-resolution scan. The code for the original SMAL model is not provided. Therefore, I tried to fit the scan with the SMAL model based on Flame (https://github.com/Rubikplayer/flame-fitting https://github.com/Rubikplayer/flame-fitting). I follow the code of FLAME and use a similar objective function ( scan-to-mesh distance, landmarks distance, pose penalty, shape penalty with similar weights). I defined 54 points (from head to tail) on both the SMAL model and our CT scan using MeshLab as FLAME. However, the result of fitting SMAL to a scan based on FLAME code is very poor. To eliminate the possibility that the SMAL model did not cover rodents, I also try to fit the Bear mesh provided in the result part of this SMALR project. It is also not working. Could you please give me some advice on it? Is it possible to share the code about how to build a SMAL model based on scan?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/silviazuffi/smalr_online/issues/4, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNYVFLJSPXPR3QXMOMRYFLTP7PONANCNFSM45XHEEYQ.

giles-hu commented 3 years ago

Hi, I do not think SMAL can work for rodents, they are too different from the animals in the SMAL training set. FLAME is for faces, I guess it is not directly applicable to an articulated animal. I can imagine you also have poor results with the bear, as the point of doing the refinement with SMALR is that you cannot get a bear with SMAL, as it is also not in the training set. If you want to fit the model to a 3D scan you can use the code from Benjamin Biggs: https://github.com/benjiebob/SMALify. Best … On 28. May 2021, at 20:39, giles-hu @.***> wrote: Hi, may I ask for how to align SMAL model to a new 3D scan? We already got the CT high-resolution scan. The code for the original SMAL model is not provided. Therefore, I tried to fit the scan with the SMAL model based on Flame (https://github.com/Rubikplayer/flame-fitting https://github.com/Rubikplayer/flame-fitting). I follow the code of FLAME and use a similar objective function ( scan-to-mesh distance, landmarks distance, pose penalty, shape penalty with similar weights). I defined 54 points (from head to tail) on both the SMAL model and our CT scan using MeshLab as FLAME. However, the result of fitting SMAL to a scan based on FLAME code is very poor. To eliminate the possibility that the SMAL model did not cover rodents, I also try to fit the Bear mesh provided in the result part of this SMALR project. It is also not working. Could you please give me some advice on it? Is it possible to share the code about how to build a SMAL model based on scan? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#4>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNYVFLJSPXPR3QXMOMRYFLTP7PONANCNFSM45XHEEYQ.

Hi, Thanks a lot. I've tried SMALify and added an MSEloss for landmarks in objective function. However, it still return poor result. I think it might be caused by that their objective function only contains a few regularization terms (in a very general way) and chamfer distance. In addition, it tries to fit all parameters at the same time. I am now trying to using the GLoSS fitting on your SMAL paper (CVPR 2017). The objective function contains more regularization terms. Could you please provide the information about the mesh segmentation of the SMAL model and some key matrix of the SMAL model like average pose displacements B{s,i}, B{p,i}? Many thanks in advance. Hope you have a good weekend.

silviazuffi commented 3 years ago

Hi, Thanks a lot. I've tried SMALify and added an MSEloss for landmarks in objective function. However, it still return poor result. I think it might be caused by that their objective function only contains a few regularization terms (in a very general way) and chamfer distance. In addition, it tries to fit all parameters at the same time. I am now trying to using the GLoSS fitting on your SMAL paper (CVPR 2017). The objective function contains more regularization terms. Could you please provide the information about the mesh segmentation of the SMAL model and some key matrix of the SMAL model like average pose displacements B{s,i}, B{p,i}? Many thanks in advance. Hope you have a good weekend.

Hi, The Gloss fitting requires the Gloss model, and we do not make the model or the alignment code for Gloss available. In this paper (https://arxiv.org/pdf/2007.11110.pdf) they apply limb scaling, you could try with a similar approach.

Best