Bug of brick attaching

[shogoinadomi]

Hello, I checked the rendered output of inference and noticed the program has some bug for attaching new block. (like the figure below). ( believe it comes from 'Connection-constrained 3D translation' so I'm now reading the code of hourglass_shape_cond_model.py 3D pose inference. Do you have any idea around this bug? Thank you in advance.

[kangyiyang]

Hello, I'm also recently replicating this paper. I gauss it's not a bug. According to the author's code, it only predicts the 3D positions of the bricks without explicitly modeling the physical constraints between the bricks. For example, the unrealistic situation of different bricks occupying the same position, as you mentioned.

[kangyiyang]

But I have some additional questions and ideas about this work. Perhaps we can exchange thoughts and discuss further. (I might share my questions and ideas in two days because I have some other matters to attend to during these two days. Looking forward to our discussion.)

[shogoinadomi]

Thank you for your comment (Sorry for the late reply).

I notice bricks.py addbrick() is the code for checking some 3D constraints, but I'm not sure whether it is comprehensive for preventing unrealistic connections. I'm now checking this part.

I found another bug, or unintended behavior. I guess in the non-regressive setting, the rendered output of bricks except the bricks that are newly arranged should be the same as previous blueprint, but it is not sometimes true. e.g. synthetic 000003 step 18. How about your visualization? Does this bug come from my environment?

[kangyiyang]

Indeed, I haven't paid much attention to the addbrick() function, perhaps some constraints are considered there. Also, when you mention "the same as the previous blueprint," are you referring to the ground truth or the previously generated output? I will take a look at my visualization results tomorrow and get back to you with a response.

[kangyiyang]

By the way, I have come across the following issue. Do you have any thoughts on it, or have you encountered similar results as well? ![Uploading 屏幕截图 2024-01-02 224131.png…]()

[kangyiyang]

[kangyiyang]

I just checked my results, and the outputs for steps 17 and 18 are as follows. However, I apologize as I still have some difficulty understanding your question.

[shogoinadomi]

• About my question : For example, the green block and red block is mistakenly swapped at step 17. The correct order is green ->red from left to right. If the inference argument is not set as 'autoregressive', I imagine the two blocks (green and red) in the output of step 18 should be corrected (green -> red) because the ground truth of step 17 is green -> red. However, the output of step 18 remains red -> green order. I think it should be fixed.

  

• About performance down : Yes, it happened at my environment too. I tested the pretrained model and the result didn't reach to the reported one. I think this is because of the unexpected behavior mentioned above.

[kangyiyang]

●Regarding your question, as per my understanding, not using autoregressive means using the ground truth 3D Lego as input and obtaining the positional information of the output bricks. However, during the rendering of the results, it adds this output to the currently constructed 3D Lego model, not the ground truth 3D Lego model. So, what is rendered is this output. ●Regarding the issue of performance degradation, the main challenge lies in not understanding why there is a significant drop in performance when comparing the model retrained by myself with the author's pre-trained model.

[shogoinadomi]

• Performance degradation for retraining : How about adding "--load_lpub \" to train_mepnet.sh ? When I deleted "--load_lpub \" from eval_mepnet.sh, the result became worse. I think changing image type to another one can improve performance. (I'm not sure the argument works for training too.)

[kangyiyang]

ok，thank you