Open brumocas opened 2 weeks ago
TontonTremblay
commented
2 weeks ago I would guess, the belief looks really good, but not your pnp results, can you visualize the raw points? I think your cuboid order is off @nv-jeff I think this is similar to the error the other person is having.
brumocas
commented
2 weeks ago I am not familiar with the raw points, can you elaborate on how to get them? However, I think you are referring to the points attributed to each cuboid vertex.
Training Dataset orientation
I have checked my training data and the debug looks like this:
Based on this generated data the X indicating the box orientation is in the Nutrition Facts side of the cracker 3D model
Config file
After that I edited my config file and saw that I had different results when using the training vs the paper weight.
In the end the config stayed like this:
# Cuboid dimension in cm x,y,z
dimensions: {
# For my training
"cracker": [7.179999828338623, 16.403600692749023, 21.343700408935547],
# For paper weights
#"cracker": [16.403600692749023, 21.343700408935547,7.179999828338623]
}I had to follow the orientation that my 3D object had in meshlab as you can see in my first meshlab screenshot available in this issue (top left corner [size is in meters]) . You guys probably had a different one in your 3D model.
Inference results
Some inference new inference results:
I think that the output now is closer to your paper training weight, but yours is something else that I am not capable to achieve :( Any suggestion in how to improve my results? How did you build your dataset?
Future Work
If I want to apply this to train a novel 3D object do you think that Diff-DOPE might be a great option to obtain the final pose?
TontonTremblay
commented
2 weeks ago can you draw these https://github.com/NVlabs/Deep_Object_Pose/blob/master/ros1/dope/src/dope/inference/cuboid_pnp_solver.py#L101
obj_2d_points
so I think when you call obj_2d_points and obj_3d_points; the points are not aligned, for example let say the top right front is at index 0 in obj_2d_points but index 1 in obj_3d_points, that will cause your issues.
possibly might be caused by not using the right of these 2. Which now I do not remember which is which. https://github.com/NVlabs/Deep_Object_Pose/blob/master/ros1/dope/src/dope/inference/cuboid.py#L82
@nv-jeff Maybe we should clear all of this so there is only one way to generate a set of cuboid points.
brumocas
commented
2 weeks ago Hello again, and sorry for the delayed response.
I managed to solve the problem by changing the cuboid_pnp_solver.py and cuboid.py with the ones you suggested, only copy and paste.
Inference results
This is how my inference looks:
00000.json
00100.json
00200.json
00300.json
00400.json
00500.json
00600.json
00700.json
00800.json
00900.json
01000.json
01100.json
01200.json
01300.json
01400.json
01500.json
01600.json
01700.json
01800.json
01900.json
02000.json
02100.json
I think this is it for now, I will continue to play with DOPE Thanks :muscle:
TontonTremblay
commented
2 weeks ago ahhh this looks amazing good job, sorry I will check with @nv-jeff to see if he could make the update.
nv-jeff
commented
2 weeks ago Good sleuthing! I will take a look at the code and refactor the changes mentioned above.
After playing around with DOPE I decided to train the cracker 3D object to compare my training with the weights obtained from the training for the project paper.
3D ObjectDatasetI generated 83K images withblenderprocfollowing the guidelines provided in this repo/issues.TrainingI have been training this object for a couple days and my loss looks like this:The loss seems to be stuck around the
0.007value after approximately200 epochs. Is it still decreasing? I am concerned because the results are far from the obtained in the paper.Some inferences with my training:
Some beliefs with my training: 00000_belief:
00100_belief:
00200_belief:
00300_belief:
What can I do to improve this result, any suggestion? Thanks in advance :muscle: