Rubik's cube does not converge

[javierrodenas]

Hello,

I would like to share my experiments done so far.

I trained a model using Rubik's cube data. For that, I generated 20k images with NDDS tool, some examples:

I found some issues:

• The model is not converging, it gets stuck at 0.005-0.006
• The beliefs maps are completely black

I trained it with different learning rates, defining --object cube (and without --object) but it always gets stuck at 0.005-0.006

Could you help me? Am I doing something wrong?

Thank you in advance!

[mintar]

When the belief maps are completely black, that usually means that you have a typo in the object name. Double check that the object name in your _object_settings.json and train.py --object <object name> match exactly. Afterward, it should be possible to train your object.

I assume you later want to recognize the Rubik's cube in arbitrary configurations, i.e., even when all colors are mixed? To achieve that, your training data also has to consist of randomized configurations. You'll run into the symmetry problem then, which will show up in the belief maps as all corners being highlighted in all maps or something like that (but not completely black). That's solvable, but takes some work. Search the existing issues (also closed ones) for "symmetry":

https://github.com/NVlabs/Deep_Object_Pose/issues?q=is%3Aissue+symmetry

[javierrodenas]

Thanks for replying so fast.

I will check the object name again and give some feedback.

Interesting this symmetry issue, I will be careful. In my case I don't want to mix colors, I just want to detecte de Rubik's cube done

[javierrodenas]

The object name matchs exactly with the object in the configuration.

I didn't explain myself properly. The target beliefmaps show a white point and the output beliefmaps generated are completly black after some epochs, at the beginning it is full of colors.

Target beliefmaps:

Beliefmaps at the beginning:

Beliefmaps after some epochs:

[mintar]

That's strange. I don't know what's going on here.

Regarding the symmetry issue: As long as all sides of the cube have different colors, symmetry shouldn't be a problem.

[TontonTremblay]

Why is the target belief map black? Can you share a json file that NDDS exported with us please?

[javierrodenas]

Find below all the files generated from one example:

001589.png

001589 cs

001589 depth 16

001589 depth cm 8

001589 depth cm 16

001589 depth mm 16

001589 depth

001589 is

json_file :

{
    "camera_data":
    {
        "location_worldframe": [ 0, 0, 0 ],
        "quaternion_xyzw_worldframe": [ 0, 0, 0, 1 ]
    },
    "objects": [
        {
            "class": "Cube",
            "instance_id": 16777200,
            "visibility": 1,
            "location": [ 3.3921000957489014, -63.717498779296875, 618.34222412109375 ],
            "quaternion_xyzw": [ -0.18389999866485596, 0.82819998264312744, -0.36050000786781311, -0.38760000467300415 ],
            "pose_transform": [
                [ -0.50940001010894775, -0.73979997634887695, -0.43950000405311584, 0 ],
                [ -0.63179999589920044, -0.025200000032782555, 0.77469998598098755, 0 ],
                [ 0.58420002460479736, -0.67229998111724854, 0.4546000063419342, 0 ],
                [ 3.3921000957489014, -63.717498779296875, 618.34222412109375, 1 ]
            ],
            "cuboid_centroid": [ 3.3921000957489014, -63.717498779296875, 618.34222412109375 ],
            "projected_cuboid_centroid": [ 643.510986328125, 294.05081176757813 ],
            "bounding_box":
            {
                "top_left": [ 169.05459594726563, 508.22030639648438 ],
                "bottom_right": [ 410.24371337890625, 774.958984375 ]
            },
            "cuboid": [
                [ -39.268901824951172, -173.79890441894531, 678.65301513671875 ],
                [ 57.332599639892578, -169.95329284667969, 560.20758056640625 ],
                [ -32.058601379394531, -67.071502685546875, 490.64230346679688 ],
                [ -128.66009521484375, -70.917098999023438, 609.08770751953125 ],
                [ 38.842899322509766, -60.363399505615234, 746.0421142578125 ],
                [ 135.44430541992188, -56.517898559570313, 627.5968017578125 ],
                [ 46.053199768066406, 46.363998413085938, 558.031494140625 ],
                [ -50.548301696777344, 42.518398284912109, 676.476806640625 ]
            ],
            "projected_cuboid": [
                [ 602.96771240234375, 196.09989929199219 ],
                [ 705.49871826171875, 165.839599609375 ],
                [ 598.18231201171875, 272.5111083984375 ],
                [ 504.81021118164063, 285.48379516601563 ],
                [ 673.32177734375, 308.21661376953125 ],
                [ 778.12109375, 302.36520385742188 ],
                [ 692.81787109375, 413.17428588867188 ],
                [ 592.17742919921875, 400.22579956054688 ]
            ]
        }
    ]
}

Apart from that, find the configuration files:

_camerta_settings.json:

{
    "camera_settings": [
        {
            "name": "Viewpoint",
            "horizontal_fov": 90,
            "intrinsic_settings":
            {
                "resX": 1280,
                "resY": 720,
                "fx": 640,
                "fy": 640,
                "cx": 640,
                "cy": 360,
                "s": 0
            },
            "captured_image_size":
            {
                "width": 1280,
                "height": 720
            }
        }
    ]
}

_object_settings.json:

{
    "exported_object_classes": [
        "Cube"
    ],
    "exported_objects": [
        {
            "class": "Cube",
            "segmentation_class_id": 255,
            "segmentation_instance_id": 16777200,
            "fixed_model_transform": [
                [ 0, 0, 1, 0 ],
                [ -1, 0, 0, 0 ],
                [ 0, -1, 0, 0 ],
                [ -50.320098876953125, -50.3843994140625, -50.539901733398438, 1 ]
            ],
            "cuboid_dimensions": [ 152.89190673828125, 153.01890563964844, 153.33090209960938 ]
        }
    ]

}

Something weird?

Thank you in advance for helping me!

[TontonTremblay]

Thank you, try --object Cube upper case. You have a upper case in your json file. I thought I had .lower but I am not sure. Does the first training version makes a tensorboard, I forgot, in the new training script (which is not compatible with ndds data) it should have GT with the belief map drawn.