search

NVlabs / Deep_Object_Pose

Deep Object Pose Estimation (DOPE) – ROS inference (CoRL 2018)
Other
1.02k stars 287 forks source link

Loss gets stuck #193

Open javierrodenas opened 2 years ago

javierrodenas commented 2 years ago

Hello,

I am experimenting the following issue:

image

It is getting stuck around 0.006-0.005. Does it happen to you? It doe not matter which kind of dataset I use, the behavior is the same.

Thank you in advance

TontonTremblay commented 2 years ago

I am not sure, what sort of GPU do you have? Which data are you using?

javierrodenas commented 2 years ago

I am using the 2080 RTX Ti (12 GB).

On the other hand, I am using this data currently (some examples): 009060 009062

My experience with this environment is that it does not matter which data I use, the loss is always around 0.005

javierrodenas commented 2 years ago

I was testing different parameters with same type of data:

Green: learning_rate 0.0001 and batch_size 16 Blue: learning_rate 0.001 and batch_size 8

TOTAL TRAIN LOSS image

AFFINITY TRAIN LOSS

image

BELIEF MAPS TRAIN LOSS

image

The result of train loss is more of less the same and it gets stuck around 0.005 (TOTAL LOSS).

Any recommendation? Did you see this behavior before?

Thank you in advance!

TontonTremblay commented 2 years ago

It might be caused by the slightly symmetrical object. Do you have non symmetrical object to try on?

javierrodenas commented 2 years ago

I can try with non symetrical object.

Another possible issue here could be the color. I comment that because in another github issue I saw a guy who was playing with different color cubes #128. Could my color be an issue? It is white/light grey.

I removed the noise and the loss got to 0.003 (It decreased from 0.005 to 0.003) but it is not able to detect the piece during the predictions

TontonTremblay commented 2 years ago

can you share how the belief maps look like?