search

tjiiv-cprg / EPro-PnP

[CVPR 2022 Oral, Best Student Paper] EPro-PnP: Generalized End-to-End Probabilistic Perspective-n-Points for Monocular Object Pose Estimation
https://www.youtube.com/watch?v=TonBodQ6EUU
Apache License 2.0
1.11k stars 106 forks source link

when I trained linemod dataset,I faced this problem about "nan value matrix in lowercholesky": #54

Open lululovecode opened 1 year ago

lululovecode commented 1 year ago 
raise ValueError(

ValueError: Expected parameter scale_tril (Tensor of shape (32, 4, 4)) of distribution AngularCentralGaussian(scale_tril: torch.Size([32, 4, 4])) to satisfy the constraint LowerCholesky(), but found invalid values: tensor([[[ 6.9278e-01, 0.0000e+00, 0.0000e+00, 0.0000e+00], [ 2.9031e-02, 1.3608e-01, 0.0000e+00, 0.0000e+00], [-4.0746e-02, -1.0482e-02, 1.4604e-01, 0.0000e+00], [-1.8699e-03, -1.1016e-01, -1.9344e-01, 6.5453e-01]],

    [[ 6.9375e-01,  0.0000e+00,  0.0000e+00,  0.0000e+00],
     [ 8.8277e-02,  1.0273e-01,  0.0000e+00,  0.0000e+00],
     [-3.0444e-02, -2.4733e-02,  1.4973e-01,  0.0000e+00],
     [-2.5767e-03, -1.1656e-01, -4.2385e-01,  5.3241e-01]],

    [[ 6.2876e-01,  0.0000e+00,  0.0000e+00,  0.0000e+00],
     [-1.1530e-01,  3.4448e-01,  0.0000e+00,  0.0000e+00],
     [ 2.3507e-02,  2.2160e-01,  2.0328e-01,  0.0000e+00],
     [ 2.8570e-02,  6.3889e-02, -6.1317e-01,  3.1275e-02]],

    [[ 7.0017e-01,  0.0000e+00,  0.0000e+00,  0.0000e+00],
     [ 6.3775e-02,  8.3764e-02,  0.0000e+00,  0.0000e+00],
     [-1.4931e-02,  1.9444e-02,  1.3604e-01,  0.0000e+00],
     [ 1.4877e-03, -9.4761e-03, -3.4578e-01,  6.0015e-01]],

    [[ 6.5502e-01,  0.0000e+00,  0.0000e+00,  0.0000e+00],
     [ 1.6822e-01,  2.1807e-01,  0.0000e+00,  0.0000e+00],
     [-1.4797e-01,  7.2127e-02,  2.0942e-01,  0.0000e+00],
     [-8.0459e-04, -4.5535e-01, -4.2245e-01,  1.9658e-01]],

    [[ 5.0006e-01,  0.0000e+00,  0.0000e+00,  0.0000e+00],
     [-2.2124e-05,  5.0106e-01,  0.0000e+00,  0.0000e+00],
     [-3.0428e-03,  1.5494e-04,  4.9882e-01,  0.0000e+00],
     [-2.6464e-04, -9.0350e-05, -6.1620e-04,  5.0104e-01]],

    [[ 6.7783e-01,  0.0000e+00,  0.0000e+00,  0.0000e+00],
     [-5.6427e-02,  2.2465e-01,  0.0000e+00,  0.0000e+00],
     [ 5.6638e-02,  2.2868e-02,  1.6187e-01,  0.0000e+00],
     [ 5.5260e-03,  2.5387e-01,  7.3131e-02,  6.2260e-01]],

    [[ 6.9216e-01,  0.0000e+00,  0.0000e+00,  0.0000e+00],
     [ 8.3768e-02,  7.2142e-02,  0.0000e+00,  0.0000e+00],
     [-5.9350e-02,  6.1372e-02,  1.3366e-01,  0.0000e+00],
     [-3.3742e-03, -1.4091e-01, -3.1739e-01,  6.0266e-01]],

    [[ 5.2461e-01,  0.0000e+00,  0.0000e+00,  0.0000e+00],
     [ 1.4181e-01,  4.8738e-01,  0.0000e+00,  0.0000e+00],
     [-2.8058e-01,  6.1036e-02,  2.9926e-01,  0.0000e+00],
     [-1.9840e-02, -1.4331e-01, -4.9047e-01,  1.8470e-01]],

    [[ 6.9892e-01,  0.0000e+00,  0.0000e+00,  0.0000e+00],
     [ 8.6890e-02,  7.2151e-02,  0.0000e+00,  0.0000e+00],
     [-6.6552e-03,  5.1003e-03,  1.4996e-01,  0.0000e+00],
     [-4.9842e-04,  8.9434e-02, -4.3535e-01,  5.2813e-01]],

    [[        nan,  0.0000e+00,  0.0000e+00,  0.0000e+00],
     [        nan,         nan,  0.0000e+00,  0.0000e+00],
     [        nan,         nan,         nan,  0.0000e+00],
     [        nan,         nan,         nan,         nan]],

    [[        nan,  0.0000e+00,  0.0000e+00,  0.0000e+00],
     [        nan,         nan,  0.0000e+00,  0.0000e+00],
     [        nan,         nan,         nan,  0.0000e+00],
     [        nan,         nan,         nan,         nan]],

    [[ 6.8809e-01,  0.0000e+00,  0.0000e+00,  0.0000e+00],
     [ 8.6090e-02,  1.2643e-01,  0.0000e+00,  0.0000e+00],
     [-4.5414e-02, -1.9174e-02,  1.7456e-01,  0.0000e+00],
     [-2.0333e-03, -1.9706e-01, -3.8389e-01,  5.3329e-01]],

    [[ 6.9466e-01,  0.0000e+00,  0.0000e+00,  0.0000e+00],
     [-9.5166e-02,  6.8402e-02,  0.0000e+00,  0.0000e+00],
     [ 6.3173e-02,  6.1081e-02,  1.1666e-01,  0.0000e+00],
     [-1.6568e-04, -6.6022e-01, -2.1303e-01,  3.3896e-02]],

    [[ 7.0403e-01,  0.0000e+00,  0.0000e+00,  0.0000e+00],
     [-1.7117e-02,  7.3840e-02,  0.0000e+00,  0.0000e+00],
     [ 1.7155e-02, -5.4969e-03,  1.1718e-01,  0.0000e+00],
     [ 2.3101e-03,  3.0500e-01,  9.6146e-02,  6.1848e-01]],

    [[        nan,  0.0000e+00,  0.0000e+00,  0.0000e+00],
     [        nan,         nan,  0.0000e+00,  0.0000e+00],
     [        nan,         nan,         nan,  0.0000e+00],
     [        nan,         nan,         nan,         nan]],

    [[ 6.9661e-01,  0.0000e+00,  0.0000e+00,  0.0000e+00],
     [ 9.8271e-02,  7.8439e-02,  0.0000e+00,  0.0000e+00],
     [-1.8928e-03, -3.1093e-02,  1.5061e-01,  0.0000e+00],
     [-6.6284e-04,  8.7804e-02, -4.2872e-01,  5.3294e-01]],

    [[ 6.9515e-01,  0.0000e+00,  0.0000e+00,  0.0000e+00],
     [-4.2702e-03,  1.2637e-01,  0.0000e+00,  0.0000e+00],
     [-6.0961e-02, -1.8748e-02,  1.2973e-01,  0.0000e+00],
     [-2.8375e-05, -2.4774e-01, -2.6542e-02,  6.4665e-01]],

    [[ 4.3950e-01,  0.0000e+00,  0.0000e+00,  0.0000e+00],
     [-1.1706e-01,  4.5726e-01,  0.0000e+00,  0.0000e+00],
     [ 2.1441e-01,  1.5311e-01,  3.8555e-01,  0.0000e+00],
     [-2.5493e-01, -3.3683e-01,  4.2722e-01,  7.2938e-02]],

    [[ 5.0024e-01,  0.0000e+00,  0.0000e+00,  0.0000e+00],
     [-2.5960e-06,  5.0026e-01,  0.0000e+00,  0.0000e+00],
     [ 4.3107e-05,  2.4820e-06,  5.0024e-01,  0.0000e+00],
     [ 9.6762e-06,  3.2536e-06, -8.9122e-06,  5.0026e-01]],

    [[        nan,  0.0000e+00,  0.0000e+00,  0.0000e+00],
     [        nan,         nan,  0.0000e+00,  0.0000e+00],
     [        nan,         nan,         nan,  0.0000e+00],
     [        nan,         nan,         nan,         nan]],

    [[ 6.4351e-01,  0.0000e+00,  0.0000e+00,  0.0000e+00],
     [ 1.6009e-01,  2.0367e-01,  0.0000e+00,  0.0000e+00],
     [-2.1300e-01,  1.4744e-01,  1.1538e-01,  0.0000e+00],
     [ 1.3387e-02, -5.5562e-01, -1.8492e-01,  3.0911e-01]],

    [[ 6.9162e-01,  0.0000e+00,  0.0000e+00,  0.0000e+00],
     [ 7.6235e-02,  1.1996e-01,  0.0000e+00,  0.0000e+00],
     [-4.5653e-02,  1.4912e-03,  1.5985e-01,  0.0000e+00],
     [-1.4997e-03, -2.1666e-01, -3.5258e-01,  5.5036e-01]],

    [[ 6.9432e-01,  0.0000e+00,  0.0000e+00,  0.0000e+00],
     [ 9.2331e-02,  9.8162e-02,  0.0000e+00,  0.0000e+00],
     [-4.2645e-02,  5.4324e-02,  1.3582e-01,  0.0000e+00],
     [ 1.3122e-03, -2.1678e-01, -3.8661e-01,  5.2948e-01]],

    [[ 6.9436e-01,  0.0000e+00,  0.0000e+00,  0.0000e+00],
     [ 9.8185e-02,  9.2633e-02,  0.0000e+00,  0.0000e+00],
     [-3.9998e-02, -1.0557e-02,  1.3649e-01,  0.0000e+00],
     [-2.7412e-03, -1.6697e-01, -4.4619e-01,  5.0259e-01]],

    [[ 6.4129e-01,  0.0000e+00,  0.0000e+00,  0.0000e+00],
     [ 6.3952e-02,  3.1922e-01,  0.0000e+00,  0.0000e+00],
     [-1.2256e-01, -7.8753e-02,  1.8769e-01,  0.0000e+00],
     [ 1.5459e-02, -2.4698e-01, -6.0131e-01,  6.0706e-02]],

    [[ 6.9315e-01,  0.0000e+00,  0.0000e+00,  0.0000e+00],
     [ 2.9403e-02,  1.4035e-01,  0.0000e+00,  0.0000e+00],
     [-1.3751e-02, -2.0118e-02,  1.5011e-01,  0.0000e+00],
     [-1.5066e-03,  1.8742e-02, -1.7723e-01,  6.6670e-01]],

    [[ 4.8264e-01,  0.0000e+00,  0.0000e+00,  0.0000e+00],
     [ 2.5738e-01,  2.7450e-01,  0.0000e+00,  0.0000e+00],
     [-4.3105e-02,  4.6674e-01,  3.4674e-01,  0.0000e+00],
     [-2.5008e-02,  3.7465e-01, -3.6509e-01,  1.0737e-01]],

    [[ 7.0231e-01,  0.0000e+00,  0.0000e+00,  0.0000e+00],
     [ 3.4718e-02,  8.4747e-02,  0.0000e+00,  0.0000e+00],
     [-3.8304e-02,  3.7433e-02,  9.7477e-02,  0.0000e+00],
     [ 2.3108e-03, -1.8914e-01, -1.5585e-01,  6.5282e-01]],

    [[ 6.3330e-01,  0.0000e+00,  0.0000e+00,  0.0000e+00],
     [ 6.3693e-02,  3.1222e-01,  0.0000e+00,  0.0000e+00],
     [-1.5724e-01, -1.3506e-01,  1.9361e-01,  0.0000e+00],
     [-1.1413e-02, -2.2387e-01, -5.9659e-01,  1.0497e-01]],

    [[ 6.6189e-01,  0.0000e+00,  0.0000e+00,  0.0000e+00],
     [ 1.2698e-01,  2.1568e-01,  0.0000e+00,  0.0000e+00],
     [-1.6756e-01,  1.6547e-01,  1.9803e-02,  0.0000e+00],
     [-3.9190e-03, -3.9201e-01,  1.0520e-01,  5.2803e-01]],

    [[        nan,  0.0000e+00,  0.0000e+00,  0.0000e+00],
     [        nan,         nan,  0.0000e+00,  0.0000e+00],
     [        nan,         nan,         nan,  0.0000e+00],
     [        nan,         nan,         nan,         nan]]],
   device='cuda:0')

Do you know how can I solve the problem?

Lakonik commented 1 year ago

Looks like training is not very stable. What data are you using? Btw, we do not recommend using the CDPN network directly for other applications, as we have not fully optimized the architecture and hyper parameters for EPro-PnP (since it's mostly used for comparison with the baseline in the paper). In fact we recently noted a design fault related to the global weight scale. An improved version with extended paper and code will be released in a few weeks.