search

google-research / multinerf

A Code Release for Mip-NeRF 360, Ref-NeRF, and RawNeRF
Apache License 2.0
3.62k stars 341 forks source link

Tests failing on TPU v3-8 #75

Closed jenkspt closed 1 year ago

jenkspt commented 1 year ago

Tests are failing on TPU v3-8, but are not failing on my local machine.

----------------------------------------------------------------------
Ran 1 test in 5.057s

OK
F...
======================================================================
FAIL: test_compute_sq_dist_reference (tests.geopoly_test.GeopolyTest)
tests.geopoly_test.GeopolyTest.test_compute_sq_dist_reference
Test against a simple reimplementation of compute_sq_dist.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/penn/multinerf/tests/geopoly_test.py", line 53, in test_compute_sq_dist_reference
    np.testing.assert_allclose(sq_dist, sq_dist_ref, atol=1e-4, rtol=1e-4)
  File "/home/penn/anaconda3/envs/multinerf/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 1527, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/home/penn/anaconda3/envs/multinerf/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 844, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=0.0001, atol=0.0001

Mismatched elements: 8832 / 10000 (88.3%)
Max absolute difference: 0.09086609
Max relative difference: 0.01285233
 x: array([[25.8455  , 25.062511, 11.742369, ..., 24.252716, 27.659002,
        16.630625],
       [ 9.788469, 12.745399, 13.647413, ..., 33.37082 , 29.989532,...
 y: array([[25.83563 , 25.035769, 11.741554, ..., 24.258469, 27.685509,
        16.636297],
       [ 9.822387, 12.731453, 13.661249, ..., 33.37429 , 29.975922,...

----------------------------------------------------------------------
Ran 4 tests in 29.940s

FAILED (failures=1)
...............................Mean Error = 0.08026992529630661, Tolerance = 0.1
.Mean Error = 0.08638286590576172, Tolerance = 0.1
........................
----------------------------------------------------------------------
Ran 56 tests in 110.836s

OK
F...........PE of degree 5 has a maximum error of 2.5331974029541016e-06
.PE of degree 10 has a maximum error of 6.483495235443115e-05
.PE of degree 15 has a maximum error of 0.002378210425376892
.PE of degree 20 has a maximum error of 0.11622805148363113
.PE of degree 25 has a maximum error of 1.999955415725708
.PE of degree 30 has a maximum error of 1.999970555305481
....
======================================================================
FAIL: test_construct_ray_warps_extents_log (tests.coord_test.CoordTest)
tests.coord_test.CoordTest.test_construct_ray_warps_extents_log
test_construct_ray_warps_extents_log(<CompiledFunction of <function jax.numpy.log at 0x7f560081eee0>>)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/penn/anaconda3/envs/multinerf/lib/python3.9/site-packages/absl/testing/parameterized.py", line 320, in bound_param_test
    return test_method(self, *testcase_params)
  File "/home/penn/multinerf/tests/coord_test.py", line 194, in test_construct_ray_warps_extents
    np.testing.assert_allclose(
  File "/home/penn/anaconda3/envs/multinerf/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 1527, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/home/penn/anaconda3/envs/multinerf/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 844, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-05, atol=1e-05

Mismatched elements: 43 / 100 (43%)
Max absolute difference: 0.00045204
Max relative difference: 7.56597e-05
 x: array([ 2.400275,  1.668342,  2.044059,  6.927439,  1.078879,  0.115673,
        0.290508,  0.686725,  0.240134,  0.186716,  0.534207,  0.409191,
        2.090983,  0.41522 ,  0.722983,  1.309822,  0.97231 ,  0.64675 ,...
 y: array([ 2.400219,  1.668289,  2.044053,  6.927345,  1.078887,  0.115672,
        0.290508,  0.686718,  0.240134,  0.186729,  0.534207,  0.409187,
        2.090986,  0.415209,  0.722943,  1.309913,  0.972313,  0.646793,...

----------------------------------------------------------------------
Ran 21 tests in 52.253s

FAILED (failures=1)
.FF...
======================================================================
FAIL: test_mse_to_psnr_golden (tests.image_test.ImageTest)
tests.image_test.ImageTest.test_mse_to_psnr_golden
A lazy golden test for mse_to_psnr.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/penn/multinerf/tests/image_test.py", line 127, in test_mse_to_psnr_golden
    np.testing.assert_allclose(psnr, psnr_gt, atol=1E-5, rtol=1E-5)
  File "/home/penn/anaconda3/envs/multinerf/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 1527, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/home/penn/anaconda3/envs/multinerf/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 844, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-05, atol=1e-05

Mismatched elements: 28 / 64 (43.8%)
Max absolute difference: 0.00061035
Max relative difference: 0.00023579
 x: array([43.429222, 42.739685, 42.050312, 41.360874, 40.671413, 39.982204,
       39.29292 , 38.603603, 37.91449 , 37.225014, 36.535473, 35.846165,
       35.156944, 34.46737 , 33.777996, 33.088787, 32.399387, 31.709982,...
 y: array([43.429447, 42.74009 , 42.050735, 41.361378, 40.672024, 39.982666,
       39.29331 , 38.603954, 37.914597, 37.22524 , 36.535885, 35.846527,
       35.15717 , 34.46781 , 33.778458, 33.0891  , 32.399746, 31.710388,...

======================================================================
FAIL: test_psnr_mse_round_trip (tests.image_test.ImageTest)
tests.image_test.ImageTest.test_psnr_mse_round_trip
PSNR -> MSE -> PSNR is a no-op.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/penn/multinerf/tests/image_test.py", line 63, in test_psnr_mse_round_trip
    np.testing.assert_allclose(
  File "/home/penn/anaconda3/envs/multinerf/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 1527, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/home/penn/anaconda3/envs/multinerf/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 844, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-05, atol=1e-05

Mismatched elements: 1 / 1 (100%)
Max absolute difference: 0.00024223
Max relative difference: 1.21116638e-05
 x: array(20.000242, dtype=float32)
 y: array(20.)

----------------------------------------------------------------------
Ran 6 tests in 9.665s

FAILED (failures=2)
..
----------------------------------------------------------------------
Ran 2 tests in 10.279s

OK
.
----------------------------------------------------------------------
Ran 1 test in 7.430s

OK
.
----------------------------------------------------------------------
Ran 1 test in 9.908s

OK
.......
----------------------------------------------------------------------
Ran 7 tests in 10.172s

OK
....................................F..F.F
======================================================================
FAIL: test_control_points (tests.render_test.RenderTest)
tests.render_test.RenderTest.test_control_points
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/penn/multinerf/tests/render_test.py", line 271, in test_control_points
    np.testing.assert_allclose(
  File "/home/penn/anaconda3/envs/multinerf/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 1527, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/home/penn/anaconda3/envs/multinerf/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 844, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-05, atol=1e-05

Mismatched elements: 39 / 40 (97.5%)
Max absolute difference: 0.01288223
Max relative difference: 0.0065701
 x: array([[[ 2.039307,  0.678894],
        [ 0.678894,  0.228417]],
...
 y: array([[[ 2.03302 ,  0.676758],
        [ 0.676758,  0.227674]],
...

======================================================================
FAIL: test_inside_conical_frustum (tests.render_test.RenderTest)
tests.render_test.RenderTest.test_inside_conical_frustum
This test only tests helper functions used by other tests.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/penn/multinerf/tests/render_test.py", line 315, in test_inside_conical_frustum
    self.assertTrue(jnp.all(check(samples)))
AssertionError: DeviceArray(False, dtype=bool) is not true

======================================================================
FAIL: test_rotated_conic_frustums (tests.render_test.RenderTest)
tests.render_test.RenderTest.test_rotated_conic_frustums
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/penn/multinerf/tests/render_test.py", line 405, in test_rotated_conic_frustums
    np.testing.assert_allclose(rot_mean, gt_rot_mean, atol=1E-4, rtol=1E-4)
  File "/home/penn/anaconda3/envs/multinerf/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 1527, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/home/penn/anaconda3/envs/multinerf/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 844, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=0.0001, atol=0.0001

Mismatched elements: 2 / 3 (66.7%)
Max absolute difference: 0.00106061
Max relative difference: 0.00320256
 x: array([[-0.079809,  0.633021, -0.332234]], dtype=float32)
 y: array([[-0.080055,  0.633012, -0.331174]], dtype=float32)

----------------------------------------------------------------------
Ran 42 tests in 40.074s

FAILED (failures=3)
jonbarron commented 1 year ago

I would worry about it, training should still work.