Open tp832944 opened 2 weeks ago
On NVIDIA GPU, TestTreeRecombination::test_reduce is failing.
Run tests/unit/test_solvers.py::TestTreeRecombination::test_reduce
3.12
0.3.1
Ubuntu 24.04 with NVIDIA GPU
jax[cuda12]
FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[with_jit-random-implicit-explicit-context0-False] - assert Array(False, dtype=bool) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[with_jit-random-implicit-explicit-context0-True] - assert Array(False, dtype=bool) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[with_jit-partial-null-implicit-explicit-context0-False] - assert Array(False, dtype=bool) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[with_jit-partial-null-implicit-explicit-context0-True] - assert Array(False, dtype=bool) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-partial-null-implicit-explicit-context0-False] - assert Array(False, dtype=bool) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-partial-null-implicit-explicit-context0-True] - assert Array(False, dtype=bool) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-partial-null-explicit-context2-False] - assert 16 <= Array(11, dtype=int32) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-partial-null-explicit-context2-True] - assert 16 <= Array(11, dtype=int32) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-random-implicit-explicit-context0-False] - assert Array(False, dtype=bool) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-random-implicit-explicit-context0-True] - assert Array(False, dtype=bool) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-random-explicit-context2-False] - assert 16 <= Array(11, dtype=int32) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-random-explicit-context2-True] - assert 16 <= Array(11, dtype=int32) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-null-implicit-explicit-context0-False] - assert 11 <= (11 - (11 - Array(1, dtype=int32))) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-null-implicit-explicit-context0-True] - assert 11 <= (11 - (11 - Array(1, dtype=int32))) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-null-explicit-context2-False] - assert 14 <= Array(1, dtype=int32) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-null-explicit-context2-True] - assert 14 <= Array(1, dtype=int32) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[with_jit-null-implicit-explicit-context0-False] - assert 11 <= (11 - (11 - Array(1, dtype=int32))) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[with_jit-null-implicit-explicit-context0-True] - assert 11 <= (11 - (11 - Array(1, dtype=int32))) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-full_rank-implicit-explicit-context0-False] - assert Array(False, dtype=bool) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-full_rank-implicit-explicit-context0-True] - assert Array(False, dtype=bool) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-full_rank-explicit-context2-False] - assert 6 <= Array(4, dtype=int32) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-full_rank-explicit-context2-True] - assert 6 <= Array(4, dtype=int32) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[with_jit-full_rank-implicit-explicit-context0-False] - assert Array(False, dtype=bool) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[with_jit-full_rank-implicit-explicit-context0-True] - assert Array(False, dtype=bool) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-rank_deficient-implicit-explicit-context0-False] - assert 3 <= (4 - (4 - Array(2, dtype=int32))) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-rank_deficient-implicit-explicit-context0-True] - assert 3 <= (4 - (4 - Array(2, dtype=int32))) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-rank_deficient-explicit-context2-False] - assert 3 <= Array(2, dtype=int32) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-rank_deficient-explicit-context2-True] - assert 3 <= Array(2, dtype=int32) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[with_jit-rank_deficient-implicit-explicit-context0-False] - assert 3 <= (4 - (4 - Array(2, dtype=int32))) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[with_jit-rank_deficient-implicit-explicit-context0-True] - assert 3 <= (4 - (4 - Array(2, dtype=int32))) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-excessive_test_functions-implicit-explicit-context0-False] - assert 84 <= (128 - (128 - Array(1, dtype=int32))) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-excessive_test_functions-implicit-explicit-context0-True] - assert 84 <= (128 - (128 - Array(1, dtype=int32))) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-excessive_test_functions-implicit-context1-False] - assert 84 <= (128 - (128 - Array(1, dtype=int32))) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-excessive_test_functions-implicit-context1-True] - assert 84 <= (128 - (128 - Array(1, dtype=int32))) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-excessive_test_functions-explicit-context2-False] - assert 84 <= Array(1, dtype=int32) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-excessive_test_functions-explicit-context2-True] - assert 84 <= Array(1, dtype=int32) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[with_jit-excessive_test_functions-implicit-explicit-context0-False] - assert 84 <= (128 - (128 - Array(1, dtype=int32))) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[with_jit-excessive_test_functions-implicit-explicit-context0-True] - assert 84 <= (128 - (128 - Array(1, dtype=int32))) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[with_jit-excessive_test_functions-implicit-context1-False] - assert 84 <= (128 - (128 - Array(1, dtype=int32))) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[with_jit-excessive_test_functions-implicit-context1-True] - assert 84 <= (128 - (128 - Array(1, dtype=int32)))
@tc85324 For visibility, we are seeing test failures on this (also see #853 ) - do you have any thoughts on the cause?
What's the problem?
On NVIDIA GPU, TestTreeRecombination::test_reduce is failing.
How can we reproduce the issue?
Run tests/unit/test_solvers.py::TestTreeRecombination::test_reduce
Python version
3.12
Package version
0.3.1
Operating system
Ubuntu 24.04 with NVIDIA GPU
Other packages
jax[cuda12]
Relevant log output
FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[with_jit-random-implicit-explicit-context0-False] - assert Array(False, dtype=bool) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[with_jit-random-implicit-explicit-context0-True] - assert Array(False, dtype=bool) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[with_jit-partial-null-implicit-explicit-context0-False] - assert Array(False, dtype=bool) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[with_jit-partial-null-implicit-explicit-context0-True] - assert Array(False, dtype=bool) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-partial-null-implicit-explicit-context0-False] - assert Array(False, dtype=bool) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-partial-null-implicit-explicit-context0-True] - assert Array(False, dtype=bool) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-partial-null-explicit-context2-False] - assert 16 <= Array(11, dtype=int32) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-partial-null-explicit-context2-True] - assert 16 <= Array(11, dtype=int32) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-random-implicit-explicit-context0-False] - assert Array(False, dtype=bool) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-random-implicit-explicit-context0-True] - assert Array(False, dtype=bool) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-random-explicit-context2-False] - assert 16 <= Array(11, dtype=int32) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-random-explicit-context2-True] - assert 16 <= Array(11, dtype=int32) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-null-implicit-explicit-context0-False] - assert 11 <= (11 - (11 - Array(1, dtype=int32))) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-null-implicit-explicit-context0-True] - assert 11 <= (11 - (11 - Array(1, dtype=int32))) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-null-explicit-context2-False] - assert 14 <= Array(1, dtype=int32) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-null-explicit-context2-True] - assert 14 <= Array(1, dtype=int32) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[with_jit-null-implicit-explicit-context0-False] - assert 11 <= (11 - (11 - Array(1, dtype=int32))) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[with_jit-null-implicit-explicit-context0-True] - assert 11 <= (11 - (11 - Array(1, dtype=int32))) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-full_rank-implicit-explicit-context0-False] - assert Array(False, dtype=bool) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-full_rank-implicit-explicit-context0-True] - assert Array(False, dtype=bool) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-full_rank-explicit-context2-False] - assert 6 <= Array(4, dtype=int32) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-full_rank-explicit-context2-True] - assert 6 <= Array(4, dtype=int32) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[with_jit-full_rank-implicit-explicit-context0-False] - assert Array(False, dtype=bool) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[with_jit-full_rank-implicit-explicit-context0-True] - assert Array(False, dtype=bool) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-rank_deficient-implicit-explicit-context0-False] - assert 3 <= (4 - (4 - Array(2, dtype=int32))) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-rank_deficient-implicit-explicit-context0-True] - assert 3 <= (4 - (4 - Array(2, dtype=int32))) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-rank_deficient-explicit-context2-False] - assert 3 <= Array(2, dtype=int32) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-rank_deficient-explicit-context2-True] - assert 3 <= Array(2, dtype=int32) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[with_jit-rank_deficient-implicit-explicit-context0-False] - assert 3 <= (4 - (4 - Array(2, dtype=int32))) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[with_jit-rank_deficient-implicit-explicit-context0-True] - assert 3 <= (4 - (4 - Array(2, dtype=int32))) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-excessive_test_functions-implicit-explicit-context0-False] - assert 84 <= (128 - (128 - Array(1, dtype=int32))) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-excessive_test_functions-implicit-explicit-context0-True] - assert 84 <= (128 - (128 - Array(1, dtype=int32))) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-excessive_test_functions-implicit-context1-False] - assert 84 <= (128 - (128 - Array(1, dtype=int32))) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-excessive_test_functions-implicit-context1-True] - assert 84 <= (128 - (128 - Array(1, dtype=int32))) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-excessive_test_functions-explicit-context2-False] - assert 84 <= Array(1, dtype=int32) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[without_jit-excessive_test_functions-explicit-context2-True] - assert 84 <= Array(1, dtype=int32) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[with_jit-excessive_test_functions-implicit-explicit-context0-False] - assert 84 <= (128 - (128 - Array(1, dtype=int32))) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[with_jit-excessive_test_functions-implicit-explicit-context0-True] - assert 84 <= (128 - (128 - Array(1, dtype=int32))) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[with_jit-excessive_test_functions-implicit-context1-False] - assert 84 <= (128 - (128 - Array(1, dtype=int32))) FAILED tests/unit/test_solvers.py::TestTreeRecombination::test_reduce[with_jit-excessive_test_functions-implicit-context1-True] - assert 84 <= (128 - (128 - Array(1, dtype=int32)))