search

automl / NASLib

NASLib is a Neural Architecture Search (NAS) library for facilitating NAS research for the community by providing interfaces to several state-of-the-art NAS search spaces and optimizers.
Apache License 2.0
528 stars 117 forks source link

Are tests expected to fail? #187

Closed tomaz-suller closed 3 months ago

tomaz-suller commented 3 months ago

I tried installing NASLib with different Python versions a few times, and all my installs lead to failing unit tests. Is it expected that the tests still fail on a working installation?

Below I'm copying the Pytest summary of the test run:

================================ test session starts ================================
platform linux -- Python 3.12.4, pytest-8.3.2, pluggy-1.5.0
rootdir: /home/tomaz/git/Politecnico/Extra/AI-Tech-Lab/NASLib
collected 85 items                                                                  

test_darts_search_space.py F.F.F..F....                                       [ 14%]
test_hierarchical_search_space.py FFFFFFF                                     [ 22%]
test_nb101_search_space.py .........                                          [ 32%]
test_nb201_search_space.py ...........                                        [ 45%]
test_nb301_search_space.py .F............                                     [ 62%]
test_optimizer_factory.py ....                                                [ 67%]
test_predictors.py ....F.F.                                                   [ 76%]
test_simple_cell_search_space.py FFFFFF                                       [ 83%]
test_utils.py ...FFFFFF.....                                                  [100%]

================================= warnings summary ==================================
../venv/lib/python3.12/site-packages/grakel/kernels/graph_hopper.py:9
  /home/tomaz/git/Politecnico/Extra/AI-Tech-Lab/NASLib/venv/lib/python3.12/site-packages/grakel/kernels/graph_hopper.py:9: PendingDeprecationWarning: Importing from numpy.matlib is deprecated since 1.19.0. The matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray. 
    from numpy.matlib import repmat

../venv/lib/python3.12/site-packages/tensorwatch/zmq_wrapper.py:7
  /home/tomaz/git/Politecnico/Extra/AI-Tech-Lab/NASLib/venv/lib/python3.12/site-packages/tensorwatch/zmq_wrapper.py:7: DeprecationWarning: zmq.eventloop.ioloop is deprecated in pyzmq 17. pyzmq now works with default tornado and asyncio eventloops.
    from zmq.eventloop import ioloop, zmqstream

tests/test_darts_search_space.py::DartsDartsIntegrationTest::test_feed_forward
tests/test_darts_search_space.py::DartsGdasIntegrationTest::test_feed_forward
tests/test_darts_search_space.py::DartsDrNasIntegrationTest::test_feed_forward
  /home/tomaz/git/Politecnico/Extra/AI-Tech-Lab/NASLib/naslib/search_spaces/nasbench301/graph.py:672: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
    alpha_softmax = F.softmax(alpha)

tests/test_darts_search_space.py::NasBench301SearchSpaceTest::test_update
tests/test_hierarchical_search_space.py::HierarchicalSearchSpaceTest::test_update
  /home/tomaz/git/Politecnico/Extra/AI-Tech-Lab/NASLib/naslib/utils/__init__.py:436: DeprecationWarning: Calling np.sum(generator) is deprecated, and in the future will give a different result. Use np.sum(np.fromiter(generator)) or the python sum builtin instead.
    np.sum(

tests/test_predictors.py::PredictorsTest::test_BLRPredictor
tests/test_predictors.py::PredictorsTest::test_BohamiannPredictor
tests/test_predictors.py::PredictorsTest::test_GPPredictor
tests/test_predictors.py::PredictorsTest::test_LGBPredictor
tests/test_predictors.py::PredictorsTest::test_NGBPredictor
tests/test_predictors.py::PredictorsTest::test_RFPredictor
tests/test_predictors.py::PredictorsTest::test_XGBPredictor
  /home/tomaz/git/Politecnico/Extra/AI-Tech-Lab/NASLib/venv/lib/python3.12/site-packages/sklearn/metrics/_regression.py:492: FutureWarning: 'squared' is deprecated in version 1.4 and will be removed in 1.6. To calculate the root mean squared error, use the function'root_mean_squared_error'.
    warnings.warn(

tests/test_predictors.py::PredictorsTest::test_BohamiannPredictor
  /home/tomaz/git/Politecnico/Extra/AI-Tech-Lab/NASLib/venv/lib/python3.12/site-packages/pybnn/bohamiann.py:118: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
    np.asarray(torch.tensor(parameter.data).numpy())

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================== short test summary info ==============================
FAILED test_darts_search_space.py::DartsDartsIntegrationTest::test_feed_forward - AssertionError: array(0.03983212, dtype=float32) != 0.5546151 within 3 places (0...
FAILED test_darts_search_space.py::DartsGdasIntegrationTest::test_feed_forward - AssertionError: array(0.03983212, dtype=float32) != 0.5546151 within 3 places (0...
FAILED test_darts_search_space.py::DartsDrNasIntegrationTest::test_feed_forward - AssertionError: array(0.03983212, dtype=float32) != 0.5546151 within 3 places (0...
FAILED test_darts_search_space.py::NasBench301SearchSpaceTest::test_update - AssertionError: 1.931898 != 1.930618
FAILED test_hierarchical_search_space.py::HierarchicalDartsIntegrationTest::test_feed_forward - RuntimeError: Given groups=1, weight of size [64, 64, 3, 3], expected input[2, 3...
FAILED test_hierarchical_search_space.py::HierarchicalDartsIntegrationTest::test_update - RuntimeError: Given groups=1, weight of size [64, 16, 3, 3], expected input[2, 3...
FAILED test_hierarchical_search_space.py::HierarchicalGdasIntegrationTest::test_feed_forward - RuntimeError: Given groups=1, weight of size [64, 64, 3, 3], expected input[2, 3...
FAILED test_hierarchical_search_space.py::HierarchicalGdasIntegrationTest::test_update - RuntimeError: Given groups=1, weight of size [64, 16, 3, 3], expected input[2, 3...
FAILED test_hierarchical_search_space.py::HierarchicalDrNasIntegrationTest::test_feed_forward - RuntimeError: Given groups=1, weight of size [64, 64, 3, 3], expected input[2, 3...
FAILED test_hierarchical_search_space.py::HierarchicalDrNasIntegrationTest::test_update - RuntimeError: Given groups=1, weight of size [64, 16, 3, 3], expected input[2, 3...
FAILED test_hierarchical_search_space.py::HierarchicalSearchSpaceTest::test_update - AssertionError: 7.900426 != 7.891546
FAILED test_nb301_search_space.py::NasBench301SearchSpaceTest::test_forward_pass_aux_head - AssertionError: torch.Size([3, 256, 8, 8]) != (3, 512, 8, 8)
FAILED test_predictors.py::PredictorsTest::test_NGBPredictor - AssertionError: 0.3333333333333333 != 0.2 within 3 places (0.1333333333333333 di...
FAILED test_predictors.py::PredictorsTest::test_XGBPredictor - AssertionError: 3.8909299214680977 != 3.68074010213 within 3 places (0.210189819...
FAILED test_simple_cell_search_space.py::SimpleCellDartsIntegrationTest::test_feed_forward - RuntimeError: Given groups=1, weight of size [64, 16, 3, 3], expected input[2, 3...
FAILED test_simple_cell_search_space.py::SimpleCellDartsIntegrationTest::test_update - RuntimeError: Given groups=1, weight of size [64, 16, 3, 3], expected input[2, 3...
FAILED test_simple_cell_search_space.py::SimpleCellGdasIntegrationTest::test_feed_forward - RuntimeError: Given groups=1, weight of size [64, 16, 3, 3], expected input[2, 3...
FAILED test_simple_cell_search_space.py::SimpleCellGdasIntegrationTest::test_update - RuntimeError: Given groups=1, weight of size [64, 16, 3, 3], expected input[2, 3...
FAILED test_simple_cell_search_space.py::SimpleCellDrNasIntegrationTest::test_feed_forward - RuntimeError: Given groups=1, weight of size [64, 16, 3, 3], expected input[2, 3...
FAILED test_simple_cell_search_space.py::SimpleCellDrNasIntegrationTest::test_update - RuntimeError: Given groups=1, weight of size [64, 16, 3, 3], expected input[2, 3...
FAILED test_utils.py::UtilsTest::test_get_config_from_args_default_bbo_bs - SystemExit: 2
FAILED test_utils.py::UtilsTest::test_get_config_from_args_default_nas - SystemExit: 2
FAILED test_utils.py::UtilsTest::test_get_config_from_args_default_nas_predictor - SystemExit: 2
FAILED test_utils.py::UtilsTest::test_get_config_from_args_default_oneshot - SystemExit: 2
FAILED test_utils.py::UtilsTest::test_get_config_from_args_default_predictor - SystemExit: 2
FAILED test_utils.py::UtilsTest::test_get_config_from_args_default_statistics - SystemExit: 2
============== 26 failed, 59 passed, 15 warnings in 118.76s (0:01:58) ===============
tomaz-suller commented 3 months ago

Never mind, realised this is a duplicate of #170