google-research / multinerf

A Code Release for Mip-NeRF 360, Ref-NeRF, and RawNeRF
Apache License 2.0
3.58k stars 339 forks source link

Tests & training fail on Google TPU VM #44

Open Palisand opened 1 year ago

Palisand commented 1 year ago

TPU type: v3-8 TPU software version: tpu-vm-tf-2.9.1

Installed PIP packages / `pip freeze` output (click to expand) ``` absl-py==1.2.0 asttokens==2.0.8 astunparse==1.6.3 backcall==0.2.0 cachetools==5.2.0 certifi @ file:///opt/conda/conda-bld/certifi_1655968806487/work/certifi charset-normalizer==2.1.1 chex==0.1.4 colorama==0.4.5 commonmark==0.9.1 cycler==0.11.0 decorator==5.1.1 dm-pix==0.3.3 dm-tree==0.1.7 etils==0.7.1 executing==1.0.0 flatbuffers==1.12 flax==0.6.0 fonttools==4.37.1 gast==0.4.0 gin-config==0.5.0 google-auth==2.11.0 google-auth-oauthlib==0.4.6 google-pasta==0.2.0 grpcio==1.48.1 h5py==3.7.0 idna==3.3 importlib-metadata==4.12.0 importlib-resources==5.9.0 ipython==8.5.0 jax==0.3.17 jaxlib==0.3.15 jedi==0.18.1 keras==2.9.0 Keras-Preprocessing==1.1.2 kiwisolver==1.4.4 libclang==14.0.6 Markdown==3.4.1 MarkupSafe==2.1.1 matplotlib==3.5.3 matplotlib-inline==0.1.6 mediapy==1.1.0 msgpack==1.0.4 numpy==1.23.3 oauthlib==3.2.1 opencv-python==4.6.0.66 opt-einsum==3.3.0 optax==0.1.3 packaging==21.3 parso==0.8.3 pexpect==4.8.0 pickleshare==0.7.5 Pillow==9.2.0 prompt-toolkit==3.0.31 protobuf==3.19.4 ptyprocess==0.7.0 pure-eval==0.2.2 pyasn1==0.4.8 pyasn1-modules==0.2.8 Pygments==2.13.0 pyparsing==3.0.9 python-dateutil==2.8.2 PyYAML==6.0 rawpy==0.17.2 requests==2.28.1 requests-oauthlib==1.3.1 rich==11.2.0 rsa==4.9 scipy==1.9.1 six==1.16.0 stack-data==0.5.0 tensorboard==2.9.1 tensorboard-data-server==0.6.1 tensorboard-plugin-wit==1.8.1 tensorflow==2.9.1 tensorflow-estimator==2.9.0 tensorflow-io-gcs-filesystem==0.27.0 termcolor==2.0.0 toolz==0.12.0 traitlets==5.3.0 typing_extensions==4.3.0 urllib3==1.26.12 wcwidth==0.2.5 Werkzeug==2.2.2 wrapt==1.14.1 zipp==3.8.1 ```

Note the tensorflow version matches the TPU software version (2.9.1).

The test failures:

2022-09-11 22:45:00.888529: F external/org_tensorflow/tensorflow/core/tpu/tpu_library_init_fns.inc:100] TpuEmbeddingEngine_CollateMemory not available in this library.
./scripts/run_all_unit_tests.sh: line 17: 14504 Aborted                 (core dumped) python -m unittest tests.camera_utils_test
2022-09-11 22:45:38.372593: F external/org_tensorflow/tensorflow/core/tpu/tpu_library_init_fns.inc:100] TpuEmbeddingEngine_CollateMemory not available in this library.
./scripts/run_all_unit_tests.sh: line 18: 14981 Aborted                 (core dumped) python -m unittest tests.geopoly_test
2022-09-11 22:45:40.651496: F external/org_tensorflow/tensorflow/core/tpu/tpu_library_init_fns.inc:100] TpuEmbeddingEngine_CollateMemory not available in this library.
./scripts/run_all_unit_tests.sh: line 19: 15432 Aborted                 (core dumped) python -m unittest tests.stepfun_test
2022-09-11 22:45:42.753890: F external/org_tensorflow/tensorflow/core/tpu/tpu_library_init_fns.inc:100] TpuEmbeddingEngine_CollateMemory not available in this library.
./scripts/run_all_unit_tests.sh: line 20: 15884 Aborted                 (core dumped) python -m unittest tests.coord_test
2022-09-11 22:45:44.940510: F external/org_tensorflow/tensorflow/core/tpu/tpu_library_init_fns.inc:100] TpuEmbeddingEngine_CollateMemory not available in this library.
./scripts/run_all_unit_tests.sh: line 21: 16335 Aborted                 (core dumped) python -m unittest tests.image_test
2022-09-11 22:45:47.053111: F external/org_tensorflow/tensorflow/core/tpu/tpu_library_init_fns.inc:100] TpuEmbeddingEngine_CollateMemory not available in this library.
./scripts/run_all_unit_tests.sh: line 22: 16788 Aborted                 (core dumped) python -m unittest tests.ref_utils_test
2022-09-11 22:45:49.244992: F external/org_tensorflow/tensorflow/core/tpu/tpu_library_init_fns.inc:100] TpuEmbeddingEngine_CollateMemory not available in this library.
./scripts/run_all_unit_tests.sh: line 23: 17259 Aborted                 (core dumped) python -m unittest tests.utils_test
2022-09-11 22:45:51.564844: F external/org_tensorflow/tensorflow/core/tpu/tpu_library_init_fns.inc:100] TpuEmbeddingEngine_CollateMemory not available in this library.
./scripts/run_all_unit_tests.sh: line 24: 17712 Aborted                 (core dumped) python -m unittest tests.datasets_test
2022-09-11 22:45:53.660296: F external/org_tensorflow/tensorflow/core/tpu/tpu_library_init_fns.inc:100] TpuEmbeddingEngine_CollateMemory not available in this library.
./scripts/run_all_unit_tests.sh: line 25: 18165 Aborted                 (core dumped) python -m unittest tests.math_test
2022-09-11 22:45:55.757259: F external/org_tensorflow/tensorflow/core/tpu/tpu_library_init_fns.inc:100] TpuEmbeddingEngine_CollateMemory not available in this library.
./scripts/run_all_unit_tests.sh: line 26: 18616 Aborted                 (core dumped) python -m unittest tests.render_test

Training errors:

~/multinerf$ python -m train \
>   --gin_configs=configs/360.gin \
>   --gin_bindings="Config.data_dir = '${DATA_DIR}'" \
>   --gin_bindings="Config.checkpoint_dir = '${DATA_DIR}/checkpoints'" \
>   --logtostderr
2022-09-11 23:06:43.482723: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/usr/local/lib
WARNING:absl:GlobalAsyncCheckpointManager is not imported correctly. Checkpointing of GlobalDeviceArrays will not be available.To use the feature, install tensorstore.
2022-09-11 23:06:44.899501: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/cv2/../../lib64::/usr/local/lib
2022-09-11 23:06:44.899561: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
I0911 23:06:44.919915 139717675481088 xla_bridge.py:350] Unable to initialize backend 'tpu_driver': NOT_FOUND: Unable to find driver in registry given worker:
I0911 23:06:44.920102 139717675481088 xla_bridge.py:350] Unable to initialize backend 'cuda': module 'jaxlib.xla_extension' has no attribute 'GpuAllocatorConfig'
I0911 23:06:44.920187 139717675481088 xla_bridge.py:350] Unable to initialize backend 'rocm': module 'jaxlib.xla_extension' has no attribute 'GpuAllocatorConfig'
2022-09-11 23:06:44.974486: F external/org_tensorflow/tensorflow/core/tpu/tpu_library_init_fns.inc:100] TpuEmbeddingEngine_CollateMemory not available in this library.
Fatal Python error: Aborted

Thread 0x00007f10558b2700 (most recent call first):
  File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/threading.py", line 316 in wait
  File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/threading.py", line 581 in wait
  File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/threading.py", line 1304 in run
  File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/threading.py", line 980 in _bootstrap_inner
  File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/threading.py", line 937 in _bootstrap

Current thread 0x00007f128e6a7800 (most recent call first):
  File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/jaxlib/xla_client.py", line 110 in make_tpu_client
  File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/jax/_src/lib/xla_bridge.py", line 195 in tpu_client_timer_callback
  File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/jax/_src/lib/xla_bridge.py", line 382 in _init_backend
  File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/jax/_src/lib/xla_bridge.py", line 331 in backends
  File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/jax/_src/lib/xla_bridge.py", line 406 in _get_backend_uncached
  File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/jax/_src/lib/xla_bridge.py", line 422 in get_backend
  File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/jax/_src/dispatch.py", line 428 in lower_xla_callable
  File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/jax/_src/profiler.py", line 313 in wrapper
  File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/jax/_src/dispatch.py", line 324 in _xla_callable_uncached
  File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/jax/_src/dispatch.py", line 195 in xla_primitive_callable
  File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/jax/_src/util.py", line 215 in cached
  File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/jax/_src/util.py", line 222 in wrapper
  File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/jax/_src/dispatch.py", line 111 in apply_primitive
  File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/jax/core.py", line 686 in process_primitive
  File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/jax/core.py", line 328 in bind_with_trace
  File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/jax/core.py", line 325 in bind
  File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/jax/_src/lax/lax.py", line 579 in _convert_element_type
  File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/jax/_src/numpy/lax_numpy.py", line 1902 in array
  File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/jax/_src/numpy/lax_numpy.py", line 1921 in asarray
  File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/jax/_src/prng.py", line 552 in random_seed
  File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/jax/_src/prng.py", line 262 in seed_with_impl
  File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/jax/_src/random.py", line 128 in PRNGKey
  File "/home/palisand/multinerf/train.py", line 45 in main
  File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/absl/app.py", line 254 in _run_main
  File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/absl/app.py", line 308 in run
  File "/home/palisand/multinerf/train.py", line 288 in <module>
  File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/runpy.py", line 87 in _run_code
  File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/runpy.py", line 197 in _run_module_as_main
Aborted (core dumped)
Palisand commented 1 year ago

Here is what I ran in the VM, up to the unit tests run:

sudo apt update
sudo apt install -y wget git

# miniconda
wget https://repo.anaconda.com/miniconda/Miniconda3-py39_4.12.0-Linux-x86_64.sh
bash Miniconda3-py39_4.12.0-Linux-x86_64.sh
source .bashrc

# COLMAP - https://colmap.github.io/install.html

sudo apt install -y \
    cmake \
    build-essential \
    libboost-program-options-dev \
    libboost-filesystem-dev \
    libboost-graph-dev \
    libboost-system-dev \
    libboost-test-dev \
    libeigen3-dev \
    libsuitesparse-dev \
    libfreeimage-dev \
    libmetis-dev \
    libgoogle-glog-dev \
    libgflags-dev \
    libglew-dev \
    qtbase5-dev \
    libqt5opengl5-dev \
    libcgal-dev

sudo apt install -y libatlas-base-dev libsuitesparse-dev
git clone https://ceres-solver.googlesource.com/ceres-solver
cd ceres-solver
git checkout $(git describe --tags) # Checkout the latest release
mkdir build
cd build
cmake .. -DBUILD_TESTING=OFF -DBUILD_EXAMPLES=OFF
make -j
sudo make install

cd ~
git clone https://github.com/colmap/colmap.git
cd colmap
git checkout dev
mkdir build
cd build
cmake ..
make -j
sudo make install
colmap -h

## Install & configure MultiNERF

cd ~
git clone https://github.com/google-research/multinerf.git
cd multinerf
conda create --name multinerf python=3.9
conda activate multinerf
conda install pip
pip install --upgrade pip
pip install -r requirements.txt
pip install tensorflow==2.9.1  # match TPU software version
git clone https://github.com/rmbrualla/pycolmap.git ./internal/pycolmap
./scripts/run_all_unit_tests.sh
jonbarron commented 1 year ago

How are you running this on a Google TPU? We train our models on Google TPUs but using the internal interface, which is different from the publicly available one. I don't think this code has yet been run through the external interface. Have you verified that you can run other models on the TPUs you're using? It seems like the issue here is at a lower level than this codebase here --- maybe a jax/cuda/driver issue?

Palisand commented 1 year ago

Ah, I see. I am using the publicly available interface, following google's Cloud TPU documentation. I haven't verified other models.

To create the TPU VM, I ran:

gcloud config set project multinerf
gcloud services enable tpu.googleapis.com
gcloud beta services identity create --service tpu.googleapis.com
gcloud alpha compute tpus tpu-vm create tpu-multinerf --zone us-central1-b --accelerator-type v3-8 --version tpu-vm-tf-2.9.1

I then SSHed into the VM:

gcloud alpha compute tpus tpu-vm ssh tpu-multinerf --zone us-central1-b

And ran the aforementioned commands.

Before using the TPU VM, I tested these commands locally, in a Docker container running Ubuntu 20.04 (just like the VM). The tests succeeded in the container.

Palisand commented 1 year ago

I tried again from scratch. This time, I removed jax, jaxlib, and tensorflow from requirements.txt and then I ran:

pip install "jax[tpu]>=0.2.16" -f https://storage.googleapis.com/jax-releases/libtpu_releases.html
pip install tensorflow==2.9.1
pip install -r requirements.txt

For jax: https://github.com/google/jax/#pip-installation-google-cloud-tpu

Some tests still fail, but at least they're not aborted. Here's some partial test output: ``` FAIL: test_construct_ray_warps_extents_log (tests.coord_test.CoordTest) tests.coord_test.CoordTest.test_construct_ray_warps_extents_log test_construct_ray_warps_extents_log(. at 0x7faae18e0e50>>) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/absl/testing/parameterized.py", line 314, in bound_param_test return test_method(self, *testcase_params) File "/home/palisand/multinerf/tests/coord_test.py", line 194, in test_construct_ray_warps_extents np.testing.assert_allclose( File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 1527, in assert_allclose assert_array_compare(compare, actual, desired, err_msg=str(err_msg), File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 844, in assert_array_compare raise AssertionError(msg) AssertionError: Not equal to tolerance rtol=1e-05, atol=1e-05 Mismatched elements: 43 / 100 (43%) Max absolute difference: 0.00045204 Max relative difference: 7.56597e-05 x: array([ 2.400275, 1.668342, 2.044059, 6.927439, 1.078879, 0.115673, 0.290508, 0.686725, 0.240134, 0.186716, 0.534207, 0.409191, 2.090983, 0.41522 , 0.722983, 1.309822, 0.97231 , 0.64675 ,... y: array([ 2.400219, 1.668289, 2.044053, 6.927345, 1.078887, 0.115672, 0.290508, 0.686718, 0.240134, 0.186729, 0.534207, 0.409187, 2.090986, 0.415209, 0.722943, 1.309913, 0.972313, 0.646793,... ====================================================================== FAIL: test_pos_enc_25_2 (tests.coord_test.CoordTest) tests.coord_test.CoordTest.test_pos_enc_25_2 test_pos_enc_25_2(25, 2) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/absl/testing/parameterized.py", line 314, in bound_param_test return test_method(self, *testcase_params) File "/home/palisand/multinerf/tests/coord_test.py", line 127, in test_pos_enc self.assertLess(max_err, tol) AssertionError: 2.3317099 not less than 2 ====================================================================== FAIL: test_pos_enc_30_2 (tests.coord_test.CoordTest) tests.coord_test.CoordTest.test_pos_enc_30_2 test_pos_enc_30_2(30, 2) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/absl/testing/parameterized.py", line 314, in bound_param_test return test_method(self, *testcase_params) File "/home/palisand/multinerf/tests/coord_test.py", line 127, in test_pos_enc self.assertLess(max_err, tol) AssertionError: 109575406000.0 not less than 2 ---------------------------------------------------------------------- Ran 21 tests in 30.823s FAILED (failures=3) .FFFF. ====================================================================== FAIL: test_mse_to_psnr_golden (tests.image_test.ImageTest) tests.image_test.ImageTest.test_mse_to_psnr_golden A lazy golden test for mse_to_psnr. ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/palisand/multinerf/tests/image_test.py", line 127, in test_mse_to_psnr_golden np.testing.assert_allclose(psnr, psnr_gt, atol=1E-5, rtol=1E-5) File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 1527, in assert_allclose assert_array_compare(compare, actual, desired, err_msg=str(err_msg), File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 844, in assert_array_compare raise AssertionError(msg) AssertionError: Not equal to tolerance rtol=1e-05, atol=1e-05 Mismatched elements: 28 / 64 (43.8%) Max absolute difference: 0.00061035 Max relative difference: 0.00023579 x: array([43.429222, 42.739685, 42.050312, 41.360874, 40.671413, 39.982204, 39.29292 , 38.603603, 37.91449 , 37.225014, 36.535473, 35.846165, 35.156944, 34.46737 , 33.777996, 33.088787, 32.399387, 31.709982,... y: array([43.429447, 42.74009 , 42.050735, 41.361378, 40.672024, 39.982666, 39.29331 , 38.603954, 37.914597, 37.22524 , 36.535885, 35.846527, 35.15717 , 34.46781 , 33.778458, 33.0891 , 32.399746, 31.710388,... ====================================================================== FAIL: test_psnr_mse_round_trip (tests.image_test.ImageTest) tests.image_test.ImageTest.test_psnr_mse_round_trip PSNR -> MSE -> PSNR is a no-op. ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/palisand/multinerf/tests/image_test.py", line 63, in test_psnr_mse_round_trip np.testing.assert_allclose( File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 1527, in assert_allclose assert_array_compare(compare, actual, desired, err_msg=str(err_msg), File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 844, in assert_array_compare raise AssertionError(msg) AssertionError: Not equal to tolerance rtol=1e-05, atol=1e-05 Mismatched elements: 1 / 1 (100%) Max absolute difference: 0.00024223 Max relative difference: 1.21116638e-05 x: array(20.000242, dtype=float32) y: array(20.) ====================================================================== FAIL: test_srgb_linearize (tests.image_test.ImageTest) tests.image_test.ImageTest.test_srgb_linearize ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/palisand/multinerf/tests/image_test.py", line 81, in test_srgb_linearize np.testing.assert_allclose( File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 1527, in assert_allclose assert_array_compare(compare, actual, desired, err_msg=str(err_msg), File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 844, in assert_array_compare raise AssertionError(msg) AssertionError: Not equal to tolerance rtol=1e-05, atol=1e-05 Mismatched elements: 3524 / 10000 (35.2%) Max absolute difference: 0.00025249 Max relative difference: 0.00018565 x: array([-1. , -0.9996 , -0.9992 , ..., 2.999342, 2.999746, 3.00015 ], dtype=float32) y: array([-1. , -0.9996, -0.9992, ..., 2.9992, 2.9996, 3. ], dtype=float32) ====================================================================== FAIL: test_srgb_to_linear_golden (tests.image_test.ImageTest) tests.image_test.ImageTest.test_srgb_to_linear_golden A lazy golden test for srgb_to_linear. ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/palisand/multinerf/tests/image_test.py", line 108, in test_srgb_to_linear_golden np.testing.assert_allclose(linear, linear_gt, atol=1E-5, rtol=1E-5) File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 1527, in assert_allclose assert_array_compare(compare, actual, desired, err_msg=str(err_msg), File "/home/palisand/miniconda3/envs/multinerf/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 844, in assert_array_compare raise AssertionError(msg) AssertionError: Not equal to tolerance rtol=1e-05, atol=1e-05 Mismatched elements: 19 / 64 (29.7%) Max absolute difference: 6.875396e-05 Max relative difference: 0.00015924 x: array([0. , 0.001229, 0.002457, 0.003725, 0.005261, 0.007113, 0.009299, 0.011834, 0.014733, 0.018009, 0.02167 , 0.025736, 0.030215, 0.035118, 0.040456, 0.04624 , 0.05248 , 0.059185,... y: array([0. , 0.001229, 0.002457, 0.003725, 0.005261, 0.007113, 0.0093 , 0.011835, 0.014732, 0.018007, 0.021671, 0.025736, 0.030215, 0.035118, 0.040456, 0.04624 , 0.052479, 0.059184,... ---------------------------------------------------------------------- Ran 6 tests in 9.887s FAILED (failures=4) ```