search

inaos / iron-array

2 stars 0 forks source link

Reproducible crash on python test suite #576

Closed FrancescAlted closed 2 years ago

FrancescAlted commented 2 years ago

After creating a fresh conda environment in my Mac with:

conda create --name py310 python=3.10
conda activate py310
conda install -c numba llvmlite llvmdev
conda install -c intel mkl-static icc_rt
pip install -r requirements-build.txt
pip install -r requirements-runtime.txt
pip install -r requirements-test.txt

I can reproduce a crash with:

francesc@Francescs-MacBook-Air ~/i/iron-array-python (develop) [1]> env PYTHONPATH=. python -m pytest -m "not heavy" -v         (py310)
========================================================= test session starts ==========================================================
platform darwin -- Python 3.10.4, pytest-7.1.2, pluggy-1.0.0 -- /Users/faltet/miniconda3/envs/py310/bin/python
cachedir: .pytest_cache
rootdir: /Users/faltet/iarray/iron-array-python, configfile: pytest.ini
collected 1596 items / 378 deselected / 1218 selected

iarray/tests/test_append.py::test_append[shape0-chunks0-blocks0-data_shape0-1-True-None-float32-None] PASSED                     [  0%]
iarray/tests/test_append.py::test_append[shape0-chunks0-blocks0-data_shape0-1-True-None-uint64->M8[M]] PASSED                    [  0%]
iarray/tests/test_append.py::test_append[shape0-chunks0-blocks0-data_shape0-1-True-None-int64-<m8[ps]] PASSED                    [  0%]
iarray/tests/test_append.py::test_append[shape0-chunks0-blocks0-data_shape0-1-True-None-int32-None] PASSED                       [  0%]
iarray/tests/test_append.py::test_append[shape0-chunks0-blocks0-data_shape0-1-True-None-uint64->i8] PASSED                       [  0%]
iarray/tests/test_append.py::test_append[shape0-chunks0-blocks0-data_shape0-1-True-None-uint32-u8] PASSED                        [  0%]
iarray/tests/test_append.py::test_append[shape1-chunks1-blocks1-data_shape1-0-False-None-float32-None] PASSED                    [  0%]
iarray/tests/test_append.py::test_append[shape1-chunks1-blocks1-data_shape1-0-False-None-uint64->M8[M]] PASSED                   [  0%]
iarray/tests/test_append.py::test_append[shape1-chunks1-blocks1-data_shape1-0-False-None-int64-<m8[ps]] PASSED                   [  0%]
iarray/tests/test_append.py::test_append[shape1-chunks1-blocks1-data_shape1-0-False-None-int32-None] PASSED                      [  0%]
iarray/tests/test_append.py::test_append[shape1-chunks1-blocks1-data_shape1-0-False-None-uint64->i8] PASSED                      [  0%]
iarray/tests/test_append.py::test_append[shape1-chunks1-blocks1-data_shape1-0-False-None-uint32-u8] PASSED                       [  0%]
iarray/tests/test_append.py::test_append[shape3-chunks3-blocks3-data_shape3-0-False-test_append_asparse.iarr-float32-None] PASSED [  1%]
iarray/tests/test_append.py::test_append[shape3-chunks3-blocks3-data_shape3-0-False-test_append_asparse.iarr-uint64->M8[M]] PASSED [  1%]
iarray/tests/test_append.py::test_append[shape3-chunks3-blocks3-data_shape3-0-False-test_append_asparse.iarr-int64-<m8[ps]] PASSED [  1%]
iarray/tests/test_append.py::test_append[shape3-chunks3-blocks3-data_shape3-0-False-test_append_asparse.iarr-int32-None] PASSED  [  1%]
iarray/tests/test_append.py::test_append[shape3-chunks3-blocks3-data_shape3-0-False-test_append_asparse.iarr-uint64->i8] PASSED  [  1%]
iarray/tests/test_append.py::test_append[shape3-chunks3-blocks3-data_shape3-0-False-test_append_asparse.iarr-uint32-u8] PASSED   [  1%]
iarray/tests/test_attrs.py::test_attrs[shape0-chunks0-blocks0-test_attr00.iarr-float64-True] PASSED                              [  1%]
iarray/tests/test_attrs.py::test_attrs[shape0-chunks0-blocks0-test_attr00.iarr-float64-False] PASSED                             [  1%]
iarray/tests/test_attrs.py::test_attrs[shape1-chunks1-blocks1-test_attr01.iarr-int16-True] PASSED                                [  1%]
iarray/tests/test_attrs.py::test_attrs[shape1-chunks1-blocks1-test_attr01.iarr-int16-False] PASSED                               [  1%]
iarray/tests/test_attrs.py::test_attrs[shape2-chunks2-blocks2-None-float32-True] PASSED                                          [  1%]
iarray/tests/test_attrs.py::test_attrs[shape2-chunks2-blocks2-None-float32-False] PASSED                                         [  1%]
iarray/tests/test_btune.py::test_btune[0-1000-shape0-chunks0-blocks0-float64-False-None] PASSED                                  [  2%]
iarray/tests/test_btune.py::test_btune[-1--2000-shape4-chunks4-blocks4-float32-True-None] PASSED                                 [  2%]
iarray/tests/test_config_params.py::test_global_config[0-Codec.ZSTD-filters0-None-None-False-None] PASSED                        [  2%]
iarray/tests/test_config_params.py::test_global_config[1-Codec.BLOSCLZ-filters1-chunks1-blocks1-True-None] PASSED                [  2%]
iarray/tests/test_config_params.py::test_global_config[9-Codec.ZSTD-filters2-chunks2-blocks2-False-test_config_params_sparse.iarr] PASSED [  2%]
iarray/tests/test_config_params.py::test_global_config[6-Codec.ZSTD-filters3-chunks3-blocks3-True-test_config_params_contiguous.iarr] PASSED [  2%]
iarray/tests/test_config_params.py::test_global_config[0-Codec.ZSTD-filters4-None-None-False-None] PASSED                        [  2%]
iarray/tests/test_config_params.py::test_global_favor[Favor.BALANCE-None-None] PASSED                                            [  2%]
iarray/tests/test_config_params.py::test_global_favor[Favor.SPEED-chunks1-blocks1] PASSED                                        [  2%]
iarray/tests/test_config_params.py::test_global_favor[Favor.CRATIO-chunks2-blocks2] PASSED                                       [  2%]
iarray/tests/test_config_params.py::test_favor_nobtune[Favor.BALANCE] PASSED                                                     [  2%]
iarray/tests/test_config_params.py::test_favor_nobtune[Favor.SPEED] PASSED                                                       [  2%]
iarray/tests/test_config_params.py::test_favor_nobtune[Favor.CRATIO] PASSED                                                      [  3%]
iarray/tests/test_config_params.py::test_btune_incompat[1-None-None] PASSED                                                      [  3%]
iarray/tests/test_config_params.py::test_btune_incompat[None-Codec.ZSTD-None] PASSED                                             [  3%]
iarray/tests/test_config_params.py::test_btune_incompat[None-None-filters2] PASSED                                               [  3%]
iarray/tests/test_config_params.py::test_btune_incompat[1-Codec.ZSTD-filters3] PASSED                                            [  3%]
iarray/tests/test_config_params.py::test_np_dtype[f8-float64] PASSED                                                             [  3%]
iarray/tests/test_config_params.py::test_np_dtype[f4-float64] PASSED                                                             [  3%]
iarray/tests/test_config_params.py::test_np_dtype[datetime64[Y]-int32] PASSED                                                    [  3%]
iarray/tests/test_config_params.py::test_np_dtype[datetime64[D]-uint32] PASSED                                                   [  3%]
iarray/tests/test_config_params.py::test_np_dtype[timedelta64[ps]-int64] PASSED                                                  [  3%]
iarray/tests/test_config_params.py::test_np_dtype[timedelta64[as]-uint64] PASSED                                                 [  3%]
iarray/tests/test_config_params.py::test_np_dtype[i8-int16] PASSED                                                               [  3%]
iarray/tests/test_config_params.py::test_np_dtype[ui2-bool_] PASSED                                                              [  4%]
iarray/tests/test_config_params.py::test_global_config_dtype[None-None-shape0] PASSED                                            [  4%]
iarray/tests/test_config_params.py::test_global_config_dtype[chunks1-blocks1-shape1] PASSED                                      [  4%]
iarray/tests/test_config_params.py::test_global_config_dtype[chunks2-blocks2-shape2] PASSED                                      [  4%]
iarray/tests/test_config_params.py::test_global_config_dtype[chunks3-blocks3-shape3] PASSED                                      [  4%]
iarray/tests/test_config_params.py::test_global_config_dtype[None-None-shape4] PASSED                                            [  4%]
iarray/tests/test_config_params.py::test_config_ctx[0-Codec.ZSTD-filters0-None-None] PASSED                                      [  4%]
iarray/tests/test_config_params.py::test_config_ctx[1-Codec.BLOSCLZ-filters1-chunks1-blocks1] PASSED                             [  4%]
iarray/tests/test_config_params.py::test_config_ctx[9-Codec.ZSTD-filters2-chunks2-blocks2] PASSED                                [  4%]
iarray/tests/test_config_params.py::test_config_ctx[6-Codec.ZSTD-filters3-chunks3-blocks3] PASSED                                [  4%]
iarray/tests/test_config_params.py::test_config_ctx[0-Codec.ZSTD-filters4-None-None] PASSED                                      [  4%]
iarray/tests/test_config_params.py::test_config_ctx_dtype[None-None-shape0] PASSED                                               [  4%]
iarray/tests/test_config_params.py::test_config_ctx_dtype[chunks1-blocks1-shape1] PASSED                                         [  5%]
iarray/tests/test_config_params.py::test_config_ctx_dtype[chunks2-blocks2-shape2] PASSED                                         [  5%]
iarray/tests/test_config_params.py::test_config_ctx_dtype[chunks3-blocks3-shape3] PASSED                                         [  5%]
iarray/tests/test_config_params.py::test_config_ctx_dtype[None-None-shape4] PASSED                                               [  5%]
iarray/tests/test_config_params.py::test_nested_contexts PASSED                                                                  [  5%]
iarray/tests/test_config_params.py::test_default_params PASSED                                                                   [  5%]
iarray/tests/test_config_params.py::test_zfp_accuracy_codec ⏎
/Users/faltet/miniconda3/envs/py310/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

Apparently it always crashes when executing the iarray/tests/test_config_params.py::test_zfp_accuracy_codec test.

FrancescAlted commented 2 years ago

For some reason, it turned out that when 2 views are chained, the current implementation seems to send the same context to parallel Blosc2 core, creating race conditions.

In https://github.com/inaos/iron-array/commit/09759ae1ebd60a2deae6789a12faae28b5dc1534 we are forcing the use of 1 thread, and we fixed this issue. One pending task is to assess how much performance are we loosing by disabling multithreading on views.