search

CDAT / cdat

Community Data Analysis Tools
Other
174 stars 68 forks source link

cdat81 uses only 1 cpu core #2238

Closed Xunius closed 5 years ago

Xunius commented 5 years ago

Hi all, I'm updating from cdat-lite to cdat81, installation is done by:

conda create -n cdat81 -c cdat/label/v81 -c conda-forge python=3.6 cdat

conda is using v4.6.11

After that I installed nothing else, but jumped into a simple test like this:

    N=1460
    import time
    t1=time.time()
    for i in range(N):

        print('i=',i)

        slab=np.random.random([1,400,500])
        slab=MV.array(slab)
        axis=cdms.createAxis([i,])
        axis.designateTime()
        axis.id='t'
        axis.units=''
        slab.setAxis(0, axis)
        if i==0:
            data=slab
        else:
            data=MV.concatenate((data, slab), axis=0)

    t2=time.time()
    print(t2-t1)

The system monitor displays high cpu usage only in 1 thread, at the process gets increasingly slower as iteration goes on.

While in old cdat-lite 6.0rc2, the same script uses all of my cores, and there is no notable slowing down.

Please help. Also, is there a python3 compatible cdat-lite? I'm not sure what's new fancy staff is added into cdat8 but it appears to me only it becoming slower.

Full conda list dump:

_libgcc_mutex 0.1 main alabaster 0.7.12 py_0 conda-forge asn1crypto 0.24.0 py36_1003 conda-forge astroid 2.2.5 py36_0 conda-forge attrs 19.1.0 py_0 conda-forge autopep8 1.4.4 py_0 conda-forge babel 2.7.0 py_0 conda-forge backcall 0.1.0 py_0 conda-forge basemap 1.2.0 py36hd759880_4 conda-forge blas 1.1 openblas conda-forge bleach 3.1.0 py_0 conda-forge bokeh 1.2.0 py36_0 conda-forge bzip2 1.0.8 h516909a_0 conda-forge ca-certificates 2019.6.16 hecc5488_0 conda-forge cdat 8.1 1 cdat/label/v81 cdat_info 8.1.1 py_0 cdat/label/v81 cdms2 3.0.0 pypi_0 pypi cdp 1.4.2 py_1 cdat/label/v81 cdtime 3.1.2 py36h6091dcd_1 conda-forge cdutil 8.1 py_1 cdat/label/v81 certifi 2019.6.16 py36_1 conda-forge cffi 1.12.3 py36h8022711_0 conda-forge chardet 3.0.4 py36_1003 conda-forge cibots 0.2 py_0 cdat/label/v81 click 7.0 py_0 conda-forge cloudpickle 1.2.1 py_0 conda-forge cryptography 2.7 py36h72c5cf5_0 conda-forge curl 7.65.3 hf8cf82a_0 conda-forge cycler 0.10.0 py_1 conda-forge cytoolz 0.10.0 py36h516909a_0 conda-forge dask 2.1.0 py_0 conda-forge dask-core 2.1.0 py_0 conda-forge dbus 1.13.6 he372182_0 conda-forge decorator 4.4.0 py_0 conda-forge defusedxml 0.5.0 py_1 conda-forge distarray 2.12.2 py_1 conda-forge distributed 2.1.0 py_0 conda-forge docutils 0.15 py36_0 conda-forge dv3d 8.1 py_0 cdat/label/v81 entrypoints 0.3 py36_1000 conda-forge eofs 1.4.0 py_0 conda-forge esmf 7.1.0 h9eb252b_1005 conda-forge esmpy 7.1.0 py36h24bf2e0_3 conda-forge expat 2.2.5 he1b5a44_1003 conda-forge ffmpeg 4.0.2 ha0c5888_2 conda-forge flake8 3.7.7 py36_0 conda-forge fontconfig 2.13.1 he4413a7_1000 conda-forge freetype 2.10.0 he983fc9_0 conda-forge future 0.17.1 py36_1000 conda-forge g2clib 1.6.0 hf3f1b0b_9 conda-forge genutil 8.1.1 py36h3010b51_0 conda-forge geos 3.7.2 he1b5a44_1 conda-forge gettext 0.19.8.1 hc5be6a0_1002 conda-forge ghostscript 9.22 hf484d3e_1001 conda-forge glib 2.58.3 h6f030ca_1002 conda-forge gmp 6.1.2 hf484d3e_1000 conda-forge gnutls 3.5.19 h2a4e5f8_1 conda-forge gst-plugins-base 1.14.5 h0935bb2_0 conda-forge gstreamer 1.14.5 h36ae1b5_0 conda-forge hdf4 4.2.13 h9a582f1_1002 conda-forge hdf5 1.10.4 nompi_h3c11f04_1106 conda-forge heapdict 1.0.0 py36_1000 conda-forge icu 58.2 hf484d3e_1000 conda-forge idna 2.8 py36_1000 conda-forge imagesize 1.1.0 py_0 conda-forge ipykernel 5.1.1 py36h5ca1d4c_0 conda-forge ipython 7.6.1 py36h5ca1d4c_0 conda-forge ipython_genutils 0.2.0 py_1 conda-forge ipywidgets 7.5.0 py_0 conda-forge isort 4.3.21 py36_0 conda-forge jasper 1.900.1 h07fcdf6_1006 conda-forge jedi 0.14.1 py36_0 conda-forge jeepney 0.4 py_0 conda-forge jinja2 2.10.1 py_0 conda-forge jpeg 9c h14c3975_1001 conda-forge jsonschema 3.0.1 py36_0 conda-forge jupyter 1.0.0 py_2 conda-forge jupyter_client 5.3.1 py_0 conda-forge jupyter_console 6.0.0 py_0 conda-forge jupyter_core 4.4.0 py_0 conda-forge keyring 19.0.2 py36_0 conda-forge kiwisolver 1.1.0 py36hc9558a2_0 conda-forge krb5 1.16.3 h05b26f9_1001 conda-forge lazy-object-proxy 1.4.1 py36h516909a_0 conda-forge libcdms 3.1.2 hcbdc9ef_1000 cdat/label/v81 libcf 1.0.2 py36h6d95edf_1007 conda-forge libcurl 7.65.3 hda55be3_0 conda-forge libdrs 3.1.2 h8c6403c_2 conda-forge libdrs_f 3.1.2 h0ebed7e_2 conda-forge libedit 3.1.20170329 hf8c457e_1001 conda-forge libffi 3.2.1 he1b5a44_1006 conda-forge libgcc 7.2.0 h69d50b8_2 conda-forge libgcc-ng 9.1.0 hdf63c60_0 libgfortran-ng 7.3.0 hdf63c60_0 libiconv 1.15 h516909a_1005 conda-forge libnetcdf 4.6.2 hbdf4f91_1001 conda-forge libpng 1.6.37 hed695b0_0 conda-forge libsodium 1.0.17 h516909a_0 conda-forge libssh2 1.8.2 h22169c7_2 conda-forge libstdcxx-ng 9.1.0 hdf63c60_0 libtiff 4.0.10 h57b8799_1003 conda-forge libuuid 2.32.1 h14c3975_1000 conda-forge libxcb 1.13 h14c3975_1002 conda-forge libxml2 2.9.9 h13577e0_2 conda-forge locket 0.2.0 py_2 conda-forge lz4-c 1.8.3 he1b5a44_1001 conda-forge markupsafe 1.1.1 py36h14c3975_0 conda-forge matplotlib 3.1.1 py36_0 conda-forge matplotlib-base 3.1.1 py36hfd891ef_0 conda-forge mccabe 0.6.1 py_1 conda-forge mistune 0.8.4 py36h14c3975_1000 conda-forge mpi 1.0 mpich conda-forge mpich 3.2.1 hc99cbb1_1013 conda-forge msgpack-python 0.6.1 py36h6bb024c_0 conda-forge nb_conda 2.2.1 py36_2 conda-forge nb_conda_kernels 2.2.2 py36_0 conda-forge nbconvert 5.5.0 py_0 conda-forge nbformat 4.4.0 py_1 conda-forge ncurses 6.1 hf484d3e_1002 conda-forge netcdf-fortran 4.4.5 hea25ff8_1000 conda-forge nettle 3.3 0 conda-forge notebook 6.0.0 py36_0 conda-forge numpy 1.16.2 py36_blas_openblash1522bff_0 [blas_openblas] conda-forge numpydoc 0.9.1 py_0 conda-forge olefile 0.46 py_0 conda-forge openblas 0.3.3 h9ac9557_1001 conda-forge openh264 1.8.0 hdbcaa40_1000 conda-forge openssl 1.1.1c h516909a_0 conda-forge output_viewer 1.2.5 py36_0 cdat/label/v81 packaging 19.0 py_0 conda-forge pandas 0.25.0 py36hb3f55d8_0 conda-forge pandoc 2.7.3 0 conda-forge pandocfilters 1.4.2 py_1 conda-forge parso 0.5.1 py_0 conda-forge partd 1.0.0 py_0 conda-forge pcre 8.41 hf484d3e_1003 conda-forge pexpect 4.7.0 py36_0 conda-forge pickleshare 0.7.5 py36_1000 conda-forge pillow 6.0.0 py36he7afcd5_0 conda-forge pip 19.1.1 py36_0 conda-forge proj4 5.2.0 he1b5a44_1004 conda-forge prometheus_client 0.7.1 py_0 conda-forge prompt_toolkit 2.0.9 py_0 conda-forge psutil 5.6.3 py36h516909a_0 conda-forge pthread-stubs 0.4 h14c3975_1001 conda-forge ptyprocess 0.6.0 py_1001 conda-forge pycodestyle 2.5.0 py_0 conda-forge pycparser 2.19 py36_1 conda-forge pyflakes 2.1.1 py_0 conda-forge pygments 2.4.2 py_0 conda-forge pylint 2.3.1 py36_0 conda-forge pyopenssl 19.0.0 py36_0 conda-forge pyparsing 2.4.1 py_0 conda-forge pyproj 1.9.6 py36h516909a_1002 conda-forge pyqt 5.9.2 py36hcca6a23_0 conda-forge pyrsistent 0.15.3 py36h516909a_0 conda-forge pyshp 2.1.0 py_0 conda-forge pysocks 1.7.0 py36_0 conda-forge pyspharm 1.0.9 py36h6091dcd_1003 conda-forge python 3.6.7 h357f687_1005 conda-forge python-dateutil 2.8.0 py_0 conda-forge pytz 2019.1 py_0 conda-forge pyyaml 5.1.1 py36h516909a_0 conda-forge pyzmq 18.0.2 py36h1768529_2 conda-forge qt 5.9.7 h52cfd70_2 conda-forge qtawesome 0.5.7 pyh8a2030e_0 conda-forge qtconsole 4.5.1 py_0 conda-forge qtpy 1.8.0 py_0 conda-forge readline 8.0 hf8c457e_0 conda-forge regrid2 3.0.0 pypi_0 pypi requests 2.22.0 py36_1 conda-forge rope 0.14.0 py_0 conda-forge scipy 1.2.1 py36_blas_openblash1522bff_0 [blas_openblas] conda-forge secretstorage 3.1.1 py36_0 conda-forge send2trash 1.5.0 py_0 conda-forge setuptools 41.0.1 py36_0 conda-forge sip 4.19.8 py36hf484d3e_1000 conda-forge six 1.12.0 py36_1000 conda-forge snowballstemmer 1.9.0 py_0 conda-forge sortedcontainers 2.1.0 py_0 conda-forge sphinx 2.1.2 py_0 conda-forge sphinxcontrib-applehelp 1.0.1 py_0 conda-forge sphinxcontrib-devhelp 1.0.1 py_0 conda-forge sphinxcontrib-htmlhelp 1.0.2 py_0 conda-forge sphinxcontrib-jsmath 1.0.1 py_0 conda-forge sphinxcontrib-qthelp 1.0.2 py_0 conda-forge sphinxcontrib-serializinghtml 1.1.1 py_0 conda-forge spyder 3.3.6 py36_0 conda-forge spyder-kernels 0.5.1 py36_0 conda-forge sqlite 3.29.0 hcee41ef_0 conda-forge tblib 1.4.0 py_0 conda-forge terminado 0.8.2 py36_0 conda-forge testpath 0.4.2 py_1001 conda-forge tk 8.6.9 hed695b0_1002 conda-forge toolchain3 1.1.1 0 conda-forge toolz 0.10.0 py_0 conda-forge tornado 6.0.3 py36h516909a_0 conda-forge traitlets 4.3.2 py36_1000 conda-forge typed-ast 1.4.0 py36h516909a_0 conda-forge udunits2 2.2.27.6 h4e0c4b3_1001 conda-forge urllib3 1.25.3 py36_0 conda-forge vcs 8.1 py_0 cdat/label/v81 vcsaddons 8.1 py36h14c3975_0 cdat/label/v81 vtk-cdat 8.2.0rc2.8.1 py36h413fb2d_0 cdat/label/v81 wcwidth 0.1.7 py_1 conda-forge webencodings 0.5.1 py_1 conda-forge wheel 0.33.4 py36_0 conda-forge widgetsnbextension 3.5.0 py36_0 conda-forge windspharm 1.7.0 py36_1000 conda-forge wk 8.1 py_0 cdat/label/v81 wrapt 1.11.2 py36h516909a_0 conda-forge wurlitzer 1.0.3 py36_0 conda-forge x264 1!152.20180806 h14c3975_0 conda-forge xorg-libxau 1.0.9 h14c3975_0 conda-forge xorg-libxdmcp 1.1.3 h516909a_0 conda-forge xz 5.2.4 h14c3975_1001 conda-forge yaml 0.1.7 h14c3975_1001 conda-forge zeromq 4.3.2 he1b5a44_2 conda-forge zict 1.0.0 py_0 conda-forge zlib 1.2.11 h516909a_1005 conda-forge zstd 1.4.0 h3b9ef0a_0 conda-forge
Xunius commented 5 years ago

Does it have something to do with mkl? If I create an empty env, install numpy first using conda install numpy, then install conda install -c conda-forge cdms2, I got:

The following packages will be UPDATED:

  blas                              pkgs/main::blas-1.0-mkl --> conda-forge::blas-1.1-openblas
  ca-certificates    pkgs/main::ca-certificates-2019.5.15-0 --> conda-forge::ca-certificates-2019.6.16-hecc5488_0
  certifi               pkgs/main::certifi-2019.6.16-py37_0 --> conda-forge::certifi-2019.6.16-py36_1
  mkl_fft            pkgs/main::mkl_fft-1.0.12-py37ha843d7~ --> conda-forge::mkl_fft-1.0.13-py36h516909a_1
  mkl_random         pkgs/main::mkl_random-1.0.2-py37hd81d~ --> conda-forge::mkl_random-1.0.4-py36hf2d7682_0

The following packages will be SUPERSEDED by a higher-priority channel:

  numpy              pkgs/main::numpy-1.16.4-py37h7e9f1db_0 --> conda-forge::numpy-1.16.2-py36_blas_openblash1522bff_0
  openssl              pkgs/main::openssl-1.1.1c-h7b6447c_1 --> conda-forge::openssl-1.1.1c-h516909a_0
  pip                          pkgs/main::pip-19.1.1-py37_0 --> conda-forge::pip-19.1.1-py36_0
  setuptools            pkgs/main::setuptools-41.0.1-py37_0 --> conda-forge::setuptools-41.0.1-py36_0
  wheel                      pkgs/main::wheel-0.33.4-py37_0 --> conda-forge::wheel-0.33.4-py36_0

The following packages will be DOWNGRADED:

  numpy-base                          1.16.4-py37hde5b4d6_0 --> 1.14.3-py36h2b20989_0
  python                                   3.7.3-h0371630_0 --> 3.6.8-h0371630_0
doutriaux1 commented 5 years ago

@Xunius you can use dask with cdms (from nightly builds) to use all core. Since you're creating more axes and data with each step I'm not surprised your system gets slower as it goes. @dnadeau4 can you point @Xunius to some dask examples? Thanks.

dnadeau4 commented 5 years ago

@xunius the downgraded package is due from going from the main channel from anaconda to conda-forge channel. conda-forge is usually not as up-to-date as anaconda.

I have a test for using Dask here: https://github.com/CDAT/cdms/blob/master/tests/test_serialize.py

I have run/demo it in pangeo in the past.
https://binder.pangeo.io/v2/gh/cdat/dask-cdms/master

the code can be found here: https://github.com/CDAT/dask-cdms

Xunius commented 5 years ago

Many thanks for the info. I think I managed to get it work by pinning numpy, numpy-base and blas in the env, and copying over a few lib*.so files. I'm surprised that cdms2 even imports. I've never used dask before, will look into that, thanks for the links @dnadeau4