search

slaclab / anarel-manage

Manage the conda based analysis release environments that include psana at LCLS.
Other
3 stars 3 forks source link

matplotlib 2.0.0 #18

Open davidslac opened 7 years ago

davidslac commented 7 years ago

Wondering why we didn't pickup the latest matplotlib with ana-1.2.0, doing a dryrun of installing it shows the changes we would get:

Package plan for installation in environment /reg/g/psdm/sw/conda/inst/miniconda2-dev-rhel7/envs/ana-1.2.0:

The following NEW packages will be INSTALLED:

    jasper:       1.900.1-3         conda-forge
    olefile:      0.44-py27_0       defaults   
    subprocess32: 3.2.7-py27_0      defaults   

The following packages will be UPDATED:

    freetype:     2.5.5-2           defaults    --> 2.6.3-1            conda-forge
    icu:          54.1-0            defaults    --> 56.1-4             conda-forge
    jpeg:         8d-2              defaults    --> 9b-0               defaults   
    libpng:       1.6.27-0          defaults    --> 1.6.28-0           conda-forge
    libtiff:      4.0.6-2           defaults    --> 4.0.6-3            defaults   
    matplotlib:   1.5.1-np111py27_0 defaults    --> 2.0.0-np111py27_1  conda-forge
    pillow:       3.4.2-py27_0      defaults    --> 4.0.0-py27_1       conda-forge

The following packages will be DOWNGRADED due to dependency conflicts:

    cairo:        1.14.8-0          defaults    --> 1.14.6-0           conda-forge
    fontconfig:   2.12.1-2          defaults    --> 2.11.1-6           conda-forge
    harfbuzz:     0.9.39-2          defaults    --> 0.9.39-1           defaults   
    opencv:       3.1.0-np111py27_1 defaults    --> 2.4.12-np111py27_2 conda-forge
    pango:        1.40.3-1          defaults    --> 1.40.1-0           conda-forge

looks like it is not in defaults, and would downgrade opencv

davidslac commented 7 years ago

@weninc and I figured out that what is pinning us to matplotlib 1.5.1 is that psana is pinned to Qt 4.8.7, and we get it from defaults. For some reason the Qt 4 dependencies from the defaults channel are not compatible with the matplotlib 2.0.0 dependencies - on the defaults channel. However if we get Qt 4 and matplotlib from the conda-forge channel, we do get matplotlib 2.

Until we drop support for rhel5, we should limit the packages we get from conda-forge. Per: https://github.com/conda-forge/numpy-feedstock/issues/20 I don't think Qt from conda-forge will run on rhel5.

We could build and maintain a matplotlib 2.0.0 package ourselves.

davidslac commented 7 years ago

We could use a different .condarc for rhel5 vs. rhel6/7, on rhel6/7, we could make conda-forge the higher priority channel, but on rhel5, we'll use defaults.

davidslac commented 7 years ago

This is not working, building psana fails - when we prioritize on conda-forge, this is some output:

Installing script: "arch/x86_64-rhel7-gcc48-opt/bin/psddlc"
Installing script: "arch/x86_64-rhel7-gcc48-opt/bin/med"
Installing script: "arch/x86_64-rhel7-gcc48-opt/bin/tcdpe"
Installing script: "arch/x86_64-rhel7-gcc48-opt/bin/tcmt"
Installing script: "arch/x86_64-rhel7-gcc48-opt/bin/test_img_gen"
Installing script: "arch/x86_64-rhel7-gcc48-opt/bin/codegen"
Installing script: "arch/x86_64-rhel7-gcc48-opt/bin/det_ndarr_average"
Installing script: "arch/x86_64-rhel7-gcc48-opt/bin/det_ndarr_raw_proc"
Installing script: "arch/x86_64-rhel7-gcc48-opt/bin/event_keys"
Installing script: "arch/x86_64-rhel7-gcc48-opt/bin/find_runs_for_data_type"
Installing script: "arch/x86_64-rhel7-gcc48-opt/bin/ddl_python"
/usr/bin/ld: warning: libicuuc.so.54, needed by arch/x86_64-rhel7-gcc48-opt/lib/libboost_regex.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libicui18n.so.54, needed by arch/x86_64-rhel7-gcc48-opt/lib/libboost_regex.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libicudata.so.54, needed by arch/x86_64-rhel7-gcc48-opt/lib/libboost_regex.so, not found (try using -rpath or -rpath-link)
arch/x86_64-rhel7-gcc48-opt/lib/libboost_regex.so: undefined reference to `u_tolower_54'
arch/x86_64-rhel7-gcc48-opt/lib/libboost_regex.so: undefined reference to `u_isspace_54'
arch/x86_64-rhel7-gcc48-opt/lib/libboost_regex.so: undefined reference to `u_isblank_54'
arch/x86_64-rhel7-gcc48-opt/lib/libboost_regex.so: undefined reference to `u_digit_54'
arch/x86_64-rhel7-gcc48-opt/lib/libboost_regex.so: undefined reference to `icu_54::Locale::Locale(icu_54::Locale const&)'
arch/x86_64-rhel7-gcc48-opt/lib/libboost_regex.so: undefined reference to `icu_54::Collator::createInstance(icu_54::Locale const&, UErrorCode&)'
arch/x86_64-rhel7-gcc48-opt/lib/libboost_regex.so: undefined reference to `icu_54::Locale::Locale()'
arch/x86_64-rhel7-gcc48-opt/lib/libboost_regex.so: undefined reference to `u_charType_54'
arch/x86_64-rhel7-gcc48-opt/lib/libboost_regex.so: undefined reference to `icu_54::Locale::~Locale()'
arch/x86_64-rhel7-gcc48-opt/lib/libboost_regex.so: undefined reference to `u_charFromName_54'
g++ -o build/x86_64-rhel7-gcc48-opt/RdbMySQL/app/rdb-mysql-dump-table.o -c -Wno-invalid-offsetof -Wno-unused-local-typedefs -O3 -Wall -Ibuild/x86_64-rhel7-gcc48-opt/RdbMySQL -IRdbMySQL -Iarch/x86_64-rhel7-gcc48-opt/geninc -Iinclude -I/reg/g/psdm/sw/conda/inst/miniconda2-dev-rhel7/conda-bld/psana-conda_1490804988431/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/include RdbMySQL/app/rdb-mysql-dump-table.cpp
collect2: error: ld returned 1 exit status
scons: *** [build/x86_64-rhel7-gcc48-opt/psana_test/xtclinedump] Error 1
scons: building terminated because of errors.
BUILD START: psana-conda-1.3.0-py27_1
    (actual version deferred until further download or env creation)

The following NEW packages will be INSTALLED:

    backports_abc:      0.5-py27_0                         conda-forge
    blas:               1.1-openblas                       conda-forge
    boost:              1.57.0-4                           defaults   
    ca-certificates:    2017.1.23-0                        conda-forge
    certifi:            2017.1.23-py27_0                   conda-forge
    cycler:             0.10.0-py27_0                      conda-forge
    cython:             0.25.2-py27_1                      conda-forge
    freetype:           2.7-1                              conda-forge
    functools32:        3.2.3.2-py27_1                     conda-forge
    h5py:               2.7.0rc2-py27_3                    local      
    hdf5:               1.8.18-openmpi_3                   local      
    icu:                58.1-1                             conda-forge
    libgfortran:        3.0.0-1                            defaults   
    libpng:             1.6.28-0                           conda-forge
    libsodium:          1.0.10-0                           conda-forge
    matplotlib:         2.0.0-np112py27_2                  conda-forge
    mpi4py:             2.0.0-py27_openmpi_104             local      
    mysql:              5.5.24-0                           defaults   
    ncurses:            5.9-10                             conda-forge
    ndarray:            1.1.8-0                            local      
    numexpr:            2.6.2-np112py27_0                  conda-forge
    numpy:              1.12.1-py27_blas_openblas_200      conda-forge [blas_openblas]
    openblas:           0.2.19-1                           conda-forge
    openmpi:            1.10.6-lsf_verbs_1                 local      
    openssl:            1.0.2h-3                           conda-forge
    pip:                9.0.1-py27_0                       conda-forge
    pyparsing:          2.2.0-py27_0                       conda-forge
    pyqt:               4.11.4-py27_2                      conda-forge
    python:             2.7.12-2                           conda-forge
    python-dateutil:    2.6.0-py27_0                       conda-forge
    pytz:               2016.10-py27_0                     conda-forge
    pyzmq:              16.0.2-py27_1                      conda-forge
    qt:                 4.8.5-0                            defaults   
    readline:           6.2-0                              conda-forge
    scipy:              0.19.0-np112py27_blas_openblas_200 conda-forge [blas_openblas]
    scons:              2.5.1-py27_0                       conda-forge
    setuptools:         33.1.1-py27_0                      conda-forge
    singledispatch:     3.4.0.3-py27_0                     conda-forge
    sip:                4.18-py27_1                        conda-forge
    six:                1.10.0-py27_1                      conda-forge
    sqlite:             3.13.0-1                           conda-forge
    ssl_match_hostname: 3.5.0.1-py27_1                     conda-forge
    subprocess32:       3.2.7-py27_0                       conda-forge
    szip:               2.1-102                            local      
    tables:             3.3.0-py27_103                     local      
    tk:                 8.5.19-1                           conda-forge
    tornado:            4.4.2-py27_0                       conda-forge
    wheel:              0.29.0-py27_0                      conda-forge
    zeromq:             4.2.1-1                            conda-forge
    zlib:               1.2.11-0                           conda-forge

Source cache directory is: /reg/g/psdm/sw/conda/inst/miniconda2-dev-rhel7/conda-bld/src_cache
Found source in cache: /reg/g/psdm/sw/conda/downloads/anarel/psana-conda-1.3.0.tar.gz
BUILD START (revised): psana-conda-1.3.0-py27_1
Package: psana-conda-1.3.0-py27_1
source tree in: /reg/g/psdm/sw/conda/inst/miniconda2-dev-rhel7/conda-bld/psana-conda_1490804988431/work/psana-conda-1.3.0
Traceback (most recent call last):
  File "/reg/g/psdm/sw/conda/inst/miniconda2-dev-rhel7/bin/conda-build", line 6, in <module>
    sys.exit(conda_build.cli.main_build.main())
  File "/reg/g/psdm/sw/conda/inst/miniconda2-dev-rhel7/lib/python2.7/site-packages/conda_build/cli/main_build.py", line 334, in main
    execute(sys.argv[1:])
  File "/reg/g/psdm/sw/conda/inst/miniconda2-dev-rhel7/lib/python2.7/site-packages/conda_build/cli/main_build.py", line 325, in execute
    noverify=args.no_verify)
  File "/reg/g/psdm/sw/conda/inst/miniconda2-dev-rhel7/lib/python2.7/site-packages/conda_build/api.py", line 97, in build
    need_source_download=need_source_download, config=config)
  File "/reg/g/psdm/sw/conda/inst/miniconda2-dev-rhel7/lib/python2.7/site-packages/conda_build/build.py", line 1502, in build_tree
    config=config)
  File "/reg/g/psdm/sw/conda/inst/miniconda2-dev-rhel7/lib/python2.7/site-packages/conda_build/build.py", line 1134, in build
    utils.check_call_env(cmd, env=env, cwd=src_dir)
  File "/reg/g/psdm/sw/conda/inst/miniconda2-dev-rhel7/lib/python2.7/site-packages/conda_build/utils.py", line 620, in check_call_env
    return _func_defaulting_env_to_os_environ(subprocess.check_call, *popenargs, **kwargs)
  File "/reg/g/psdm/sw/conda/inst/miniconda2-dev-rhel7/lib/python2.7/site-packages/conda_build/utils.py", line 616, in _func_defaulting_env_to_os_environ
    return func(_args, **kwargs)
  File "/reg/g/psdm/sw/conda/inst/miniconda2-dev-rhel7/lib/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/bin/bash', '-x', '-e', '/reg/g/psdm/sw/conda/inst/miniconda2-dev-rhel7/conda-bld/psana-conda_1490804988431/work/psana-conda-1.3.0/conda_build.sh']' returned non-zero exit status 2
Traceback (most recent call last):
  File "/reg/g/psdm/sw/conda/manage-dm/bin/ana-rel-admin", line 556, in <module>
    commandFunction(args)
  File "/reg/g/psdm/sw/conda/manage-dm/bin/ana-rel-admin", line 107, in buildPackageCommand
    if pkgBuilder.run():
  File "/reg/g/psdm/sw/conda/manage-dm/pylib/anarelmanage/packagebuilder.py", line 113, in run
    tm = os.path.getctime(packageFileName)-t0
  File "/reg/g/psdm/sw/conda/inst/miniconda2-dev-rhel7/envs/manage/lib/python2.7/genericpath.py", line 72, in getctime
    return os.stat(filename).st_ctime
OSError: [Errno 2] No such file or directory: '/reg/g/psdm/sw/conda/inst/miniconda2-dev-rhel7/conda-bld/linux-64/psana-conda-1.3.0-py27_1.tar.bz2'

The isse seems to be these undefined symbols from boost. You see that boost is coming from defaults because I couldn't get anything higher than 57 to work, but in conda-forge, boost starts at 60. This probably creates weird problems like the above.

I think to properly switch to conda-forge, we should get psana to build against boost 60