search

LSSTDESC / Twinkles

10 years. 6 filters. 1 tiny patch of sky. Thousands of time-variable cosmological distance probes.
MIT License
13 stars 12 forks source link

Request for input concerning next DMstack release #210

Closed heather999 closed 8 years ago

heather999 commented 8 years ago

I was wondering when the next DMstack release would come along. Do we have any guidance or requests to offer @jchiang87? Anyone else? https://community.lsst.org/t/do-we-know-when-winter2016-will-be-released/655

jchiang87 commented 8 years ago

For Twinkles, @SimonKrughoff is better placed to comment than I am. Ideally, that release should contain any needed development in obs_lsstSim such that we don't have to build it from a dev branch. There were some data butler issues as well, e.g., butler.subset seemed to be broken.

danielsf commented 8 years ago

Depending on what you need, I have adopted the policy of releasing a development version of lsst_apps tagged 'sims' every time I update the sims stack.

Which is to say

eups distrib install lsst_apps -t sims

will get you a version of the DM stack that is consistent with their last tagged weekly build, if that is helpful.

rbiswas4 commented 8 years ago

Since we would like the ability to run comparisons between sims outputs and DM outputs, the sims tagged ones are probably the ones we will need in Twinkles.

ktlim commented 8 years ago

@jchiang87 Could someone file a DM JIRA ticket for the butler.subset breakage you mentioned? I'm not aware of any particular problems, and I don't see anything related in the existing tickets.

jchiang87 commented 8 years ago

sure. will do

jchiang87 commented 8 years ago

@ktlim Here's the ticket: DM-5735

danielsf commented 8 years ago

For what it is worth:

There is now a sims-compatible conda install of lsst_apps

You just need to start a conda environment, add the dev channel with

conda config --add channels http://eupsforge.net/conda/dev

and then run

conda install lsst-apps

and you should get lsst-apps in binary form. Let me know if any of those statement prove to be untrue.

heather999 commented 8 years ago

Thanks, @danielsf, I'll give that a try.

heather999 commented 8 years ago

I gave it a try at SLAC on RHEL6-64, using Anaconda Python 2.3 (python 2.7.10) and hit a little snag (I would not rule out pilot error, and I will try it again on my own laptop - perhaps with a newer Anaconda). Any suggestions @danielsf? Here is the ultimate error, with the rest of the output following: Error: ERROR: placeholder '/home/ilan/minonda/envs/_build_placehold_placehold_placehold_placehold_placehold' too short in: swig-3.0.2-0

The following packages will be downloaded:

package build
libgcc-4.8.5 1 922 KB
mkl-11.3.1 0 121.2 MB
nomkl-1.0 0 402 B
openssl-1.0.2g 0 3.2 MB
pcre-8.31 0 534 KB
pixman-0.32.6 0 2.4 MB
sqlite-3.9.2 0 3.9 MB
freetype-2.5.5 0 2.2 MB
swig-3.0.2 0 1.9 MB
fontconfig-2.11.1 5 402 KB
cairo-1.12.18 6 594 KB
qt-4.8.7 1 36.4 MB
eups-2.0.2 0 669 KB
lsst-cfitsio-3360.4 0 3.4 MB
lsst-doxygen-1.8.5.1 0 3.0 MB
lsst-eigen-3.2.5.1 70497dd_0 438 KB
lsst-fftw-3.3.4.2 0 1.8 MB
lsst-gsl-1.16.3 0 6.9 MB
lsst-mariadbclient-0.10.1.11.2 0 27.3 MB
lsst-minuit2-5.28.00.2.1 dae2fb7_0 7.7 MB
lsst-python-d2to1-0.2.12.1 0 44 KB
lsst-xpa-2.1.15.3 0 616 KB
lsst-stsci-distutils-0.3.7.1.post1 b22a065_0 41 KB
lsst-wcslib-5.13.1 0 3.2 MB
lsst-astrometry-net-data-0.10.0.post68 0 1.7 MB
lsst-display-ds9-w.2016.15 0 2.1 MB
lsst-geom-w.2016.15 0 183 KB
lsst-product-configs-1.0.1925 0 7 KB
lsst-astrometry-net-0.50.2.post6 0 25.6 MB
lsst-boost-1.59.5 0 13.5 MB
lsst-base-w.2016.15 0 4.2 MB
lsst-pex-exceptions-w.2016.15 0 5.3 MB
lsst-mysqlpython-1.2.3.1.1.post6 120147b_0 103 KB
lsst-pex-logging-w.2016.15 0 6.4 MB
lsst-psfex-w.2016.15 0 2.4 MB
lsst-pyfits-3.4.0.post2 0 2.0 MB
lsst-scons-2.3.5 0 753 KB
lsst-sconsutils-w.2016.15 0 211 KB
lsst-utils-w.2016.15 0 5.4 MB
lsst-daf-base-w.2016.15 0 7.9 MB
lsst-pex-policy-w.2016.15 0 7.7 MB
lsst-daf-persistence-w.2016.15 0 41.2 MB
lsst-pex-config-w.2016.15 0 2.8 MB
lsst-skypix-w.2016.15 0 145 KB
lsst-daf-butlerutils-w.2016.15 0 27.9 MB
numpy-1.11.0 py27_0 6.2 MB
lsst-coadd-chisquared-w.2016.15 0 3.6 MB
lsst-esutil-0.5.3 0 1.2 MB
lsst-ndarray-w.2016.15 0 10.8 MB
lsst-afw-w.2016.15 0 162.4 MB
lsst-obs-test-w.2016.15 0 38.9 MB
lsst-shapelet-w.2016.15 0 17.2 MB
lsst-skymap-w.2016.15 0 252 KB
lsst-pipe-base-w.2016.15 0 294 KB
lsst-coadd-utils-w.2016.15 0 4.5 MB
lsst-meas-base-w.2016.15 0 18.6 MB
lsst-meas-algorithms-w.2016.15 0 26.4 MB
lsst-ip-isr-w.2016.15 0 5.1 MB
lsst-meas-extensions-psfex-w.2016.15 0 5.7 MB
lsst-meas-astrom-w.2016.15 0 27.3 MB
lsst-meas-deblender-w.2016.15 0 7.8 MB
lsst-ip-diffim-w.2016.15 0 17.9 MB
lsst-obs-lsstsim-w.2016.15 0 21.1 MB
lsst-obs-sdss-w.2016.15 0 4.9 MB
lsst-pipe-tasks-w.2016.15 0 9.9 MB
lsst-apps-w.2016.15 0 20 KB
lsst-meas-modelfit-w.2016.15 0 23.4 MB
pycairo-1.10.0 py27_0 81 KB
python-2.7.11 0 12.0 MB
pytz-2016.3 py27_0 178 KB
setuptools-20.6.7 py27_0 452 KB
sip-4.16.9 py27_0 247 KB
six-1.10.0 py27_0 16 KB
wheel-0.29.0 py27_0 81 KB
cycler-0.10.0 py27_0 11 KB
pip-8.1.1 py27_1 1.5 MB
pyqt-4.11.4 py27_1 3.5 MB
python-dateutil-2.5.2 py27_0 236 KB
matplotlib-1.5.1 np111py27_0 8.2 MB 
------------------------------------------------------------
                                       Total:       826.2 MB

The following NEW packages will be INSTALLED:

cairo:                      1.12.18-6                
cycler:                     0.10.0-py27_0            
eups:                       2.0.2-0                  
fontconfig:                 2.11.1-5                 
freetype:                   2.5.5-0                  
libgcc:                     4.8.5-1                  
libpng:                     1.6.17-0                 
libxml2:                    2.9.2-0                  
lsst-afw:                   w.2016.15-0              
lsst-apps:                  w.2016.15-0              
lsst-astrometry-net:        0.50.2.post6-0           
lsst-astrometry-net-data:   0.10.0.post68-0          
lsst-base:                  w.2016.15-0              
lsst-boost:                 1.59.5-0                 
lsst-cfitsio:               3360.4-0                 
lsst-coadd-chisquared:      w.2016.15-0              
lsst-coadd-utils:           w.2016.15-0              
lsst-daf-base:              w.2016.15-0              
lsst-daf-butlerutils:       w.2016.15-0              
lsst-daf-persistence:       w.2016.15-0              
lsst-display-ds9:           w.2016.15-0              
lsst-doxygen:               1.8.5.1-0                
lsst-eigen:                 3.2.5.1-70497dd_0        
lsst-esutil:                0.5.3-0                  
lsst-fftw:                  3.3.4.2-0                
lsst-geom:                  w.2016.15-0              
lsst-gsl:                   1.16.3-0                 
lsst-ip-diffim:             w.2016.15-0              
lsst-ip-isr:                w.2016.15-0              
lsst-mariadbclient:         0.10.1.11.2-0            
lsst-meas-algorithms:       w.2016.15-0              
lsst-meas-astrom:           w.2016.15-0              
lsst-meas-base:             w.2016.15-0              
lsst-meas-deblender:        w.2016.15-0              
lsst-meas-extensions-psfex: w.2016.15-0              
lsst-meas-modelfit:         w.2016.15-0              
lsst-minuit2:               5.28.00.2.1-dae2fb7_0    
lsst-mysqlpython:           1.2.3.1.1.post6-120147b_0
lsst-ndarray:               w.2016.15-0              
lsst-obs-lsstsim:           w.2016.15-0              
lsst-obs-sdss:              w.2016.15-0              
lsst-obs-test:              w.2016.15-0              
lsst-pex-config:            w.2016.15-0              
lsst-pex-exceptions:        w.2016.15-0              
lsst-pex-logging:           w.2016.15-0              
lsst-pex-policy:            w.2016.15-0              
lsst-pipe-base:             w.2016.15-0              
lsst-pipe-tasks:            w.2016.15-0              
lsst-product-configs:       1.0.1925-0               
lsst-psfex:                 w.2016.15-0              
lsst-pyfits:                3.4.0.post2-0            
lsst-python-d2to1:          0.2.12.1-0               
lsst-scons:                 2.3.5-0                  
lsst-sconsutils:            w.2016.15-0              
lsst-shapelet:              w.2016.15-0              
lsst-skymap:                w.2016.15-0              
lsst-skypix:                w.2016.15-0              
lsst-stsci-distutils:       0.3.7.1.post1-b22a065_0  
lsst-utils:                 w.2016.15-0              
lsst-wcslib:                5.13.1-0                 
lsst-xpa:                   2.1.15.3-0               
matplotlib:                 1.5.1-np111py27_0        
mkl:                        11.3.1-0                 
nomkl:                      1.0-0                    
numpy:                      1.11.0-py27_0            
openssl:                    1.0.2g-0                 
pcre:                       8.31-0                   
pip:                        8.1.1-py27_1             
pixman:                     0.32.6-0                 
pycairo:                    1.10.0-py27_0            
pyparsing:                  2.0.3-py27_0             
pyqt:                       4.11.4-py27_1            
python:                     2.7.11-0                 
python-dateutil:            2.5.2-py27_0             
pytz:                       2016.3-py27_0            
pyyaml:                     3.11-py27_1              
qt:                         4.8.7-1                  
readline:                   6.2-2                    
setuptools:                 20.6.7-py27_0            
sip:                        4.16.9-py27_0            
six:                        1.10.0-py27_0            
sqlite:                     3.9.2-0                  
swig:                       3.0.2-0                  
tk:                         8.5.18-0                 
wheel:                      0.29.0-py27_0            
yaml:                       0.1.6-0                  
zlib:                       1.2.8-0

Proceed ([y]/n)? y

Fetching packages ... libgcc-4.8.5-1 100% |################################| Time: 0:00:00 1.26 MB/s mkl-11.3.1-0.t 100% |################################| Time: 0:00:07 17.56 MB/s nomkl-1.0-0.ta 100% |################################| Time: 0:00:00 17.99 kB/s openssl-1.0.2g 100% |################################| Time: 0:00:01 2.65 MB/s pcre-8.31-0.ta 100% |################################| Time: 0:00:00 789.68 kB/s pixman-0.32.6- 100% |################################| Time: 0:00:01 1.86 MB/s sqlite-3.9.2-0 100% |################################| Time: 0:00:01 2.23 MB/s freetype-2.5.5 100% |################################| Time: 0:00:01 1.53 MB/s swig-3.0.2-0.t 100% |################################| Time: 0:00:01 1.55 MB/s fontconfig-2.1 100% |################################| Time: 0:00:00 757.91 kB/s cairo-1.12.18- 100% |################################| Time: 0:00:00 883.74 kB/s qt-4.8.7-1.tar 100% |################################| Time: 0:00:03 9.94 MB/s eups-2.0.2-0.t 100% |################################| Time: 0:00:00 838.71 kB/s lsst-cfitsio-3 100% |################################| Time: 0:00:01 2.19 MB/s lsst-doxygen-1 100% |################################| Time: 0:00:01 2.09 MB/s lsst-eigen-3.2 100% |################################| Time: 0:00:00 872.49 kB/s lsst-fftw-3.3. 100% |################################| Time: 0:00:01 1.57 MB/s lsst-gsl-1.16. 100% |################################| Time: 0:00:02 3.49 MB/s lsst-mariadbcl 100% |################################| Time: 0:00:04 6.71 MB/s lsst-minuit2-5 100% |################################| Time: 0:00:02 3.32 MB/s lsst-python-d2 100% |################################| Time: 0:00:00 382.96 kB/s lsst-xpa-2.1.1 100% |################################| Time: 0:00:00 779.39 kB/s lsst-stsci-dis 100% |################################| Time: 0:00:00 377.92 kB/s lsst-wcslib-5. 100% |################################| Time: 0:00:01 2.08 MB/s lsst-astrometr 100% |################################| Time: 0:00:01 1.41 MB/s lsst-display-d 100% |################################| Time: 0:00:01 1.64 MB/s lsst-geom-w.20 100% |################################| Time: 0:00:00 449.54 kB/s lsst-product-c 100% |################################| Time: 0:00:00 382.14 kB/s lsst-astrometr 100% |################################| Time: 0:00:04 6.31 MB/s lsst-boost-1.5 100% |################################| Time: 0:00:02 4.93 MB/s lsst-base-w.20 100% |################################| Time: 0:00:01 2.72 MB/s lsst-pex-excep 100% |################################| Time: 0:00:02 2.64 MB/s lsst-mysqlpyth 100% |################################| Time: 0:00:00 407.89 kB/s lsst-pex-loggi 100% |################################| Time: 0:00:02 3.18 MB/s lsst-psfex-w.2 100% |################################| Time: 0:00:01 1.68 MB/s lsst-pyfits-3. 100% |################################| Time: 0:00:01 1.96 MB/s lsst-scons-2.3 100% |################################| Time: 0:00:00 886.89 kB/s lsst-sconsutil 100% |################################| Time: 0:00:00 521.96 kB/s lsst-utils-w.2 100% |################################| Time: 0:00:02 2.68 MB/s lsst-daf-base- 100% |################################| Time: 0:00:02 3.77 MB/s lsst-pex-polic 100% |################################| Time: 0:00:02 3.81 MB/s lsst-daf-persi 100% |################################| Time: 0:00:06 6.81 MB/s lsst-pex-confi 100% |################################| Time: 0:00:01 1.84 MB/s lsst-skypix-w. 100% |################################| Time: 0:00:00 576.65 kB/s lsst-daf-butle 100% |################################| Time: 0:00:04 5.99 MB/s numpy-1.11.0-p 100% |################################| Time: 0:00:01 3.70 MB/s lsst-coadd-chi 100% |################################| Time: 0:00:01 2.25 MB/s lsst-esutil-0. 100% |################################| Time: 0:00:01 1.16 MB/s lsst-ndarray-w 100% |################################| Time: 0:00:02 4.39 MB/s lsst-afw-w.201 100% |################################| Time: 0:00:21 8.10 MB/s lsst-obs-test- 100% |################################| Time: 0:00:06 6.69 MB/s lsst-shapelet- 100% |################################| Time: 0:00:03 5.74 MB/s lsst-skymap-w. 100% |################################| Time: 0:00:00 509.11 kB/s lsst-pipe-base 100% |################################| Time: 0:00:00 591.90 kB/s lsst-coadd-uti 100% |################################| Time: 0:00:01 2.47 MB/s lsst-meas-base 100% |################################| Time: 0:00:03 5.43 MB/s lsst-meas-algo 100% |################################| Time: 0:00:04 6.22 MB/s lsst-ip-isr-w. 100% |################################| Time: 0:00:01 2.83 MB/s lsst-meas-exte 100% |################################| Time: 0:00:01 3.12 MB/s lsst-meas-astr 100% |################################| Time: 0:00:04 6.18 MB/s lsst-meas-debl 100% |################################| Time: 0:00:02 3.41 MB/s lsst-ip-diffim 100% |################################| Time: 0:00:03 5.25 MB/s lsst-obs-lssts 100% |################################| Time: 0:00:03 5.63 MB/s lsst-obs-sdss- 100% |################################| Time: 0:00:01 2.77 MB/s lsst-pipe-task 100% |################################| Time: 0:00:02 3.76 MB/s lsst-apps-w.20 100% |################################| Time: 0:00:00 337.85 kB/s lsst-meas-mode 100% |################################| Time: 0:00:04 6.07 MB/s pycairo-1.10.0 100% |################################| Time: 0:00:00 389.48 kB/s python-2.7.11- 100% |################################| Time: 0:00:03 3.67 MB/s pytz-2016.3-py 100% |################################| Time: 0:00:00 426.10 kB/s setuptools-20. 100% |################################| Time: 0:00:00 849.02 kB/s sip-4.16.9-py2 100% |################################| Time: 0:00:00 449.63 kB/s six-1.10.0-py2 100% |################################| Time: 0:00:00 209.58 kB/s wheel-0.29.0-p 100% |################################| Time: 0:00:00 396.75 kB/s cycler-0.10.0- 100% |################################| Time: 0:00:00 489.99 kB/s pip-8.1.1-py27 100% |################################| Time: 0:00:01 1.31 MB/s pyqt-4.11.4-py 100% |################################| Time: 0:00:01 3.15 MB/s python-dateuti 100% |################################| Time: 0:00:00 453.08 kB/s matplotlib-1.5 100% |################################| Time: 0:00:02 3.58 MB/s Extracting packages ... [lsst-astrometry-net-da############## | 29% [ COMPLETE ]|###################################################| 100% Linking packages ... Error: ERROR: placeholder '/home/ilan/minonda/envs/_build_placehold_placehold_placehold_placehold_placehold' too short in: swig-3.0.2-0

danielsf commented 8 years ago

I have never seen that error before (and it's rare enough that stackoverflow doesn't seem to have any answers). Did it work with the newer anaconda? I would try updating anaconda on the SLAC machine and see if that fixes it.

heather999 commented 8 years ago

An update.. I installed anaconda 2.5 and did "conda update conda" before trying again. It failed the same way. I did find a GitHub issue, which seems to imply that the problem stems from my path being longer than the one used by conda as a placeholder. I can try doing this in another directory with a shorter path to see if that takes care of it: https://github.com/rtfd/readthedocs.org/issues/1902 Ultimately though, I will be hoping to do this at NERSC, and that path length may also present trouble.

danielsf commented 8 years ago

@mjuric Is this something we can easily change within conda-lsst (give the packages larger place-holder names so people installing into environments with ridiculously large directory structures don't have a problem)?

heather999 commented 8 years ago

and just to confirm.. using a less than ridiculously large directory path worked, please note some of us have little control over this! :) I haven't tested the installation yet, but it completed successfully.

danielsf commented 8 years ago

Thanks, @heather999 we will try to fix the conda build to be more accommodating. Sorry this is a problem.

heather999 commented 8 years ago

Thank you @danielsf for the conda install and the help!

mjuric commented 8 years ago

@heather999 This is a generic issue when one tries to make binaries relocatable w/o changing their source code.

The way conda does relocation is that it patches the binary files by looking for placeholder strings and replacing them with the directories into which the code is being installed. This is easy to do in text files, but not so in binary files. If the placeholder is too short, there'll be no room to replace it with the destination path. This is the error that you're seeing.

This is an upstream issue with anaconda -- it's be better if they uses longer placeholders in their builds. I've reported a very similar issue once before -- https://github.com/ContinuumIO/anaconda-issues/issues/623 -- and they fixed it quickly (in that case the placeholder was abnormally short). I know they've discussed increasing the length of placeholder once before, but it seems no action was taken. I think it'd be good to open an issue with them so they know this is a problem; could you do that?

jchiang87 commented 8 years ago

@danielsf I was able to install lsst-apps and lsst-sims on cori in my user directory. The installation seemed to succeed, but when I try to use the code I get this:

[cori11] module load python/2.7-anaconda
[cori11] source '/global/homes/j/jchiang8/.conda/envs/my_root/bin/eups-setups.sh'
[cori11] setup lsst_apps
[cori11] python
Python 2.7.11 |Anaconda 2.3.0 (64-bit)| (default, Dec  6 2015, 18:08:32) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import lsst.afw.image
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/global/homes/j/jchiang8/.conda/envs/my_root/opt/lsst/afw/python/lsst/afw/image/__init__.py", line 26, in <module>
    from .imageLib import *
  File "/global/homes/j/jchiang8/.conda/envs/my_root/opt/lsst/afw/python/lsst/afw/image/imageLib.py", line 34, in <module>
    _imageLib = swig_import_helper()
  File "/global/homes/j/jchiang8/.conda/envs/my_root/opt/lsst/afw/python/lsst/afw/image/imageLib.py", line 30, in swig_import_helper
    _mod = imp.load_module('_imageLib', fp, pathname, description)
  File "/global/homes/j/jchiang8/.conda/envs/my_root/opt/lsst/base/python/lsstimport.py", line 102, in imp_load_module
    module = orig_imp_load_module(name, *args)
ImportError: libssl.so.10: cannot open shared object file: No such file or directory
>>>

Any suggestions?

mjuric commented 8 years ago

@jchiang87 Is openssl installed on Cori (and if so, could you check the version)? Also, what is the version of the OS? @danielsf Where do you build your binaries?

heather999 commented 8 years ago

Looks like openssl is there: heatherk@cori10:~> which openssl /usr/bin/openssl heatherk@cori10:~> ls /usr/lib/ssl /usr/lib/libssl.so.0.9.8 /usr/lib/libssl.so.1.0.0 /usr/lib/libssl3.so heatherk@cori10:~> but wrong version?

cwwalter commented 8 years ago

The OS on Cori is a (cray modified?) version of open suse.

jchiang87 commented 8 years ago

@mjuric Here is the info I could find on the openssl I'm using:

(my_root)[cori11] ls -l libssl*
lrwxrwxrwx 1 jchiang8 jchiang8 75 Apr 12 10:54 libssl.a -> /usr/common/software/python/2.7-anaconda/pkgs/openssl-1.0.2g-0/lib/libssl.a
lrwxrwxrwx 1 jchiang8 jchiang8 15 Apr 12 10:54 libssl.so -> libssl.so.1.0.0*
lrwxrwxrwx 1 jchiang8 jchiang8 82 Apr 12 10:54 libssl.so.1.0.0 -> /usr/common/software/python/2.7-anaconda/pkgs/openssl-1.0.2g-0/lib/libssl.so.1.0.0*
(my_root)[cori11]

As Chris notes, the OS is based on a version of SuSE Linux Enterprise Server distribution.

mjuric commented 8 years ago

@heather999, @jchiang87 -- try creating a link to /usr/lib/libssl.so.1.0.0 named libssl.so.10, and place it somewhere onto the LD_LIBRARY_PATH where the code can see it. The issue is that RHEL installs the openssl library as libssl.so.10, rather than libssl.so.1.0.0 ( e.g., see an analogous issue at http://askubuntu.com/questions/339364/libssl-so-10-cannot-open-shared-object-file-no-such-file-or-directory). See if that will help.

Also, could you open an issue on http://github.com/mjuric/conda-lsst (the right way to fix this is to link packages requiring OpenSSL against the version distributed with Anaconda)?

cwwalter commented 8 years ago

OK FWIW I have done this and it works... but for a previous version.

Jim I think maybe the way you are calling it is wrong. Don't do the module setup.

Just do something like this:

set path = (/global/u1/c/cwalter/LSST-Conda/bin $path) ource /global/u1/c/cwalter/LSST-Conda/bin/eups-setups.csh

then which which python you are using and just try python then you can try a setup pipe_tasks or something to make sure you can import things.

cwwalter commented 8 years ago

Just to be even more explicit: Don't use cori's anaconda setup. First do the anaconda installation yourself. The, install lsat-sims as part of that installation. Use that closed system.

heather999 commented 8 years ago

@cwwalter I've been going back and forth wondering if we wanted to install our own shared anaconda at NERSC - but the convenience factor of using theirs had won out in my recent experiments. So.. are you suggesting we should go with our own shared Anaconda python installation if setting up a shared stack install at NERSC? I'm not advocating one way or the other.. just wondering..

jchiang87 commented 8 years ago

Thanks, Chris. I followed the procedure as best I could ascertain it from reading the NERSC documentation on using Anaconda. I'll give your suggestion a try.

@mjuric That symlink got past the openssl linkage problem, but now I see:

ImportError: libcrypto.so.10: cannot open shared object file: No such file or directory
cwwalter commented 8 years ago

@cwwalter I've been going back and forth wondering if we wanted to install our own shared anaconda at NERSC - but the convenience factor of using theirs had won out in my recent experiments. So.. are you suggesting we should go with our own shared Anaconda python installation if setting up a shared stack install at NERSC?

Hmm... That's how I did it. I would actually be surprised if it worked using the cori shared distribution. Scott built the lsst-sims using his installed anaconda python. You are going to want to use the same system I think. Actually, I don't quite even understand how you installed the lsst-sims packages in your own area using their setup.

In terms of convenience I don't really think there is much of a difference. You just have to do the binary anaconda distribution before you add the channel and do the 'conda install lsst-sim'.

This worked fine for me before. I am using it now. Not promising this will fix the openssl problem with the new version though! :)

jchiang87 commented 8 years ago

For the record, setting a similar symlink for libcrypto.so.10 gets me to the point where the Twinkles unit tests pass. I'll do more tests, but I suspect the cori anaconda can actually work for this.

cwwalter commented 8 years ago

I think the other reason for having our own contained system is that we may need particular versions of anaconda for our versions of the stack/sims. We can't really control what happens to the cori anaconda. So things might either break from under us, or not be up-to-date enough for a new version we need.

heather999 commented 8 years ago

@jchiang87 where does your libcrypto.so.10 point to? Never mind I found it. For fun I installed the conda version at: /project/projectdirs/lsst/lsstDM/Cori/LSST_STACK-conda-2016-04-12 so if you do: source activate /project/projectdirs/lsst/lsstDM/Cori/LSST_STACK-conda-2016-04-12 others could try it out

I do see your point @cwwalter and I don't know yet how often we'll want different or more up to date versions of anaconda. In a conda environment we could potentially install whatever we want... so use the local NERSC anaconda install, but we have our own playground so to speak. I'm certainly open to doing a few different installs to see what we prefer.

mjuric commented 8 years ago

@cwwalter -- I think the LSST conda binaries should pull the correct versions of everything they need into the conda environment (if they don't, it's a bug and we need to fix it). So (in theory), there should not be a significant gain from having your own install of anaconda [*].

[*] The only time you may need to do that is if you're using packages that can only be installed into the root environment, and I believe that's only conda-build, which you don't want to be using anyway :-).

mjuric commented 8 years ago

@jchiang87, @heather999 Good to know that the symlinks worked! Could you still open a bug on conda-lsst, so we don't forget to solve this going forward?

jchiang87 commented 8 years ago

sure, will do.

cwwalter commented 8 years ago

@cwwalter -- I think the LSST conda binaries should pull the correct versions of everything they need into the conda environment (if they don't, it's a bug and we need to fix it). So (in theory), there should not be a significant gain from having your own install of anaconda [*].

Hmm... so how does this actually work? You declare your own environment and it installs everything it needs there? Somehow if you didn't explicitly install it (including through dependencies) it can fall back on the other environment on cori?

heather999 commented 8 years ago

@jchiang87 I'm sure it's documented somewhere .. can you point me to how one runs the Twinkles unit tests?

jchiang87 commented 8 years ago

@heather999 We're still sorting out the documentation on setting up the environment (#189).
This is what I did (after setting up lsst_apps):

cd <Twinkles repo>
eups declare -r . Twinkles $USER
setup Twinkles
nosetests

It will skip the db tests if you don't have a mysql server running, but that is by design.

mjuric commented 8 years ago

@cwwalter: I think it's a matter of passing the --prefix flag to conda create to create an environment in a directory that's writable to you. Here's an example (run on my mac, but should apply everywhere):

[mjuric@D-69-91-143-158 ~]$ conda create --prefix $HOME/foo lsst-galsim
Fetching package metadata: ........
Solving package specifications: .........

Package plan for installation in environment /Users/mjuric/foo:

The following NEW packages will be INSTALLED:

    eups:                 2.0.2-0
    lsst-boost:           1.59.5-0
    lsst-fftw:            3.3.4.2-0
    lsst-galsim:          1.3.2-0
    lsst-product-configs: 1.0.1925-0
    lsst-pyfits:          3.4.0.post2-0
    lsst-python-d2to1:    0.2.12.1-0
    lsst-scons:           2.3.5-1
    lsst-stsci-distutils: 0.3.7.1.post1-b22a065_0
    lsst-tmv:             0.72.4-0
    nomkl:                1.0-0
    numpy:                1.11.0-py27_nomkl_0     [nomkl]
    openssl:              1.0.2g-0
    pip:                  8.1.1-py27_1
    python:               2.7.11-0
    readline:             6.2-2
    setuptools:           20.6.7-py27_0
    sqlite:               3.9.2-0
    tk:                   8.5.18-0
    wheel:                0.29.0-py27_0
    zlib:                 1.2.8-0

Proceed ([y]/n)? y

Linking packages ...
[      COMPLETE      ]|##########################################################################################| 100%
#
# To activate this environment, use:
# $ source activate /Users/mjuric/foo
#
# To deactivate this environment, use:
# $ source deactivate
#
[mjuric@D-69-91-143-158 ~]$ source activate /Users/mjuric/foo
discarding /Users/mjuric/projects/lsst_packaging_conda/miniconda/bin from PATH
prepending /Users/mjuric/foo/bin to PATH
(/Users/mjuric/foo)[mjuric@D-69-91-143-158 ~]$ source '/Users/mjuric/foo/bin/eups-setups.sh'
(/Users/mjuric/foo)[mjuric@D-69-91-143-158 ~]$ setup galsim
(/Users/mjuric/foo)[mjuric@D-69-91-143-158 ~]$ python
Python 2.7.11 |Continuum Analytics, Inc.| (default, Dec  6 2015, 18:57:58)
[GCC 4.2.1 (Apple Inc. build 5577)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import galsim
>>>

Note how it pulled in python, numpy, etc.., including setting up the nomkl versions.

jchiang87 commented 8 years ago

@heather999 I'm getting this:

[cori11] source /project/projectdirs/lsst/lsstDM/Cori/LSST_STACK-conda-2016-04-12/bin/eups-setups.sh
[cori11] setup lsst_apps
Unable to find an acceptable version of lsst_apps
[cori11] setup lsst-apps
Unable to find an acceptable version of lsst-apps
[cori11] eups list lsst_apps
eups list: [Errno 13] Permission denied: '/project/projectdirs/lsst/lsstDM/Cori/LSST_STACK-conda-2016-04-12/var/opt/eups/ups_db/cfitsio/b2003.chain'
[cori11]
cwwalter commented 8 years ago

@mjuric Ah I see. So really the only thing you are using is the hosts 'conda' executable itself. Anything you need needs to be put in your new environment anyway right? It can't access the "default" environment anymore.

heather999 commented 8 years ago

@mjuric One more slightly off-topic question.. I went to add lsst-sims to the conda environment I already installed lsst-apps, it did warn me about pulling in a different tag of numpy: numpy: 1.11.0-py27_0 --> 1.11.0-py27_nomkl_0
Is there any reason to be concerned if lsst-apps and lsst-sims are looking for different versions of numpy? even if only slightly different..

Thanks for trying @jchiang87 I'm poking around the group permissions..

heather999 commented 8 years ago

@jchiang87 this might be why: ls -l /project/projectdirs/lsst/lsstDM/Cori/LSST_STACK-conda-2016-04-12/var/opt/eups/ups_db/cfitsio/ total 128 -rw-rw-r-- 1 heatherk lsst 411 Apr 12 13:01 3360.lsst4.version lrwxrwxrwx 1 heatherk lsst 104 Apr 12 13:01 b2003.chain -> /global/homes/h/heatherk/.conda/envs/.pkgs/lsst-cfitsio-3360.4-0/var/opt/eups/ups_db/cfitsio/b2003.chain lrwxrwxrwx 1 heatherk lsst 104 Apr 12 13:01 conda.chain -> /global/homes/h/heatherk/.conda/envs/.pkgs/lsst-cfitsio-3360.4-0/var/opt/eups/ups_db/cfitsio/conda.chain lrwxrwxrwx 1 heatherk lsst 106 Apr 12 13:01 current.chain -> /global/homes/h/heatherk/.conda/envs/.pkgs/lsst-cfitsio-3360.4-0/var/opt/eups/ups_db/cfitsio/current.chain

I think this is related to a question I had for @mjuric back some weeks ago about setting up a shared conda installation in a new environment - it appeared that some bits were in my home directory... so.. hmm.. this might be demonstrating the limits of my "conda install" knowledge.. Is there something I should be doing beyond a naive: conda create -m -c http://eupsforge.net/conda/dev -p /project/projectdirs/lsst/lsstDM/Cori/LSST_STACK-conda-2016-04-12 lsst-apps

danielsf commented 8 years ago

@mjuric The OSX binaries are built on my laptop; the Linux binaries are built on lsst-dev

mjuric commented 8 years ago

@heather999 -- the numpy changes you're seeing are unfortunate but tough to avoid -- Continuum rebuilt numpy & co. with a MKL backend. MKL provides an (incomplete) implementation of BLAS and FFTW, and that implementation uses exactly the same symbols (function names) as do the regular ones. So once our code (that's been built against FFTW proper) gets loaded into the same address space with numpy (which loads the MKL implementation of BLAS, FFTW, etc), all sorts of symbol collisions and nasty things begin happening, resulting in a crash. So right now you have to acquiesce to 'nomkl' variants of all packages to be able to run (or even build) the stack :(

P.S.: Pure Python code is unaffected by any of this.

mjuric commented 8 years ago

@heather999 And lsst-sims will also pick up the nomkl versions from now on, so there should be no need to worry.

In general, we're striving for conda install lsst-XXX to always does the Right Thing (tm) so there's no need to futz around with anything else.

mjuric commented 8 years ago

Also see here for more about the MKL issue.

heather999 commented 8 years ago

Thanks, @mjuric - that helps a lot. I guess I'm wondering.. why didn't that version of numpy (1.11.0-py27_nomkl_0) get pulled in when I installed lsst-apps initially? To be more clear.. initially I created the conda environment and installed lsst-apps (which installed numpy: 1.11.0-py27_0) .. then I went to install lsst-sims, and I was prompted that numpy would be updated. Wouldn't they both be looking for the same "nomkl"? Maybe I need to read the links you kindly provided! I'll take a peek.

mjuric commented 8 years ago

I'm guessing the lsst-apps build that you had installed is an old one (which channel did they come from?). Those don't get updated regularly, unfortunately (it's a resource issue -- we need a person to make those builds, and everyone's busy with other things right now). In this particular case, I suspect that w/o the nomkl versions that older code may not even run any more. If you get a chance, try python -c 'import lsst.afw.image (in an environ that doesn't have the nomkl feature turned on); I think that used to crash it.

Because of that, conda releases are primarily sims focused -- @danielsf is doing a great job putting them out whenever there's a new sims release. I'm poking around for resources to see if we can put out all of lsst-distrib at the same time as well.

heather999 commented 8 years ago

I'm using the same dev channel for both apps and sims, so i thought i was getting a fairly new lsst-apps. http://eupsforge.net/conda/dev

No complaints from me! Maintaining binary installations is no small task and it's greatly appreciated.

danielsf commented 8 years ago

the lsst_apps installed on the dev channel should have come from the same build as lsst_sims (we have recently decided that, to facilitate use by DESC, we would release a version of lsst_apps tagged as 'sims' and built against the same basic libraries as lsst_sims whenever we release lsst_sims). If one pulls in nomkl and the other does not, that is odd.

mjuric commented 8 years ago

the lsst_apps installed on the dev channel should have come from the same build as lsst_sims (we have recently decided that, to facilitate use by DESC, we would release a version of lsst_apps tagged as 'sims'

Awesome!!

If one pulls in nomkl and the other does not, that is odd.

Yup. I can't reproduce this on lsst-dev, though:

[mjuric@lsst-dev ~]$ conda create --prefix foo --channel http://eupsforge.net/conda/dev lsst-apps
Fetching package metadata: ......
Solving package specifications: ...........

Package plan for installation in environment /nfs/home/mjuric/foo:

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    nomkl-1.0                  |                0          402 B
    pixman-0.32.6              |                0         2.4 MB
    openblas-0.2.14            |                4         3.6 MB
    eups-2.0.2                 |                0         669 KB
    numpy-1.11.0               |     py27_nomkl_0         6.2 MB
    pytz-2016.3                |           py27_0         178 KB
    setuptools-20.6.7          |           py27_0         452 KB
    cairo-1.12.18              |                6         594 KB
    cycler-0.10.0              |           py27_0          11 KB
    lsst-cfitsio-3360.4        |                0         3.4 MB
    lsst-doxygen-1.8.5.1       |                0         3.0 MB
    lsst-eigen-3.2.5.1         |        70497dd_0         438 KB
    lsst-fftw-3.3.4.2          |                0         1.8 MB
    lsst-gsl-1.16.3            |                0         6.9 MB
    lsst-mariadbclient-0.10.1.11.2|                0        27.3 MB
    lsst-minuit2-5.28.00.2.1   |        dae2fb7_0         7.7 MB
    lsst-product-configs-1.0.1925|                0           7 KB
    lsst-python-d2to1-0.2.12.1 |                0          44 KB
    lsst-xpa-2.1.15.3          |                0         616 KB
    pip-8.1.1                  |           py27_1         1.5 MB
    python-dateutil-2.5.2      |           py27_0         236 KB
    lsst-boost-1.59.5          |                0        13.5 MB
    lsst-esutil-0.5.3          |                0         1.2 MB
    lsst-mysqlpython-1.2.3.1.1.post6|        120147b_0         103 KB
    lsst-scons-2.3.5           |                0         753 KB
    lsst-stsci-distutils-0.3.7.1.post1|        b22a065_0          41 KB
    lsst-wcslib-5.13.1         |                0         3.2 MB
    pycairo-1.10.0             |           py27_0          81 KB
    pyqt-4.11.4                |           py27_1         3.5 MB
    lsst-astrometry-net-0.50.2.post6|                0        25.6 MB
    lsst-pyfits-3.4.0.post2    |                0         2.0 MB
    lsst-sconsutils-w.2016.15  |                0         211 KB
    matplotlib-1.5.1           |      np111py27_0         8.2 MB
    lsst-astrometry-net-data-0.10.0.post68|                0         1.7 MB
    lsst-base-w.2016.15        |                0         4.2 MB
    lsst-geom-w.2016.15        |                0         183 KB
    lsst-psfex-w.2016.15       |                0         2.4 MB
    lsst-ndarray-w.2016.15     |                0        10.8 MB
    lsst-pex-exceptions-w.2016.15|                0         5.3 MB
    lsst-utils-w.2016.15       |                0         5.4 MB
    lsst-daf-base-w.2016.15    |                0         7.9 MB
    lsst-pex-logging-w.2016.15 |                0         6.4 MB
    lsst-pex-policy-w.2016.15  |                0         7.7 MB
    lsst-daf-persistence-w.2016.15|                0        41.2 MB
    lsst-pex-config-w.2016.15  |                0         2.8 MB
    lsst-afw-w.2016.15         |                0       162.4 MB
    lsst-display-ds9-w.2016.15 |                0         2.1 MB
    lsst-shapelet-w.2016.15    |                0        17.2 MB
    lsst-skymap-w.2016.15      |                0         252 KB
    lsst-skypix-w.2016.15      |                0         145 KB
    lsst-daf-butlerutils-w.2016.15|                0        27.9 MB
    lsst-obs-test-w.2016.15    |                0        38.9 MB
    lsst-pipe-base-w.2016.15   |                0         294 KB
    lsst-coadd-utils-w.2016.15 |                0         4.5 MB
    lsst-coadd-chisquared-w.2016.15|                0         3.6 MB
    lsst-meas-base-w.2016.15   |                0        18.6 MB
    lsst-meas-algorithms-w.2016.15|                0        26.4 MB
    lsst-ip-isr-w.2016.15      |                0         5.1 MB
    lsst-meas-astrom-w.2016.15 |                0        27.3 MB
    lsst-meas-deblender-w.2016.15|                0         7.8 MB
    lsst-meas-extensions-psfex-w.2016.15|                0         5.7 MB
    lsst-ip-diffim-w.2016.15   |                0        17.9 MB
    lsst-pipe-tasks-w.2016.15  |                0         9.9 MB
    lsst-meas-modelfit-w.2016.15|                0        23.4 MB
    lsst-obs-lsstsim-w.2016.15 |                0        21.1 MB
    lsst-obs-sdss-w.2016.15    |                0         4.9 MB
    lsst-apps-w.2016.15        |                0          20 KB
    ------------------------------------------------------------
                                           Total:       647.0 MB

The following NEW packages will be INSTALLED:

    cairo:                      1.12.18-6
    cycler:                     0.10.0-py27_0
    eups:                       2.0.2-0
    fontconfig:                 2.11.1-5
    freetype:                   2.5.5-0
    libgcc:                     4.8.5-1
    libgfortran:                3.0-0
    libpng:                     1.6.17-0
    libxml2:                    2.9.2-0
    lsst-afw:                   w.2016.15-0
    lsst-apps:                  w.2016.15-0
    lsst-astrometry-net:        0.50.2.post6-0
    lsst-astrometry-net-data:   0.10.0.post68-0
    lsst-base:                  w.2016.15-0
    lsst-boost:                 1.59.5-0
    lsst-cfitsio:               3360.4-0
    lsst-coadd-chisquared:      w.2016.15-0
    lsst-coadd-utils:           w.2016.15-0
    lsst-daf-base:              w.2016.15-0
    lsst-daf-butlerutils:       w.2016.15-0
    lsst-daf-persistence:       w.2016.15-0
    lsst-display-ds9:           w.2016.15-0
    lsst-doxygen:               1.8.5.1-0
    lsst-eigen:                 3.2.5.1-70497dd_0
    lsst-esutil:                0.5.3-0
    lsst-fftw:                  3.3.4.2-0
    lsst-geom:                  w.2016.15-0
    lsst-gsl:                   1.16.3-0
    lsst-ip-diffim:             w.2016.15-0
    lsst-ip-isr:                w.2016.15-0
    lsst-mariadbclient:         0.10.1.11.2-0
    lsst-meas-algorithms:       w.2016.15-0
    lsst-meas-astrom:           w.2016.15-0
    lsst-meas-base:             w.2016.15-0
    lsst-meas-deblender:        w.2016.15-0
    lsst-meas-extensions-psfex: w.2016.15-0
    lsst-meas-modelfit:         w.2016.15-0
    lsst-minuit2:               5.28.00.2.1-dae2fb7_0
    lsst-mysqlpython:           1.2.3.1.1.post6-120147b_0
    lsst-ndarray:               w.2016.15-0
    lsst-obs-lsstsim:           w.2016.15-0
    lsst-obs-sdss:              w.2016.15-0
    lsst-obs-test:              w.2016.15-0
    lsst-pex-config:            w.2016.15-0
    lsst-pex-exceptions:        w.2016.15-0
    lsst-pex-logging:           w.2016.15-0
    lsst-pex-policy:            w.2016.15-0
    lsst-pipe-base:             w.2016.15-0
    lsst-pipe-tasks:            w.2016.15-0
    lsst-product-configs:       1.0.1925-0
    lsst-psfex:                 w.2016.15-0
    lsst-pyfits:                3.4.0.post2-0
    lsst-python-d2to1:          0.2.12.1-0
    lsst-scons:                 2.3.5-0
    lsst-sconsutils:            w.2016.15-0
    lsst-shapelet:              w.2016.15-0
    lsst-skymap:                w.2016.15-0
    lsst-skypix:                w.2016.15-0
    lsst-stsci-distutils:       0.3.7.1.post1-b22a065_0
    lsst-utils:                 w.2016.15-0
    lsst-wcslib:                5.13.1-0
    lsst-xpa:                   2.1.15.3-0
    matplotlib:                 1.5.1-np111py27_0
    nomkl:                      1.0-0
    numpy:                      1.11.0-py27_nomkl_0       [nomkl]
    openblas:                   0.2.14-4
    openssl:                    1.0.2g-0
    pcre:                       8.31-0
    pip:                        8.1.1-py27_1
    pixman:                     0.32.6-0
    pycairo:                    1.10.0-py27_0
    pyparsing:                  2.0.3-py27_0
    pyqt:                       4.11.4-py27_1
    python:                     2.7.11-0
    python-dateutil:            2.5.2-py27_0
    pytz:                       2016.3-py27_0
    pyyaml:                     3.11-py27_1
    qt:                         4.8.7-1
    readline:                   6.2-2
    setuptools:                 20.6.7-py27_0
    sip:                        4.16.9-py27_0
    six:                        1.10.0-py27_0
    sqlite:                     3.9.2-0
    swig:                       3.0.2-0
    tk:                         8.5.18-0
    wheel:                      0.29.0-py27_0
    yaml:                       0.1.6-0
    zlib:                       1.2.8-0

Proceed ([y]/n)?

@heather999 -- is it reproducible on cori?