SimonKrughoff
commented
8 years ago Ping me if you have problems with this.
Closed heather999 closed 8 years ago
SimonKrughoff
commented
8 years ago Ping me if you have problems with this.
heather999
commented
8 years ago @SimonKrughoff Thanks - actually I am having problems :) High level summary, it looks to me that when I try to "rebuild" the w.2016.20 build, what I'm actually getting is the v12.0 tag.
I have a fresh copy of lsstsw from github - so this is not necessarily the same version I grabbed back in May when this was done on Cori. I've set up the environment on Edison and installed my own cmake just as we did on Cori last time. Then:
$> source bin/setup.sh
$> rebuild -r w.2016.20 lsst_apps
$> rebuild -r w.2016.20 lsst_sims
And that went perfectly fine as far as I could see. Then if I tried to setup lsst_apps b2055 b2017 or lsst_sims b2056 b2018 or even leaving off a specific tag, I was told there wasn't a compatible version available. Below is the output from doing a "eups list" then I tried rerunning "rebuild -r w.2016.20 lsst_apps" just to see if I could install the version I was looking for and it tells me that everything is up to date. Am I doing something wrong? Or has that build disappeared? but I would have expected the rebulid command to complain if that "w.2016.20" was unknown. Any ideas?
heatherk@edison03:/project/projectdirs/lsst/lsstDM/Edison/w.2016.20/lsstsw> eups list
afw 2.2016.10-16-g7c15dd3+1 b2108 b2109
afwdata 12.0+2 b2108 b2109
astrometry_net 0.50.lsst3+1 b2108 b2109
astrometry_net_data 8.0.0.0+17 b2108 b2109
astropy 0.0.1.lsst2 b2109
base 12.0+1 b2108 b2109
boost 1.60+1 b2108 b2109
cfitsio 3360.lsst4 b2108 b2109
coadd_chisquared 12.0+3 b2108 b2109
coadd_utils 12.0+3 b2108 b2109
daf_base 12.0+1 b2108 b2109
daf_butlerUtils 2016_01.0-6-g74d5a09+6 b2108 b2109
daf_persistence 12.0+1 b2108 b2109
display_ds9 12.0+2 b2108
doxygen 1.8.5.lsst1 b2108 b2109
eigen 3.2.5.lsst2 b2108 b2109
esutil 0.5.3+1 b2108 b2109
fftw 3.3.4.lsst2 b2108 b2109
freetds 0.91.112.lsst2 b2109
galsim 1.3.2.lsst2+2 b2109
geom 12.0+1 b2108 b2109
gsl 1.16.lsst3 b2108 b2109
healpy 1.8.1.lsst2+1 b2109
ip_diffim 2016_01.0-10-gad2afe7+1 b2108 b2109
ip_isr 2016_01.0-8-g92a194c+6 b2108 b2109
lmfit 0.9.3+2 b2108 b2109
lsst_apps 12.0+8 b2108
lsst_build LOCAL:/project/projectdirs/lsst/lsstDM/Edison/w.2016.20/lsstsw/lsst_build setup
lsst_sims 12.0+9 b2109
mariadbclient 10.1.11.lsst3 b2108 b2109
matplotlib 0.0.2-1-g49c793a b2108 b2109
meas_algorithms 2016_01.0-13-gd3d4253+1 b2108 b2109
meas_astrom 2016_01.0-9-g37dbcbe+1 b2108 b2109
meas_base 2016_01.0-20-g803754d+1 b2108 b2109
meas_deblender 2016_01.0-5-ga00db6a+1 b2108 b2109
meas_extensions_psfex 2016_01.0-7-g55fec7f+1 b2108
meas_extensions_simpleShape 12.0+2 b2108
meas_modelfit 2016_01.0-9-g93ba6e1+1 b2108
minuit2 5.34.14 b2108 b2109
mysqlpython 1.2.3.lsst2+2 b2108 b2109
ndarray 12.0+6 b2108 b2109
numpy 0.0.2+1 b2108 b2109
obs_lsstSim 2016_01.0-19-g85d3ac2+10 b2108 b2109
obs_sdss 2016_01.0-7-g186727e+8 b2108
obs_test 2016_01.0-3-gafa6dd0+22 b2108 b2109
oorb 12.0 b2109
palpy 1.7.0.lsst2 b2109
pex_config 12.0+1 b2108 b2109
pex_exceptions 12.0+3 b2108 b2109
pex_logging 12.0+3 b2108 b2109
pex_policy 12.0+1 b2108 b2109
pipe_base 12.0+3 b2108 b2109
pipe_tasks 2016_01.0-42-g8d87e0e+2 b2108 b2109
psfex 12.0+2 b2108
pyephem 3.7.5.1.lsst1 b2109
pyfits 3.4.0+4 b2108 b2109
pykg_config 1.2.0+6 b2109
pymssql 2.1.1+4 b2109
python 0.0.4 b2108 b2109
python_d2to1 0.2.12.lsst1 b2108 b2109
python_psutil 4.1.0+1 b2108 b2109
pyyaml 3.11.lsst1+1 b2108 b2109
scipy 0.0.1.lsst1+1 b2108 b2109
scons 2.5.0.lsst1 b2108 b2109
sconsUtils 2016_01.0-7-g94b384b+1 b2108 b2109
shapelet 2016_01.0-4-g1b6a188+2 b2108
sims_GalSimInterface 12.0+7 b2109
sims_catUtils 12.0-2-ga8167b4+2 b2109
sims_catalogs_generation 12.0+3 b2109
sims_catalogs_measures 12.0+3 b2109
sims_coordUtils 12.0+5 b2109
sims_data 10.1 b2109
sims_maf 12.0+7 b2109
sims_maps 12.0+3 b2109
sims_movingObjects 12.0+3 b2109
sims_photUtils 12.0+3 b2109
sims_sed_library 12.0 b2109
sims_skybrightness 12.0-1-g21364b6 b2109
sims_skybrightness_data 12.0 b2109
sims_utils 12.0+3 b2109
skymap 12.0+7 b2108 b2109
skypix 12.0+3 b2108 b2109
sncosmo 12.0 b2109
sqlalchemy 1.0.8.lsst3+1 b2109
stsci_distutils 0.3.7.lsst1 b2108 b2109
swig 3.0.2.lsst1 b2108 b2109
throughputs 12.0 b2109
tmv 0.73+2 b2109
utils 2016_01.0-3-g5dd904a+1 b2108 b2109
wcslib 5.13.lsst1 b2108 b2109
xpa 2.1.15.lsst3 b2108 
jchiang87
commented
8 years ago Is there any reason not to just use the conda distribution for v12_0 instead of building from source? It has both lsst_apps and lsst_sims.
heather999
commented
8 years ago I thought I should stick with w.2016.20 @jchiang87 but perhaps I do not understand the requirements for an Edision install. I gather it's not for a Twinkles Run 1.1 check against the SLAC runs - so is v12_0 what we'd want? Bryce is the one who put in the request, but I do not know if he has specific needs. I'll set up v12_0, just as I need to do at SLAC.
danielsf
commented
8 years ago @heather999 "And that went perfectly fine as far as I could see. Then if I tried to setup lsst_apps b2055 or lsst_sims b2056 or even leaving off a specific tag"
Where did the tags b2055 or b2056 come from?
Leaving off a specific tag should not have worked. If you do not specify a tag, eups tries to setup the tag 'current'. You do not appear to have a 'current' version on Edison (and probably shouldn't; current is special and a local install of lsstsw should not create a 'current' tag).
jchiang87
commented
8 years ago I would have thought that v12_0 has everything we need that would be in w.2016.20, so since we are going to install v12_0 anyways (I would hope), we could save some effort here. I guess I would recommend installing v12_0 first, then if Bryce or whoever really needs w.2016.20, they could chime in.
heather999
commented
8 years ago @danielsf sorry.. it should be b2017 and b2018 for sims, as indicated in issue #239
b2055 I think was the tag for the obs_sdss to run the demo. "setup lsst_apps -t b2017" fails similarly
I have the output when running the original rebuild, which seems to be grabbing b2108
heatherk@edison04:/project/projectdirs/lsst/lsstDM/Edison/w.2016.20/lsstsw> rebuild -r w.2016.20 lsst_apps
lsst_apps: ok (1.1 sec).
meas_deblender: ok (6.9 sec).
afw: ok (25.5 sec).
daf_base: ok (4.4 sec).
utils: ok (3.4 sec).
base: ok (1.8 sec).
boost: ok (23.8 sec).
python: ok (1.1 sec).
sconsUtils: ok (2.5 sec).
scons: ok (4.2 sec).
doxygen: ok (7.0 sec).
swig: ok (10.8 sec).
pex_exceptions: ok (3.4 sec).
numpy: ok (1.5 sec).
python_psutil: ok (1.9 sec).
daf_persistence: ok (7.9 sec).
mariadbclient: ok (22.3 sec).
pex_logging: ok (2.4 sec).
pex_policy: ok (3.4 sec).
pyfits: ok (4.2 sec).
python_d2to1: ok (1.1 sec).
stsci_distutils: ok (1.2 sec).
pyyaml: ok (1.8 sec).
pex_config: ok (3.0 sec).
ndarray: ok (2.5 sec).
eigen: ok (6.6 sec).
fftw: ok (6.6 sec).
cfitsio: ok (8.0 sec).
wcslib: ok (5.4 sec).
minuit2: ok (3.3 sec).
gsl: ok (5.1 sec).
matplotlib: ok (1.1 sec).
afwdata: ok (402.1 sec).
meas_algorithms: ok (5.3 sec).
meas_base: ok (5.2 sec).
coadd_utils: ok (1.9 sec).
pipe_base: ok (2.2 sec).
obs_test: ok (5.8 sec).
daf_butlerUtils: ok (6.4 sec).
skypix: ok (1.3 sec).
geom: ok (1.4 sec).
meas_modelfit: ok (6.0 sec).
pipe_tasks: ok (6.9 sec).
esutil: ok (3.3 sec).
meas_astrom: ok (6.7 sec).
astrometry_net: ok (5.5 sec).
astrometry_net_data: ok (2.3 sec).
ip_isr: ok (2.8 sec).
ip_diffim: ok (4.9 sec).
lmfit: ok (1.9 sec).
scipy: ok (1.0 sec).
coadd_chisquared: ok (1.6 sec).
skymap: ok (1.6 sec).
shapelet: ok (2.9 sec).
obs_lsstSim: ok (8.3 sec).
mysqlpython: ok (1.3 sec).
obs_sdss: ok (3.5 sec).
meas_extensions_simpleShape: ok (1.1 sec).
display_ds9: ok (1.1 sec).
xpa: ok (2.0 sec).
meas_extensions_psfex: ok (1.7 sec).
psfex: ok (3.5 sec).
# BUILD ID: b2108
cfitsio: 3360.lsst4 ................................ok (85.4 sec).
minuit2: 5.34.14 ..............ok (44.2 sec).
eigen: 3.2.5.lsst2 ok (5.2 sec).
python: 0.0.4 ok (4.2 sec).
swig: 3.0.2.lsst1 ..........................ok (76.0 sec).
mariadbclient: 10.1.11.lsst3 ..............................................................................................................................................................................................ok (408.1 sec).
pyyaml: 3.11.lsst1+1 ok (5.0 sec).
doxygen: 1.8.5.lsst1 ................................ok (87.9 sec).
wcslib: 5.13.lsst1 .............................ok (79.8 sec).
numpy: 0.0.2+1 ok (5.9 sec).
xpa: 2.1.15.lsst3 .ok (27.4 sec).
fftw: 3.3.4.lsst2 .................................................................................................................................................................................................................................................................................................ok (600.7 sec).
esutil: 0.5.3+1 ...........ok (36.4 sec).
scons: 2.5.0.lsst1 ok (6.3 sec).
python_d2to1: 0.2.12.lsst1 ok (7.2 sec).
sconsUtils: 2016_01.0-7-g94b384b+1 ok (11.4 sec).
mysqlpython: 1.2.3.lsst2+2 ok (9.2 sec).
boost: 1.60+1 ............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................ok (1381.5 sec).
gsl: 1.16.lsst3 ............................................................ok (141.6 sec).
astrometry_net: 0.50.lsst3+1 ...............................................................................................................ok (247.2 sec).
python_psutil: 4.1.0+1 ok (10.2 sec).
matplotlib: 0.0.2-1-g49c793a ok (5.4 sec).
scipy: 0.0.1.lsst1+1 ok (4.7 sec).
afwdata: 12.0+2 .ok (16.1 sec).
psfex: 12.0+2 .............................ok (71.4 sec).
stsci_distutils: 0.3.7.lsst1 ok (7.8 sec).
base: 12.0+1 .ok (14.4 sec).
geom: 12.0+1 ..ok (17.4 sec).
astrometry_net_data: 8.0.0.0+17 ok (7.3 sec).
lmfit: 0.9.3+2 ok (6.6 sec).
pex_exceptions: 12.0+3 .........ok (31.5 sec).
ndarray: 12.0+6 ....................................ok (86.2 sec).
pyfits: 3.4.0+4 ...................................ok (85.4 sec).
utils: 2016_01.0-3-g5dd904a+1 ok (30.6 sec).
daf_base: 12.0+1 .................................................ok (110.6 sec).
pex_policy: 12.0+1 ..............................ok (73.4 sec).
pex_logging: 12.0+3 ..................ok (48.8 sec).
pex_config: 12.0+1 ..........ok (32.8 sec).
daf_persistence: 12.0+1 ...........................ok (68.2 sec).
afw: 2.2016.10-16-g7c15dd3+1 .........................................................................................................................................................................................................................................................................................................................................................................................................................................ok (897.5 sec).
skypix: 12.0+3 ....ok (21.8 sec).
daf_butlerUtils: 2016_01.0-6-g74d5a09+6 ok (22.3 sec).
shapelet: 2016_01.0-4-g1b6a188+2 .................................................ok (143.7 sec).
display_ds9: 12.0+2 ...ok (19.0 sec).
obs_test: 2016_01.0-3-gafa6dd0+22 ok (20.9 sec).
skymap: 12.0+7 ....ok (21.2 sec).
pipe_base: 12.0+3 .......ok (27.3 sec).
coadd_utils: 12.0+3 .....................................ok (88.3 sec).
coadd_chisquared: 12.0+3 .............................................ok (104.3 sec).
meas_base: 2016_01.0-20-g803754d+1 ....................................................................................................................................................ok (343.3 sec).
meas_extensions_simpleShape: 12.0+2 .........................................................................ok (160.0 sec).
meas_algorithms: 2016_01.0-13-gd3d4253+1 ..............................................................................................................ok (268.2 sec).
ip_isr: 2016_01.0-8-g92a194c+6 ...........................................................................................ok (228.2 sec).
meas_extensions_psfex: 2016_01.0-7-g55fec7f+1 .........................................................................ok (191.3 sec).
meas_deblender: 2016_01.0-5-ga00db6a+1 .....................................................ok (152.3 sec).
ip_diffim: 2016_01.0-10-gad2afe7+1 ...........................................................................................................ok (261.6 sec).
meas_astrom: 2016_01.0-9-g37dbcbe+1 ........................................................ok (157.0 sec).
pipe_tasks: 2016_01.0-42-g8d87e0e+2 ...................ok (86.8 sec).
obs_sdss: 2016_01.0-7-g186727e+8 ok (31.8 sec).
obs_lsstSim: 2016_01.0-19-g85d3ac2+10 ...........................................ok (135.3 sec).
meas_modelfit: 2016_01.0-9-g93ba6e1+1 .......................................................................................................................................ok (316.8 sec).
lsst_apps: 12.0+8 .....ok (23.4 sec).
# BUILD b2108 completed. The bNNNN numbers are generated locally by lsstsw each time it rebuilds the stack. If I interpret this conversation correctly, b2017 was a build generated on a different machine (probably lsst-dev). The only way you could get access to that build on this machine was if it was published as a release, in which case you could install it with eups distrib install. Given that you are rebuilding the stack from source with lsstsw, you will only have access to the bNNNNs generated locally by this machine with this installation of lsstsw. In this case, the rebuild you just ran was given the number b2108, so that is all you can set up.
As far as I know, there is no way to control what bNNNN is assigned to each rebuild, so you just need to keep notes correlating build numbers with version of the software you built.
Does this make sense?
danielsf
commented
8 years ago Regarding all of the packages that seem to be building the 12.0 tag, these are just packages for which 12.0 and w.2016.20 correspond to the same commit (at least, I just verified that this was true for daf_base, which shows up as 12.0 in your build log). If you look at, for example, afw in your build log, it is building commit 7c15dd3 which, if you look here
https://github.com/lsst/afw/releases
is the commit corresponding to w.2016.20. So: at least you are building what you think you are building. I do not know what the build log cannot say that more explicitly.
heather999
commented
8 years ago ah.. @danielsf I clearly did not understand how the b#### were set up :) Ok that helps a lot. I'll try to test w.2016.20 now and hopefully declare success.
I've gone ahead with trying out a conda installation of v12_0 as well. At the moment, it seems "stuck" extracting lsst-pex-policy. I'll give it a little more time, and then I'll try again.
Extracting packages ... [lsst-pex-policy ]|############################## | 57%
heather999
commented
8 years ago I let the conda installation go a really long time (hours) and it finally finished. Maybe there was something going on at NERSC. @jchiang87 in the past I've added nose, coverage, iminuit and ipython to the python modules.. I see ipython already in place and pandas too. Do you still want nose or does pytest suffice?
jchiang87
commented
8 years ago might as well add both if it's not too much trouble.
heather999
commented
8 years ago Just a question.. @jchiang87 installing nose, coverge, iminuit, all result in a request to update python itself (at least if I blindly grab the current release of those modules). For example: The following packages will be downloaded:
| package | build |
|---|---|
| python-2.7.11 | 5 12.0 MB |
The following NEW packages will be INSTALLED: nose: 1.3.7-py27_1 (soft-linkaThe following packages will be UPDATED: python: 2.7.11-0 --> 2.7.11-5 (soft-link)
I believe that is harmless enough.. agreed?
jchiang87
commented
8 years ago I think it should be, but I honestly don't know for sure.
heather999
commented
8 years ago Ok - I'm willing to try it out @jchiang87 .. but first I decided to run lsst_dm_stack_demo on the fresh conda installation of v12_0, before updating anything. It ran just fine, however running ./bin/compare detected_sources.txt reported.. well there was one UserWarning:
/project/projectdirs/lsst/lsstDM/Edison/v12_0/opt/lsst/stsci_distutils/lib/python/stsci.distutils-0.3.7-py2.7.egg/stsci/__init__.py:7: UserWarning: Module pyfits was already imported from /project/projectdirs/lsst/lsstDM/Edison/v12_0/opt/lsst/pyfits/lib/python/pyfits-3.4-py2.7-linux-x86_64.egg/pyfits/__init__.pyc, but /global/project/projectdirs/lsst/lsstDM/Edison/v12_0/opt/lsst/pyfits/lib/python/pyfits-3.4-py2.7-linux-x86_64.egg is being added to sys.path __import__('pkg_resources').declare_namespace(__name__)
Processing completed successfully. The results are in detected-sources.txt.
heatherk@edison03:/project/projectdirs/lsst/lsstDM/Edison/lsst_dm_stack_demo-12.0> ./bin/compare detected-sources.txt
Failed (absolute difference 1.00002e-10, relative difference 4.0431e-12 over tolerance 0) in column base_PsfFlux_fluxSigma.
Failed (absolute difference 1.00002e-10, relative difference 4.51474e-12 over tolerance 0) in column base_PsfFlux_fluxSigma.Meanwhile, the w.2016.20 installation reported no such failures after running ./bin/demo.sh, followed by running ./bin/compare. Just note that I have a separate lsst_dm_stack_demo set up (copied from the lsst_dm_stack_demo I obtained for the Cori w.2016.20 install) while for v12_0, I followed the testing directions and grabbed:
curl -L https://github.com/lsst/lsst_dm_stack_demo/archive/12.0.tar.gz | tar xvzf -
https://pipelines.lsst.io/install/demo.html
heatherk@edison04:/project/projectdirs/lsst/lsstDM/Edison/w.2016.20/lsst_dm_stack_demo> ./bin/compare detected-sources.txt
Ok.My plan is to repeat the conda installation at SLAC and see if those failures can be replicated - and then send a note to community.lsst.org.
On the positive side.. we now have 2 installs on Edison :) Once I get this sorted out for v12_0, I'll see about adding in nose, etc and repeat the test.
heather999
commented
8 years ago Getting back to this - there is now a SLAC install of 12_0 and at NERSC on Edison an install of w.2016.20. Sorry @jbkalmbach I finally figured out your id :) The 12_0 install at NERSC did not fully pass the demo testing and I have submitted a note to our friends on community.lsst.org. Cori is scheduled to return on Jun 30th, but we will have to rebuild or reinstall DMstack before we can use Cori again.
So, Edison may be our best bet at least for this week. To use w.2016.20 source /project/projectdirs/lsst/lsstDM/Edison/w.2016.20/setupDM-w_2016_20.sh That will set up your environment and then all you need do is a typical: setup lsst_apps -t b2108 setup lsst_sims -t b2109
Just let me know if you run into trouble. I'll make 12_0 available at NERSC hopefully later this week. If need be, I'll install from source.
heather999
commented
8 years ago Here's a link to the conversation on community.lsst.org: https://community.lsst.org/t/two-failed-quantities-running-demo-for-12-0-at-nersc/894/3 Generally it seems agreed the failures in the demo are likely insignificant. Being that this is on Edison, I'd like to repeat this install on Cori once it comes back up. @tony-johnson @jchiang87 do we see any need to run a comparison between the conda installation of 12_0 and the same version installed via source? I'm just a little curious if we'll see differences. Unfortunately the upcoming re-run of Run 1.1 won't reveal such differences since that version (w.2016.20) has to be installed from source.
jchiang87
commented
8 years ago The differences look insignificant to me. I don't see any need for us to compare source vs conda installs for v12_0. In general, I don't think we can do regressions on every new version of the software that's installed.
heather999
commented
8 years ago Sure, we don't want to be testing every time, for every version, but as we're just getting going on NERSC, i might be nice to be sure we understand if there is any significant difference between installing the conda binaries versus building from source. Yes - I'm being overly cautious :) But I know, we don't exactly have oodles of time for this.
heather999
commented
8 years ago So I think we've decided the 12_0 conda installation on Edison is good to go. For those who want to try it out: source /project/projectdirs/lsst/lsstDM/Edison/setupStack-12_0.sh optionally add "-v" for a verbose output indicating what the script is up to. When finished with DMstack: 'source deactivate'
I'm going to see about adding some doc to the ComputingInfrastructure space to keep track of these installs.
We're hoping Cori will be back tomorrow afternoon. At that time, I'll start a reinstall of w_2016.20 and introduce 12_0.. see #268
I'll close this issue later today.
Cori is down for the next few weeks and it would be nice to have DMstack installed and available on Edison in the meantime.