search

LSSTDESC / Twinkles

10 years. 6 filters. 1 tiny patch of sky. Thousands of time-variable cosmological distance probes.
MIT License
13 stars 12 forks source link

DM Stack and Sims for phoSim Run 3 at SLAC #319

Open drphilmarshall opened 8 years ago

drphilmarshall commented 8 years ago

We said that "we should get the changes we need for Twinkles into the sims part of the stack and get that to run at SLAC, from source." Would you say we could now start this process, Scott?

drphilmarshall commented 8 years ago

In #306, @heather999 reported on the latest installation. Great if you can advise us further when you know what we will need, Scott! Thanks :-)

danielsf commented 8 years ago

@rbiswas4 Has a branch that still needs to be merged into the CatSim master before I can publish a Twinkles-friendly version. I will let y'all know when that happens.

heather999 commented 7 years ago

@danielsf very kindly pointed me at the new lsst_sims that is now available.

The new lsst_sims is out (at long last!) either run eups distrib install lsst_sims -t sims or, if you're using lsstsw rebuild -r sims_2.3.1 -r w.2016.40 lsst_sims

Now for a silly question.. I am doing a fresh installation of both lsst_apps and lsst_sims at SLAC, via "eups distrib install". In that case, what tag of lsst_apps should I use? "sims"? The latest v12_1 release? Maybe "w.2016.40"? I'm going to randomly poke some people who might have an opinion: @jchiang87 @rbiswas4 @SimonKrughoff

jchiang87 commented 7 years ago

This seems like something @danielsf should be able to say definitively. IMO, we should stick to versioned releases whenever possible, but if the version of lsst_sims requires w.2016.40 of lsst_apps or something else, then we should of course use that.

heather999 commented 7 years ago

Ok - Scott responded to my email.. the "sims" tag it is.. for both lsst_apps and lsst_sims. Off I go!!

danielsf commented 7 years ago

@jchiang87 I thought we had moved beyond DM v12 in processing Twinkles. Am I mistaken?

jchiang87 commented 7 years ago

I'm not sure what we decided, tbh. v12.1 just came out a week or so ago. Is the current weekly already that different? I guess my concern is that using non-release versions makes code provenence slightly more complicated and runs counter to my feeling that we should avoid the bleeding edge when possible. Since v12_0 became available at SLAC, I've been using it to rerun processEimage.py on Run1.1 data, and it seems to be doing what we need, but if there are reasons that it (or v12_1) isn't up to the task for Run3, then we should certainly use something more recent.

danielsf commented 7 years ago

I'm sorry. I did not hear about the 12.1 release. It looks like w.2016.40 is 12.1, so everything should be fine.

Whenever I release a new sims version, I give it a unique tag, so that we can always re-create a sims version of the code with, for example,

eups distrib install lsst_apps -t sims_2_3_1
danielsf commented 7 years ago

Incidentally, on August 23, I asked (via email) @SimonKrughoff if we needed something more advanced than version 12.0 for Twinkles. He said he thought we did, based on advancements in obs_lsstSim.

heather999 commented 7 years ago

and that last bit is important to me... knowing what tag(s) are associated with an installation. The "sims" tag is convenient for an install, but it's not reproducible 2 months from now (at least I don't think so!) Let's say when Cori comes up back in a few weeks and we want to reproduce this particular installation at NERSC. I would do: eups distrib install lsst_apps -t sims_2_3_1 eups distrib install lsst_sims -t sims_2_3_1 I expect for convenience, there is some eups command I would use locally to set up the "sims" tag to point to "sims_2_3_1".

danielsf commented 7 years ago

I'm not sure if you can alias eups tags. If you are worried about how to setup an older version of the stack, the setup command would be

setup lsst_apps -t sims_2_3_1
setup lsst_sims -t sims_2_3_1
danielsf commented 7 years ago

But, yes, the rest of your last post,@heather999, is correct

heather999 commented 7 years ago

sadly I have an error for lsst_apps -t sims.. I'm sending a note to community.lsst.org to gain some insight, but I'm open to suggestions.

 [ 58/70 ]  meas_base 12.1+3 ... 

***** error: from /nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/w.2016.40/EupsBuildDir/Linux64/meas_base-12.1+3/build.log:

tests/testApertureFlux.py

tests/testMeasureSources.py

tests/testSincPhotSums.py

tests/testScaledApertureFlux.py

The following tests failed:
/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/w.2016.40/EupsBuildDir/Linux64/meas_base-12.1+3/meas_base-12.1+3/tests/.tests/testClassification.py.failed
/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/w.2016.40/EupsBuildDir/Linux64/meas_base-12.1+3/meas_base-12.1+3/tests/.tests/testJacobian.py.failed
/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/w.2016.40/EupsBuildDir/Linux64/meas_base-12.1+3/meas_base-12.1+3/tests/.tests/testApertureFlux.py.failed
/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/w.2016.40/EupsBuildDir/Linux64/meas_base-12.1+3/meas_base-12.1+3/tests/.tests/testMeasureSources.py.failed
/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/w.2016.40/EupsBuildDir/Linux64/meas_base-12.1+3/meas_base-12.1+3/tests/.tests/testSincPhotSums.py.failed
/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/w.2016.40/EupsBuildDir/Linux64/meas_base-12.1+3/meas_base-12.1+3/tests/.tests/testScaledApertureFlux.py.failed
6 tests failed
scons: *** [checkTestStatus] Error 1
scons: building terminated because of errors.
+ exit -4
eups distrib: Failed to build meas_base-12.1+3.eupspkg: Command:
    source /nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/w.2016.40/eups/bin/setups.sh; export EUPS_PATH=/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/w.2016.40; (/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/w.2016.40/EupsBuildDir/Linux64/meas_base-12.1+3/build.sh) >> /nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/w.2016.40/EupsBuildDir/Linux64/meas_base-12.1+3/build.log 2>&1 4>/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/w.2016.40/EupsBuildDir/Linux64/meas_base-12.1+3/build.msg 
exited with code 252
heather999 commented 7 years ago

New silly question @danielsf, I have gotten much further now.. but ran into the error pasted below. I understood this to suggest that I should have installed git-lfs on this system before doing the installation via newinstall. That is new behavior - well I haven't done a newinstall installation in awhile..and perhaps I wouldn't see this problem if I had done a lsstsw or conda installation. In the past (let's say 6 months ago), I didn't need git or git lfs available. Can you confirm that I should run like a bunny and install git-lsf at SLAC.. actually it seems we would need at least git 1.8.2 available, and these systems have git 1.8.1 :) I can fix that too...

  [ 55/85 ]  sims_maps 2016.08.23+6 ... 

***** error: from /nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/hmk_test/EupsBuildDir/Linux64/sims_maps-2016.08.23+6/build.log:
======================================================================
FAIL: testStarMaps (__main__.MapSizeUnitTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/testFileSize.py", line 64, in testStarMaps
    msg=self.lfs_msg)
AssertionError: 133 not greater than 1048576 : 
You may not have git-lfs installed on your system
http://developer.lsst.io/en/latest/tools/git_lfs.html

----------------------------------------------------------------------
Ran 4 tests in 0.041s

FAILED (failures=2)
The following tests failed:
/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/hmk_test/EupsBuildDir/Linux64/sims_maps-2016.08.23+6/sims_maps-2016.08.23+6/tests/.tests/testFileSize.py.failed
1 tests failed
scons: *** [checkTestStatus] Error 1
scons: building terminated because of errors.
+ exit -4
eups distrib: Failed to build sims_maps-2016.08.23+6.eupspkg: Command:
    source /nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/hmk_test/eups/bin/setups.sh; export EUPS_PATH=/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/hmk_test; (/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/hmk_test/EupsBuildDir/Linux64/sims_maps-2016.08.23+6/build.sh) >> /nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/hmk_test/EupsBuildDir/Linux64/sims_maps-2016.08.23+6/build.log 2>&1 4>/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/hmk_test/EupsBuildDir/Linux64/sims_maps-2016.08.23+6/build.msg 
exited with code 252
danielsf commented 7 years ago

@heather999 Yes. You will need git-lfs to do the from-source install. Sorry.

heather999 commented 7 years ago

Ok - I have git and git-lfs set up at SLAC. Almost there!! Final hurtle, I hope, is that I do not have access to the UW DB - it was not required previously to do the installation, but now it is (at least that's how I am interpreting the message below). I vaguely recall a github issue about this.. Anyway, I have sent @danielsf my public SSH key in the hopes of gaining access to the DB. Once that's done, I have hope that the installation will finish to completion.


***** error: from /nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/w.2016.40-sims_2.3.1/EupsBuildDir/Linux64/sims_catUtils-12.0-20-gd0952e3/build.log:
  File "src/DbAuth.cc", line 83, in std::pair<std::basic_string<char>, std::basic_string<char> > search(const string&, const string&)
    /u/gl/heather/.lsst directory is missing or accessible by others {0}
lsst::pex::exceptions::RuntimeError: '/u/gl/heather/.lsst directory is missing or accessible by others'

----------------------------------------------------------------------
Ran 3 tests in 0.286s

FAILED (errors=1)
To query opsim database: 0.041138 seconds

For more information on database authentication using the db-auth.paf policy file see: https://confluence.lsstcorp.org/display/SIM/Accessing+the+UW+CATSIM+Database

The following tests failed:
/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/w.2016.40-sims_2.3.1/EupsBuildDir/Linux64/sims_catUtils-12.0-20-gd0952e3/sims_catUtils-12.0-20-gd0952e3/tests/.tests/testAllObjects.py.failed
/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/w.2016.40-sims_2.3.1/EupsBuildDir/Linux64/sims_catUtils-12.0-20-gd0952e3/sims_catUtils-12.0-20-gd0952e3/tests/.tests/testSSM_Diasources.py.failed
2 tests failed
scons: *** [checkTestStatus] Error 1
scons: building terminated because of errors.
+ exit -4
eups distrib: Failed to build sims_catUtils-12.0-20-gd0952e3.eupspkg: Command:
    source /nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/w.2016.40-sims_2.3.1/eups/bin/setups.sh; export EUPS_PATH=/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/w.2016.40-sims_2.3.1; (/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/w.2016.40-sims_2.3.1/EupsBuildDir/Linux64/sims_catUtils-12.0-20-gd0952e3/build.sh) >> /nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/w.2016.40-sims_2.3.1/EupsBuildDir/Linux64/sims_catUtils-12.0-20-gd0952e3/build.log 2>&1 4>/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/w.2016.40-sims_2.3.1/EupsBuildDir/Linux64/sims_catUtils-12.0-20-gd0952e3/build.msg 
exited with code 252
wmwv commented 7 years ago

What is the state of or the permissions on /u/gl/heather/.lsst?

heather999 commented 7 years ago

Up to now that directory didn't exist @wmwv :) meaning I also don't have $HOME/.lsst/db-auth.paf setup with the required username/password. Is there a way to set this up without the credentials? I've been looking here for inspiration: https://confluence.lsstcorp.org/display/SIM/Accessing+the+UW+CATSIM+Database

wmwv commented 7 years ago

Very good. If ~/.lsst does not exist, then the install should then skip this test. This error should be triggered at run time, not install time.

The fact that it does not is a bug in sims_catUtils that should be fixed. Please file a ticket at

https://jira.lsstcorp.org/

jchiang87 commented 7 years ago

This is a bit of a hack, but if you go into 

/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/w.2016.40-sims_2.3.1/EupsBuildDir/Linux64/sims_catUtils-12.0-20-gd0952e3/sims_catUtils-12.0-20-gd0952e3/config

and edit db.py to look like this:

config.driver='mssql+pymssql'
config.host='fatboy.phys.washington.edu'
config.port='1433'
config.database='LSSTCATSIM'

you should be able to connect from SLAC without the ssh tunneling since our computers have been whitelisted. We'll need to make this change anyways to run in the SLAC workflow engine. You'll also need to modify your ~/.lsst/db-auth.paf to contain the entry described here, making similar changes for host and port, of course.

heather999 commented 7 years ago

and now it's finally installed at SLAC @TomGlanzman might be interested in this: /nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/w.2016.40-sims_2.3.1 I modified "loadLSST.*" to update the path to our local installation of python, git 1.8.4.5 (the SLAC admins promise me a new git installation "soon" but for now, I just installed our own), and git-lfs Before use, one would also have to do:`

scl enable devtoolset-3 <shell>

Let me know if you run into trouble with it..

jchiang87 commented 7 years ago

@heather999 I'm getting this error when I try to run the Twinkles test code using this build:

  File "test_PostageStampMaker.py", line 8, in <module>
    from desc.twinkles import PostageStampMaker, create_postage_stamps
  File "/nfs/farm/g/desc/u1/twinkles/Twinkles/python/desc/twinkles/__init__.py", line 5, in <module>
    from .twinkles_sky import *
  File "/nfs/farm/g/desc/u1/twinkles/Twinkles/python/desc/twinkles/twinkles_sky.py", line 18, in <module>
    from lsst.sims.catUtils.exampleCatalogDefinitions import\
  File "/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/w.2016.40-sims_2.3.1/Linux64/sims_catUtils/12.0-20-gd0952e3/python/lsst/sims/catUtils/exampleCatalogDefinitions/__init__.py", line 3, in <module>
    from .obsCatalogExamples import *
  File "/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/w.2016.40-sims_2.3.1/Linux64/sims_catUtils/12.0-20-gd0952e3/python/lsst/sims/catUtils/exampleCatalogDefinitions/obsCatalogExamples.py", line 10, in <module>
    class ObsStarCatalogBase(InstanceCatalog, AstrometryStars, PhotometryStars, CameraCoords):
  File "/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/w.2016.40-sims_2.3.1/Linux64/sims_catUtils/12.0-20-gd0952e3/python/lsst/sims/catUtils/exampleCatalogDefinitions/obsCatalogExamples.py", line 12, in ObsStarCatalogBase
    camera = loadCamera(lsst.utils.getPackageDir('obs_lsstSim'))
  File "/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/w.2016.40-sims_2.3.1/Linux64/obs_lsstSim/12.1-1-g8d21232+5/python/lsst/obs/lsstSim/utils.py", line 17, in loadCamera
    camConfig.load(camConfigPath)
  File "/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/w.2016.40-sims_2.3.1/Linux64/pex_config/12.1/python/lsst/pex/config/config.py", line 554, in load
    with open(filename, "r") as f:
IOError: [Errno 13] Permission denied: '/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/w.2016.40-sims_2.3.1/Linux64/obs_lsstSim/12.1-1-g8d21232+5/description/camera/camera.py'

That file is indeed read-protected. Can you open up the permissions please? It might be worth enabling group rw permissions for everything in /nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/w.2016.40-sims_2.3.1 in case there are other files set similarly.

heather999 commented 7 years ago

Ok - I think all the files have updated permissions now @jchiang87

jchiang87 commented 7 years ago

@heather999 Could you install two third-party packages please:

scikit-learn
wcsaxes

I think you can just pip install them. The default versions should be fine. The first one is needed by Twinkles and the latter by Monitor. I had thought there were others needed, but maybe they're part of the new Stack. Thanks!

heather999 commented 7 years ago

@jchiang87 I added both.. I dumbly had always_yes set for conda so it didn't ask permission before installing. scikit-learn updated a number of other packages, see below. I believe this means that in the context of DM, it will use whatever version of the modules it requests, but when we're using python outside of DM, our versions may not precisely match. I'm probably being overly cautious, but I find it bothersome, particularly when we're dealing with astropy, numpy and scipy. I can uninstall and try to backtrack to some earlier version of scikit-learn that doesn't impact the other modules.. or leave it as it is... which is probably just fine. Opinions welcome! My inclination is to revert and install scikit-learn 0.17.1 - would that be ok?

The following packages will be UPDATED:

    astropy:      1.1.1-np110py27_0        --> 1.2.1-np111py27_0      
    conda:        4.1.4-py27_0             --> 4.2.9-py27_0           
    conda-env:    2.5.1-py27_0             --> 2.6.0-0                
    libgfortran:  1.0-0                    --> 3.0.0-1                
    matplotlib:   1.5.1-np110py27_0        --> 1.5.1-np111py27_0      
    numpy:        1.10.4-py27_nomkl_0      [nomkl] --> 1.11.2-py27_nomkl_0      [nomkl]
    openblas:     0.2.14-3                 --> 0.2.14-4               
    pandas:       0.17.1-np110py27_0       --> 0.19.0-np111py27_0     
    ruamel_yaml:  0.11.7-py27_0            --> 0.11.14-py27_0         
    scipy:        0.17.0-np110py27_nomkl_1 [nomkl] --> 0.18.1-np111py27_nomkl_0 [nomkl]