Closed humnaawan closed 2 years ago
humnaawan
commented
3 years ago Looking at the released versions of sims_maf, I see version 2.13.0 is not that recent, so I don't know what exactly has changed to make sims_maf behave differently in the recent weeks. I have opened an issue re the missing Stacker in sims_maf but I would like to solve the issue with setup regardless of what happens with the official sims_maf.
heather999
commented
3 years ago First some details: The desc-stack-weekly is updated to follow those LSST Science Pipeline weekly builds. We were paused on w_2021_03 for a number of weeks as we waited for the next successful build of lsst_sims, which appeared in w_2021_08. During those weeks, DM has moved from python 3.7 to python 3.8. This may be associated with those errors you see about matplotlib so I can't say it's really possible to use that older version of sims_maf 2.13.0 with a very recent version of the LSST Science Pipelines.
I think the choice is to either use your local sims_maf with an older LSST Science Pipelines - we do have them available. Do you recall how recently things worked?
Alternatively, you would need to move to a more recent sims_maf and sort out with the developers the status of SeasonStacker. I did a little search in Slack for "SeasonStacker" and found some relevant messages from Lynne Jones back in August, and more recently this January. It sounds like they have moved away from that in favor of SeasonLengthMetric. And we can see in the commit history on sims_maf, that SeasonStacker was removed 15 days ago.
humnaawan
commented
3 years ago hi heather, i think i was able to successfully run things on Feb 19-20. i think for now, i'd prefer to just run things with the old stack, and find time later to upgrade everything. thanks for the resources on the status of the stacker!
heather999
commented
3 years ago Ok - so that would likely mean w_2021_03 should work. It sounds like you need your environment to be set up on the Cori command line - so I'm going to ignore any details about setting up a jupyter kernel for use on jupyter.nersc.gov.
For the Cori command line, you can do the steps that start-kernel-cli.py plus desc-stack-weekly is doing for you and explicitly choose what version of the LSST Science Pipelines & sims you run, but you need to know that desc-stack-weekly, is really just a shifter image. In this case the shifter image's name is lsstdesc/stack-jupyter:w_2021_03-sims_w_2021_03-v3
It is "-v3" because this shifter image was updated a couple of times as new versions of GCRCatalogs became available. You can do:
heatherk@cori11:/global/cscratch1/sd/heatherk> shifter --image=lsstdesc/stack-jupyter:w_2021_03-sims_w_2021_03-v3 /bin/bash
bash-4.2$ source /opt/lsst/software/stack/loadLSST.bash
(lsst-scipipe-cb4e2dc) bash-4.2$ setup lsst_distrib
(lsst-scipipe-cb4e2dc) bash-4.2$ setup lsst_sims
(lsst-scipipe-cb4e2dc) bash-4.2$ which python
/opt/lsst/software/stack/conda/miniconda3-py37_4.8.2/envs/lsst-scipipe-cb4e2dc/bin/pythonNow you should hopefully be able to set up your sims_maf and it will be happy.
humnaawan
commented
3 years ago thanks @heather999! so i am able to set up sims_maf_contrib but not sims_maf:
awan@cori09:~: shifter --image=lsstdesc/stack-jupyter:w_2021_03-sims_w_2021_03-v3 /bin/bash
awan@cori09:~: source /opt/lsst/software/stack/loadLSST.bash
(lsst-scipipe-cb4e2dc) awan@cori09:~: setup lsst_distrib
(lsst-scipipe-cb4e2dc) awan@cori09:~: setup lsst_sims
(lsst-scipipe-cb4e2dc) awan@cori09:~: python
Python 3.7.8 | packaged by conda-forge | (default, Jul 23 2020, 03:54:19)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import lsst.sims.maf as maf
>>> maf.__version__
'2.13.0.sims-64-g788d3062+6d85247962'
>>> exit()
(lsst-scipipe-cb4e2dc) awan@cori09:~: setup sims_maf -r /global/homes/a/awan/LSST/lsstRepos/sims_maf
setup: in file /global/homes/a/awan/LSST/lsstRepos/sims_maf/ups/sims_maf.table: Product numpy not found
(lsst-scipipe-cb4e2dc) awan@cori09:~: python
Python 3.7.8 | packaged by conda-forge | (default, Jul 23 2020, 03:54:19)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import lsst.sims.maf as maf
>>> maf.__version__
'2.13.0.sims-64-g788d3062+6d85247962'
>>> exit()
(lsst-scipipe-cb4e2dc) awan@cori09:~: setup sims_maf_contrib -r /global/homes/a/awan/LSST/lsstRepos/sims_maf_contrib
(lsst-scipipe-cb4e2dc) awan@cori09:~: python
Python 3.7.8 | packaged by conda-forge | (default, Jul 23 2020, 03:54:19)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import lsst.sims.maf as maf
>>> maf.__version__
'2.13.0.sims-64-g788d3062+6d85247962'
>>> import mafContrib
>>> exit()I honestly don't know if the sims_maf was working back in February since I went through my work log and I see that I only really used the code in sims_maf_contrib. I do know that I used my local sims_maf about a year ago (latest May 2020).
Do you have ideas about the source of the error (I'm assuming its an error): setup: in file /global/homes/a/awan/LSST/lsstRepos/sims_maf/ups/sims_maf.table: Product numpy not found?
heather999
commented
3 years ago This probably means you need to move back further to an earlier weekly build. Last Spring, DM moved to use conda-forge, my guess is that you were using a weekly before that switch. It's hard to say precisely which one will work, but anything before w_2020_18 might work. We have lsstdesc/stack-jupyter:w_2020_07-v1 available at NERSC
I'm going under the assumption you'd like to have the additional DESC packages that are available in these 'desc-stack' installs, but clearly this w_2020_07 image won't have the latest GCRCatalogs - but maybe that is ok for what you need to do. Let's see if w_2020_07 works and go from there:
heatherk@cori11:/global/cscratch1/sd/heatherk> shifter --image=lsstdesc/stack-jupyter:w_2020_07-sims_w_2020_07-v1 /bin/bash
bash-4.2$ source /opt/lsst/software/stack/loadLSST.bash
(lsst-scipipe-cb4e2dc) bash-4.2$ setup lsst_distrib
(lsst-scipipe-cb4e2dc) bash-4.2$ setup lsst_simsthank you so much! it works:
awan@cori10:~: shifter --image=lsstdesc/stack-jupyter:w_2020_07-sims_w_2020_07-v1 /bin/bash
awan@cori10:~: source /opt/lsst/software/stack/loadLSST.bash
(lsst-scipipe-4d7b902) awan@cori10:~: setup lsst_distrib
(lsst-scipipe-4d7b902) awan@cori10:~: setup lsst_sims
(lsst-scipipe-4d7b902) awan@cori10:~: setup sims_maf -r /global/homes/a/awan/LSST/lsstRepos/sims_maf
(lsst-scipipe-4d7b902) awan@cori10:~: python
Python 3.7.2 (default, Dec 29 2018, 06:19:36)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import lsst.sims.maf as maf
>>> maf.__version__
'unknown'thanks so much again - you're awesome.
heather999
commented
3 years ago Happy to hear it works! I'll close this but feel free to reopen or start a new issue if something comes up.
humnaawan
commented
3 years ago sorry to bother again but it appears that while i am able to load my local sim-maf successfully when i use the 2020-07-v1 stack, something is up with matplotlib :
Traceback (most recent call last):
File "/global/u2/a/awan/LSST/lsstRepos/os-systematics/likelihood-analysis/helpers_plotutils.py", line 54, in plot_skymap
ticks=ticks, format=num_format, cax=cbaxes
File "/global/homes/a/awan/.local/lib/python3.7/site-packages/matplotlib/pyplot.py", line 2194, in colorbar
ret = gcf().colorbar(mappable, cax=cax, ax=ax, **kw)
File "/global/homes/a/awan/.local/lib/python3.7/site-packages/matplotlib/figure.py", line 2343, in colorbar
cb = cbar.colorbar_factory(cax, mappable, **cb_kw)
File "/global/homes/a/awan/.local/lib/python3.7/site-packages/matplotlib/colorbar.py", line 1731, in colorbar_factory
cb = Colorbar(cax, mappable, **kwargs)
File "/global/homes/a/awan/.local/lib/python3.7/site-packages/matplotlib/colorbar.py", line 1225, in __init__
ColorbarBase.__init__(self, ax, **kwargs)
File "/global/homes/a/awan/.local/lib/python3.7/site-packages/matplotlib/cbook/deprecation.py", line 451, in wrapper
return func(*args, **kwargs)
File "/global/homes/a/awan/.local/lib/python3.7/site-packages/matplotlib/colorbar.py", line 491, in __init__
self.draw_all()
File "/global/homes/a/awan/.local/lib/python3.7/site-packages/matplotlib/colorbar.py", line 508, in draw_all
self._process_values()
File "/global/homes/a/awan/.local/lib/python3.7/site-packages/matplotlib/colorbar.py", line 960, in _process_values
b = self.norm.inverse(self._uniform_y(self.cmap.N + 1))
File "/opt/lsst/software/stack/stack/miniconda3-4.7.10-4d7b902/Linux64/healpy/1.10.3.lsst2+5/lib/python/healpy-1.10.3-py3.7-linux-x86_64.egg/healpy/projaxes.py", line 1069, in inverse
if matplotlib.cbook.iterable(value):
AttributeError: module 'matplotlib.cbook' has no attribute 'iterable'is there a simple way to fix the issue or do i just need to abandon trying to load the local sim-maf? i do not have the matplotlib issue when i run things with the 2021-03_v3 stack (but then i can't load sims-maf).
heather999
commented
3 years ago Hi @humnaawan That looks like a mismatch between the version of matplotlib you have installed under /global/homes/a/awan/.local/lib/python3.7/site-packages/matplotlib and what healpy might be expecting. Search on the error message in google brought up this response on StackOverflow https://stackoverflow.com/questions/63198347/attributeerror-module-matplotlib-cbook-has-no-attribute-iterable
I would take a moment and review what you have installed in your area under /global/homes/a/awan/.local/lib/python3.7/site-packages Ideally, you only install what you absolutely need to while working with lsstdesc/stack-jupyter:w_2020_07-sims_w_2020_07-v1. matplotlib will already be in that environment and likely your install of matplotlib was done for some other environment. What I would probably do, is set aside what you have under /global/homes/a/awan/.local/lib/python3.7/site-packages and set up your own packages again, with the lsstdesc/stack-jupyter:w_2020_07-sims_w_2020_07-v1 environment enabled - that way when you do something like pip install --user it will do the installation with full knowledge of what is in w_2020_07. So how can you do that?
There are a few ways.. and what is best really depends on how much you depend on /global/homes/a/awan/.local/lib/python3.7/site-packages for other work outside of running with w_2020_07. If you don't need that area at all - I'd move that directory directory aside and try re-installing exactly what you need, including sim-maf:
mv /global/homes/a/awan/.local/lib/python3.7 /global/homes/a/awan/.local/lib/python3.7-save
shifter --image=lsstdesc/stack-jupyter:w_2020_07-sims_w_2020_07-v1 /bin/bash
awan@cori10:~: source /opt/lsst/software/stack/loadLSST.bash
(lsst-scipipe-4d7b902) awan@cori10:~: setup lsst_distrib
(lsst-scipipe-4d7b902) awan@cori10:~: setup lsst_simsNow you can do some fresh install of sim-maf or use the one you already have with the w_2020_07 env enabled and then add in any other python packages you need using pip install --user and those installations will be stored under /global/homes/a/awan/.local/lib/python3.7
OR
If you really need all those packages under your $HOME directory for other work.. I'd set up a separate pip install area outside of /global/homes/a/awan/.local specifically for working wth w_2020_07:
Let's call that area: /global/homes/a/awan/desc/w_2020_07-python
Start up the w_2020_07 env on a Cori command line and do:
shifter --image=lsstdesc/stack-jupyter:w_2020_07-sims_w_2020_07-v1 /bin/bash
source /opt/lsst/software/stack/loadLSST.bash
setup lsst_distrib
setup lsst_simsI would next re-install or try your sims_maf installation and get it set up, then install any other packages you need by doing:
pip install --root=/global/homes/a/awan/desc/w_2020_07-python <yourPackage>You'll end up with a deep directory structure under your /global/homes/a/awam/desc/w_2020_07-python that mimics the directory structure in the DM stack. Once you've installed your packages, and each time you start up w_2020_07 in shifter, you will need to update your PYTHONPATH environment variable to point to this new area:
shifter --image=lsstdesc/stack-jupyter:w_2020_07-sims_w_2020_07-v1 /bin/bash
source /opt/lsst/software/stack/loadLSST.bash
setup lsst_distrib
setup lsst_sims
setup <your sims_maf>
export PYTHONPATH=/global/homes/a/awan/desc/w_2020_07-python/opt/lsst/software/stack/python/miniconda3-4.7.10/envs/lsst-scipipe-4d7b902/lib/python3.7/site-packages:$PYTHONPATHNow your locally installed python pacakges will be available.
humnaawan
commented
3 years ago thank you, heather! i just ran
mv /global/homes/a/awan/.local/lib/python3.7 /global/homes/a/awan/.local/lib/python3.7-saveand re-ran my code and things are working. i dont think i really need any packages beyond whats available with DESC environments at least right now (now that e.g. CCL, NaMaster, SACC etc are all available via DESC environments). thanks much for all the help!
edit: thanks for the directions re how to install local packages with a specific stack. i'll need that since my code needs a different version of matplotlib than is available via the old stack.
johannct
commented
3 years ago I would contend that only option 2 is viable. Even if you do a mv now and install some new stuff with --user under .local, in some months you will have forgotten why some stuff were installed there, and what project they correspond to. And in the meantime everything installed under .local will be visible to any python session (for the same python version), rather than only to the project for which you installed these in the first place..... and that is a call for repeated problems. The small cost is to write a little script at the top of your project work directory, where you add the PYTHONPATH setup on top of the rest of the setup. But then the ecosystem of the project is very clear to read!
heather999
commented
3 years ago @humnaawan also note, it really isn't recommended to update underlying packages like matplotlib when using the LSST Science Pipelines. The DM packages themselves depend on specific versions - so if you update matplotlib, you may again see problems as you reported above. The only safe way to get a more recent matplotlib would be to move to a more recent version of LSST Science Pipelines.
johannct
commented
3 years ago right, I agree with @heather999 . What I have been doing in the past when I had such an issue occurring was to split my work in two : saving everything I needed to build the plot, and call matplotlib separately in another environment. You should really open a dedicated issue about a matplotlib upgrade need. This is orthogonal to the problem you faced here.
heather999
commented
2 years ago old issue, closing
I am unable to set up my local
sim_maf(and thereforesims_maf_contrib) with the newdesc-stack-weekly. Specifically, the way it was working up until a few weeks ago:python /global/common/software/lsst/common/miniconda/start-kernel-cli.py desc-stack-weekly)setup sims_maf -r /global/homes/a/awan/LSST/lsstRepos/sims_mafandsetup sims_maf_contrib -r /global/homes/a/awan/LSST/lsstRepos/sims_maf_contribrespectively)A few weeks ago, I was unable to set up
sims_maf_contrib(as it'd say it can't findmafContribwhich is a module one is supposed to be able to import ifsims_maf_contribis effectively set up). Since I really didn't have enough time, I temporarily addedsys.append(/global/homes/a/awan/LSST/lsstRepos/sims_maf_contrib)in the python module that needed to importmafContrib. However, now even that is not working since apparently my localsims_mafis not set up (and presumably was not being set up before too; but, I dont think it was a problem before since there were no discrepancies between my localsims_mafand the version in the stack).Here's what happens when I do what was working before:
As expected, without any further
setupcommands, I am getting what is presumably the version installed in the stack. Now onto setting up my local versions:I have no idea what this error means and I think it essentially disables the set up since the module version doesn't change from before:
I have also tried following the instructions on the confluence page, which entails running some more commands, but that hasn't worked either. Specifically, I have:
Now, I thought maybe there's something wrong with my
sims_mafrepo; I synced it with themasterbranch from the official repo but I think there's a bug there (which, honestly, I would like not to deal with). Specifically, its missing aStackerthat mymafContribcode needs:(This is why I said earlier that my local version was synced with the version in the stack; my local repo has the
SeasonStackerbut the released version is missing it as far as I can tell). I have reverted back to my fork now since at least theSeasonStackeris present there and I think my problem will resolve if I can just set my localsims_mafto be set up.I can also try to follow the instructions re to MAF specifically, i.e. using
eupsetc., but I think I stopped using that for a reason (though I cannot remember what it was .. ).My apologies for the long description but I wanted to mention all that I know so far. I have sunk so much time into this already, and I would really appreciate some help.
Choose all applicable topics by placing an 'X' between the [ ]: