numcosmo and ccl backends in documentation

[cavestruz]

• [x] Test that numcosmo and ccl installation works on NERSC
• [x] Add the steps to the documentation for working in nersc (e.g. in contributing.md)
• [x] Test non nersc user installation of numcosmo and ccl: install the issue/282/modeling_structure branch of clmm in nersc and run pytest
• [ ] Test the notebook that uses numcosmo and ccl
• [ ] Note - need to check for clustertoolkit installation contradiction (gsl error?)
• [ ] Add the steps for non nersc user installation to documentation.

[cavestruz]

Note: Steps from @combet on NERSC: For CCL (in the clmmenv at NERSC)

conda install -c conda-forge swig
conda install -c conda-forge cmake
git clone git@github.com:LSSTDESC/CCL.git  
# Note:  I needed to run git clone https://github.com/LSSTDESC/CCL.git
cd CCL
pip install -e .
# Note `-e` is for developer mode, may not did that

For Numcosmo (in the clmmenv at NERSC) conda install -c conda-forge numcosmo

[cavestruz]

When trying: import cluster_toolkit as ct I get: OSError: cannot load library '/global/u1/a/avestruz/desc/cluster_toolkit/cluster_toolkit/_cluster_toolkit.so': libgsl.so.23: cannot open shared object file: No such file or directoryOSError: cannot load library '/global/u1/a/avestruz/desc/cluster_toolkit/cluster_toolkit/_cluster_toolkit.so': libgsl.so.23: cannot open shared object file: No such file or directory

[cavestruz]

Note for others:

• Need clustertoolkit installed AFTER ccl(or numcosmo?). Otherwise, a gsl error will appear. If you have installed clustertoolkit already, pip uninstall cluster_toolkit

[cavestruz]

Ran into a cluster toolkit error in demo_modeling_functionality_oo.ipynb when ccl is selected as the backend:

note: did not have any issue with nc as the backend.

Realized that I hadn't successfully gotten ccl as a backend, rechecking my installation.

• Now, (possibly re-)installed ccl, pytest worked in the modeling branch despite not un-installing and re-installing

[shenmingfu]

I used the command-line window at https://jupyter.nersc.gov and went through the steps in https://github.com/LSSTDESC/CLMM/blob/issue/300/numcosmo_ccl_installdocs/CONTRIBUTING.md

When I ran conda install ipython , I saw Solving environment: failed with initial frozen solve But it seems ipython was successfully installed.

Then my jupyter webpage showed "504 Gateway Time-out", so I switched to Terminal app and ssh to cori.nersc.gov When I installed CCL with pip install -e . (actually, it would be nice to add a cd CCL command before this command, like Camille mentioned above), I saw this

Successfully built pyyaml
Installing collected packages: numpy, pyyaml, pyccl
  Running setup.py develop for pyccl
ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

clmm 0.1 requires astropy, which is not installed.
clmm 0.1 requires scipy, which is not installed.
Successfully installed numpy-1.19.1 pyccl pyyaml-5.3.1

[shenmingfu]

After that I installed astropy and scipy by pip install astropy scipy and re-did pip install -e .. They were successful and no warning or error showed up.

Then I installed numcosmo, cluster_toolkit, CLMM, GCR and GCRCatalog packages successfully by going through the steps.

Then I tried install_clmm_pipeline(upgrade=True) in a Jupyter notebook on NERSC and I got some OSError, so I manually ran

!{sys.executable} -m pip install --user --upgrade git+https://github.com/tmcclintock/cluster_toolkit.git
!{sys.executable} -m pip install --user --upgrade git+https://github.com/LSSTDESC/CLMM

After that I tried import clmm but I got some ImportError. I closed the Jupyter notebook and shut down kernel (right click the icon of the notebook), and re-opened the notebook and re-ran

install_clmm_pipeline(upgrade=True)
import clmm
import cluster_toolkit

and they were successful.

[shenmingfu]

Then I tested running notebook demo_modeling_functionality in branch issue/300. I passed all steps successfully. If I use

import os
os.environ['CLMM_MODELING_BACKEND'] = 'ccl' 

import pyccl as ccl
cosmo_ccl = ccl.Cosmology(Omega_c=0.27, Omega_b=0.045, h=0.67, A_s=2.1e-9, n_s=0.96)

at the beginning to define cosmo_ccl, then I would get

TypeError: Only astropy LambdaCDM objects or dicts can be made CCL-like.

at rho = m.get_3d_density(r3d, mdelta=cluster_mass, cdelta=cluster_concentration, z_cl=z_cl, cosmo=cosmo_ccl) i.e. the code would need to be adjusted to use the ccl's cosmo.

---

Then in branch issue/282, I tested running notebook demo_modeling_functionality_oo. All steps passed until I got

AttributeError: module 'clmm' has no attribute 'Modeling'

at moo = clmm.Modeling (massdef = 'mean', delta_mdef = 200, halo_profile_model = 'nfw').

[vitenti]

@shenmingfu We need to decide how to deal with the cosmo object, currently it still acts as before (it is a simple dict or an astropy cosmology which is then converted to a simple dict). If we just extend the current solution and translate cosmo from ccl to a simple dict, then, the problem is that the cosmology object can be much more complicated: massive neutrinos, reionization, primordial cosmology parameters, etc, etc. Therefore, this conversion will end-up being very complicated.

I suggest that we implement for each backend a class that creates a simple cosmo object that will just encapsulate the cosmology object of that backend and provide a uniform access to the necessary methods (distances, H(z), etc, etc). That is, a simple interface that creates the cosmo from a small set of cosmological parameters, internally it instantiates the cosmology of the current backend in a consistent way. However, the final user can use the API of that backend to construct more complicated cosmology objects and then feed it to the simpler cosmo interface. By doing that we keep the simple approach possible and allow the user to use more complicated approaches if necessary.

I didn't understand your second error, it just throws an error after running it "out of the box"?

[shenmingfu]

Hi @vitenti, I see, thank you. Yes, here is a screenshot. This is from demo_modeling_functionality_oo.ipynb in branch issue/282.

[aimalz]

I think we forgot to close this issue when the corresponding PR was merged in, so I'm closing it now. (If I'm mistaken, please feel free to reopen it!)