[NERSC] dtype issue in healpy affecting rubin_sims

[reneehlozek]

Description 'dtype' issue when trying to run the MAF rubin_sim_notebooks (in healpy and therefore also in astropy)

Choose all applicable topics by placing an 'X' between the [ ]:

• [X ] jupyter

To Reproduce Steps to reproduce the behavior:

1. Using the rubin_sim conda environment
2. Tried running the SNIa workbook (healpy issue) and
3. Tried running the recently updated survey footprint notebook

Output { "name": "TypeError", "message": "concatenate() got an unexpected keyword argument 'dtype'", "stack": "--------------------------------------------------------------------------- TypeError Traceback (most recent call last) /global/homes/r/rhlozek/rubin_sim_notebooks/maf/tutorial/Survey_footprint.ipynb Cell 3 line 3 1 # import rubin_sims maf packages ----> 3 import rubin_sim.maf as maf 4 from rubin_sim.data import get_baseline

File ~/.conda/envs/rubin-sim/lib/python3.11/site-packages/rubin_sim/maf/init.py:24 1 # 2 # LSST Data Management System 3 # Copyright 2008, 2009, 2010 LSST Corporation. (...) 20 # see http://www.lsstcorp.org/LegalNotices/. 21 # 22 \"\"\"Python interface to the metrics analysis framework. 23 \"\"\" ---> 24 from .batches import 25 from .db import 26 from .maf_contrib import *

File ~/.conda/envs/rubin-sim/lib/python3.11/site-packages/rubin_sim/maf/batches/init.py:1 ----> 1 from .altaz_batch import 2 from .col_map_dict import 3 from .common import *

File ~/.conda/envs/rubin-sim/lib/python3.11/site-packages/rubin_sim/maf/batches/altaz_batch.py:3 1 all = (\"altazHealpix\", \"altazLambert\") ----> 3 import rubin_sim.maf.metric_bundles as mb 4 import rubin_sim.maf.metrics as metrics 5 import rubin_sim.maf.plots as plots

File ~/.conda/envs/rubin-sim/lib/python3.11/site-packages/rubin_sim/maf/metric_bundles/init.py:1 ----> 1 from .metric_bundle import 2 from .metric_bundle_group import 3 from .mo_metric_bundle import *

File ~/.conda/envs/rubin-sim/lib/python3.11/site-packages/rubin_sim/maf/metric_bundles/metric_bundle.py:10 7 import numpy as np 8 import numpy.ma as ma ---> 10 import rubin_sim.maf.maps as maps 11 import rubin_sim.maf.metrics as metrics 12 import rubin_sim.maf.plots as plots

File ~/.conda/envs/rubin-sim/lib/python3.11/site-packages/rubin_sim/maf/maps/init.py:2 1 from .base_map import ----> 2 from .dust_map import 3 from .dust_map_3d import 4 from .ebv_3d_hp import

File ~/.conda/envs/rubin-sim/lib/python3.11/site-packages/rubin_sim/maf/maps/dust_map.py:7 3 import warnings 5 from rubin_sim.maf.maps import BaseMap ----> 7 from .ebv_hp import eb_vhp 10 class DustMap(BaseMap): 11 \"\"\" 12 Compute the E(B-V) for each point in a given spatial distribution of slicePoints. 13 (...) 28 Default None, which uses RUBIN_SIM_DATA_DIR. 29 \"\"\"

File ~/.conda/envs/rubin-sim/lib/python3.11/site-packages/rubin_sim/maf/maps/ebv_hp.py:5 1 all = (\"eb_vhp\",) 3 import os ----> 5 import healpy as hp 6 import numpy as np 8 from rubin_sim.data import get_data_dir

File ~/.conda/envs/rubin-sim/lib/python3.11/site-packages/healpy/init.py:95 92 from ._pixelfunc import ringinfo, pix2ring 94 from ._sphtools import rotate_alm ---> 95 from .rotator import Rotator, vec2dir, dir2vec 96 from ._healpy_pixel_lib import UNSEEN 97 from .visufunc import ( 98 mollview, 99 graticule, (...) 107 azeqview, 108 )

File ~/.conda/envs/rubin-sim/lib/python3.11/site-packages/healpy/rotator.py:43 33 x, y, z = np.eye(3) 34 astropy_ecliptic_frame = \"BarycentricMeanEcliptic\" 35 e2g = ( 36 SkyCoord( 37 x=x, 38 y=y, 39 z=z, 40 frame=astropy_ecliptic_frame.lower(), 41 representation_type=\"cartesian\", 42 ) ---> 43 .transform_to(\"galactic\") 44 .data.to_cartesian() 45 .get_xyz() 46 .value 47 ) 48 e2q = ( 49 SkyCoord( 50 x=x, (...) 59 .value 60 ) 63 class ConsistencyWarning(Warning):

File ~/.conda/envs/rubin-sim/lib/python3.11/site-packages/astropy/coordinates/sky_coordinate.py:675, in SkyCoord.transform_to(self, frame, merge_attributes) 671 generic_frame = GenericFrame(frame_kwargs) 673 # Do the transformation, returning a coordinate frame of the desired 674 # final type (not generic). --> 675 new_coord = trans(self.frame, generic_frame) 677 # Finally make the new SkyCoord object from the new_coord and 678 # remaining frame_kwargs that are not frame_attributes in new_coord. 679 for attr in (set(new_coord.get_frame_attr_names()) & 680 set(frame_kwargs.keys())):

File ~/.conda/envs/rubin-sim/lib/python3.11/site-packages/astropy/coordinates/transformations.py:1479, in CompositeTransform.call(self, fromcoord, toframe) 1476 frattrs[inter_frame_attr_nm] = attr 1478 curr_toframe = t.tosys(**frattrs) -> 1479 curr_coord = t(curr_coord, curr_toframe) 1481 # this is safe even in the case where self.transforms is empty, because 1482 # coordinate objects are immutable, so copying is not needed 1483 return curr_coord

File ~/.conda/envs/rubin-sim/lib/python3.11/site-packages/astropy/coordinates/transformations.py:1252, in BaseAffineTransform.call(self, fromcoord, toframe) 1250 def call(self, fromcoord, toframe): 1251 params = self._affine_params(fromcoord, toframe) -> 1252 newrep = self._apply_transform(fromcoord, *params) 1253 return toframe.realize_frame(newrep)

File ~/.conda/envs/rubin-sim/lib/python3.11/site-packages/astropy/coordinates/transformations.py:1174, in BaseAffineTransform._apply_transform(self, fromcoord, matrix, offset) 1170 # Only do transform if matrix is specified. This is for speed in 1171 # transformations that only specify an offset (e.g., LSR) 1172 if matrix is not None: 1173 # Note: this applies to both representation and differentials -> 1174 rep = rep.transform(matrix) 1176 # TODO: if we decide to allow arithmetic between representations that 1177 # contain differentials, this can be tidied up 1178 if offset is not None:

File ~/.conda/envs/rubin-sim/lib/python3.11/site-packages/astropy/coordinates/representation.py:1399, in CartesianRepresentation.transform(self, matrix) 1363 \"\"\" 1364 Transform the cartesian coordinates using a 3x3 matrix. 1365 (...) 1396 [ 3. , 4. ]] pc> 1397 \"\"\" 1398 # erfa rxp: Multiply a p-vector by an r-matrix. -> 1399 p = erfa_ufunc.rxp(matrix, self.get_xyz(xyz_axis=-1)) 1400 # transformed representation 1401 rep = self.class(p, xyz_axis=-1, copy=False)

File ~/.conda/envs/rubin-sim/lib/python3.11/site-packages/astropy/coordinates/representation.py:1351, in CartesianRepresentation.get_xyz(self, xyz_axis) 1346 return np.moveaxis(self._xyz, self._xyz_axis, xyz_axis) 1348 # Create combined array. TO DO: keep it in _xyz for repeated use? 1349 # But then in-place changes have to cancel it. Likely best to 1350 # also update components. -> 1351 return np.stack([self._x, self._y, self._z], axis=xyz_axis)

File <__array_function__ internals>:200, in stack(*args, **kwargs)

File ~/.conda/envs/rubin-sim/lib/python3.11/site-packages/astropy/units/quantity.py:1683, in Quantity.array_function(self, function, types, args, kwargs) 1670 # A function should be in one of the following sets or dicts: 1671 # 1. SUBCLASS_SAFE_FUNCTIONS (set), if the numpy implementation 1672 # supports Quantity; we pass on to ndarray.array_function. (...) 1680 # function is in none of the above, we simply call the numpy 1681 # implementation. 1682 if function in SUBCLASS_SAFE_FUNCTIONS: -> 1683 return super().__array_function__(function, types, args, kwargs) 1685 elif function in FUNCTION_HELPERS: 1686 function_helper = FUNCTION_HELPERS[function]

File ~/.conda/envs/rubin-sim/lib/python3.11/site-packages/numpy/core/shape_base.py:471, in stack(arrays, axis, out, dtype, casting) 469 sl = (slice(None),) * axis + (_nx.newaxis,) 470 expanded_arrays = [arr[sl] for arr in arrays] --> 471 return _nx.concatenate(expanded_arrays, axis=axis, out=out, 472 dtype=dtype, casting=casting)

File <__array_function__ internals>:200, in concatenate(*args, **kwargs)

File ~/.conda/envs/rubin-sim/lib/python3.11/site-packages/astropy/units/quantity.py:1688, in Quantity.__array_function__(self, function, types, args, kwargs) 1686 function_helper = FUNCTION_HELPERS[function] 1687 try: -> 1688 args, kwargs, unit, out = function_helper(*args, **kwargs) 1689 except NotImplementedError: 1690 return self._not_implemented_or_raise(function, types)

TypeError: concatenate() got an unexpected keyword argument 'dtype'" }

TypeError: concatenate() got an unexpected keyword argument 'dtype' Screenshots If applicable, add screenshots to help explain your problem.

[heather999]

HI @reneehlozek Can you provide some details about how you set up rubin_sim and what version? I suspect it's a fairly recent version since it's using python 3.11 - but I'm also seeing it's a local install in your $HOME, so it's harder to provide suggestions.

[reneehlozek]

hi @heather999 I was following the instructions from https://github.com/lsst/rubin_sim to the letter (just saving the data to SCRATCH) - happy to do it any other way!

[reneehlozek]

I hadn't appreciated Humna's notes on OS work on NERSC existed, so I'll follow that and try it using DESC environments (and make a note for a mini-sprint to update confluence with these instructions)

[reneehlozek]

I can now confirm that those instructions work well and I can run the WFD metric demo and the SNIa metric notebooks. Thanks and sorry the alarm!

[heather999]

no worries - I should mention a couple of things We put a version of those notes specific to dealing with desc-stack and rubin_sim on confluence: https://confluence.slac.stanford.edu/display/LSSTDESC/Setting+Up+the+DM+Stack+at+NERSC (it could definitely be clearer and maybe needs to be linked elsewhere)

Are you looking to set up rubin_sim purely for use or to make modifications & develop it?

desc-stack-weekly is an older version - you may want to use desc-stack-weekly-latest instead to pick up the most recent weekly build of the LSST Sci Pipelines

Do you need any updated data in /global/cfs/cdirs/lsst/shared/external/rubin-sim-dbs

[reneehlozek]

hi @heather999, a few things

• I think we need the latest version of rubin_sim (which had a big update recently afaik https://github.com/lsst/rubin_sim/releases -- is there a way to check that the DESC environment version includes that update?
• The baseline being pulled by the rubin_sim version is 2.1 but the version we need to compare to is the most up to date version (3.2 see e.g. here)
• I will also need to be comparing to some of the additional opsim runs (e.g. 'no roll 3.2' and some of the other rolling variations (the whole list is here). Would it be possible to pull those new opsim runs? I am trying to copy them over to the /global/cfs/cdirs/lsst/shared/external/rubin-sim-dbs directory using Humna's directions in that folder

[heather999]

HI @reneehlozek

• the latest release of rubin_sim I see is v1.3.2
• Try the desc-stack-weekly-latest rather than desc-stack-weekly, latest has the most recent LSST Sci Pipeilnes and rubin_sim. All the instructions are the same, just use desc-stack-weekly-latest wherever you see desc-stack-weekly
• I will update desc-stack-weekly to bring it up to date - I can imagine it is older and needs to be updated.
• I'll look at pulling over those opsim runs (running to the doc for a checkup and then back online!) I've been relying on Humna to pull over the files - is there a way to identify what needs to be brought over and what can be ignored? I can just start with no roll 3.2 for now. I don't suppose U Washington has a Globus endpoint to facilitate downloads?

[reneehlozek]

hi @heather999

• I am now using desc-stack-weekly-latest and can verify that it's giving me v1.3.2, however the opsim_fname = rubin_sim.data.get_baseline() still returns baseline_v2.1_10yrs which it is getting from /global/cfs/cdirs/lsst/groups/CO/rubin_sim/sim_baseline/baseline_v2.1_10yrs.db but I'm assuming you'll switch that out for the new baseline (see the next point)

• I've downloaded the rubin sims we need to /pscratch/sd/r/rhlozek/rubin_sim_data/noroll --> @rmandelb reminded me that there are actually v3.3 no roll sims. You should be able to see them and move them wherever needed too. The directory /pscratch/sd/r/rhlozek/rubin_sim_data/ also contains the new baseline (v3.3)

[heather999]

@reneehlozek We're going to need to fix the permissions on your $PSCRATCH area so I can read the files:

ls /pscratch/sd/r/rhlozek/rubin_sim_data/
ls: cannot access '/pscratch/sd/r/rhlozek/rubin_sim_data/': Permission denied
heatherk@perlmutter:login28:~> ls /pscratch/sd/r/rhlozek/
ls: cannot open directory '/pscratch/sd/r/rhlozek/': Permission denied
heatherk@perlmutter:login28:~> getfacl  /pscratch/sd/r/rhlozek/
getfacl: Removing leading '/' from absolute path names
# file: pscratch/sd/r/rhlozek/
# owner: rhlozek
# group: rhlozek
user::rwx
group::---
other::---

Can you do the following to change the permissions?

setfacl -R -m g:lsst:x $PSCRATCH
chgrp -R lsst rubin_sim_data
setfacl -R -m "group::rx" $PSCRATCH/rubin_sim_data       # grants read access to the default group, which is now lsst
setfacl -R -d -m "group::rx" $PSCRATCH/rubin_sim_data  # sets lsst rx access as a default for any new files/directories

[rmandelb]

I'm not Renee but... she and I sat together this morning to ensure I could run the notebook and access the files, and she changed permissions so I was able to read them, hopefully fixing the above issue.

[heather999]

Is it possible you know the names of the files and you are accessing them directly? I still can't do a ls on the noroll or the rubin_sim_data directory or copy the noroll subdirectory. I'm not sure how Rachel is able to do that, even though I clearly see other has rx, but the lsst group only has x:

heatherk@perlmutter:login10:~> getfacl /pscratch/sd/r/rhlozek/rubin_sim_data/noroll
getfacl: Removing leading '/' from absolute path names
# file: pscratch/sd/r/rhlozek/rubin_sim_data/noroll
# owner: rhlozek
# group: rhlozek
user::rwx
group::r-x
group:lsst:--x
mask::r-x
other::r-x

heatherk@perlmutter:login10:~> ls /pscratch/sd/r/rhlozek/rubin_sim_data/noroll
ls: cannot open directory '/pscratch/sd/r/rhlozek/rubin_sim_data/noroll': Permission denied

[reneehlozek]

I've just executed the commands you asked for above @heather999

[reneehlozek]

In order to run a dust map computation, we also need the scheduler directory to be copied from /pscratch/sd/r/rhlozek/rubin_sim_data

Be copied over to /global/cfs/cdirs/lsst/groups/CO/rubin_sim/ (sorry to be a pain @heather999 !)

[cwwalter]

I don't want to add to noise (so ignore this if isn't useful). But in the cvmfs extended distribution get it like (e.g.) this:

source /cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2023_39/loadLSST-ext.zsh

we are now including rubin-sim and all of it's dependencies (including the dust stuff). You will still need to download all the datafile, but if you just want a complete working Rubin-sim env, I guess that might be enough.

We added this because we use part of rubin-sim (the thouputs and sky model) in imSim.

Do this is in a fresh session.

[rmandelb]

Unfortunately doesn't work - the routine we are using within rubin_sim package expects the dust map to live in a specific location in the rubin_sim data file directory tree, and don't let me point at it elsewhere.

[heather999]

sorry for the delay @reneehlozek I've copied the scheduler directory over to /global/cfs/cdirs/lsst/groups/CO/rubin_sim/ All DESC NERSC users have rw access to anything in /global/cfs/cdirs/lsst/groups (or at least they should, if not, let me know!) So, you can feel free to update that /global/cfs/cdirs/lsst/groups/CO/rubin_sim directory as needed - but happy to help too.

I'm also happy to meet up with you via zoom.. I had this passing notion to connect today during your zoom session at 1pm, but was attending to other things!

As Chris points out, there are different ways to set this up.. so that can be part of the discussion too.

And a to do item, is for us to review who has access to the desc collaboration account - which manages the contents of the /global/cfs/cdirs/lsst/shared area.

[rmandelb]

Thanks for the update about permissions - I did not realize this. @reneehlozek do you want to move some of the other sims we were interested in into the appropriate place within /global/cfs/cdirs/lsst/groups/CO/rubin_sim?

[rmandelb]

... and I should have said, thanks also for moving the dust maps, rubin_sim is happy now.

[github-actions[bot]]

This issue is stale because it has been 90 days since last activity. If no further activities take place, this issue will be closed in 14 days.
Add label keep to keep this issue.