search

galsci / pysm

PySM 3: Sky emission simulations for Cosmic Microwave Background experiments
https://pysm3.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
36 stars 23 forks source link

pysm does not use local `healpy` weights. #173

Open iparask opened 1 year ago

iparask commented 1 year ago

Hello team,

I'm running the following example script on a compute node that does not have internet access:

import os
os.environ['PYSM_LOCAL_DATA']='/scratch/gpfs/ip8725/pysm-data/'

import numpy as np
import healpy as hp
import pysm3
import pysm3.units as u
nside = 64 
nu = 90
unit = u.uK_CMB
np.random.seed(1000)
cmps = ["c1"] # sky components, currently CMB only

sky = pysm3.Sky(nside=nside, preset_strings=cmps, output_unit="uK_CMB")

mapp = sky.get_emission(nu * u.GHz)

map = pysm3.apply_smoothing_and_coord_transform(mapp, rot=hp.Rotator(coord="GC"))

and I am getting an error that it cannot download the weights from the internet.

Looking at your code I found that you are not using the datapath parameter (here) that healpy map2alm is offering.

Can you add that as a parameter where ever is needed or do you prefer a PR for this issue?

Thank you, Ioannis

cc: @susannaaz

zonca commented 1 year ago

Can you please paste the error message? Can you reproduce the error with healpy only?

iparask commented 1 year ago

Can you please paste the error message?


---------------------------------------------------------------------------
gaierror                                  Traceback (most recent call last)
File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/urllib/request.py:1348, in AbstractHTTPHandler.do_open(self, http_class, req, **http_conn_args)
1347 try:
-> 1348     h.request(req.get_method(), req.selector, req.data, headers,
1349               encode_chunked=req.has_header('Transfer-encoding'))
1350 except OSError as err: # timeout error

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/http/client.py:1283, in HTTPConnection.request(self, method, url, body, headers, encode_chunked) 1282 """Send a complete request to the server.""" -> 1283 self._send_request(method, url, body, headers, encode_chunked)

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/http/client.py:1329, in HTTPConnection._send_request(self, method, url, body, headers, encode_chunked) 1328 body = _encode(body, 'body') -> 1329 self.endheaders(body, encode_chunked=encode_chunked)

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/http/client.py:1278, in HTTPConnection.endheaders(self, message_body, encode_chunked) 1277 raise CannotSendHeader() -> 1278 self._send_output(message_body, encode_chunked=encode_chunked)

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/http/client.py:1038, in HTTPConnection._send_output(self, message_body, encode_chunked) 1037 del self._buffer[:] -> 1038 self.send(msg) 1040 if message_body is not None: 1041 1042 # create a consistent interface to message_body

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/http/client.py:976, in HTTPConnection.send(self, data) 975 if self.auto_open: --> 976 self.connect() 977 else:

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/http/client.py:1448, in HTTPSConnection.connect(self) 1446 "Connect to a host on a given (SSL) port." -> 1448 super().connect() 1450 if self._tunnel_host:

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/http/client.py:942, in HTTPConnection.connect(self) 941 sys.audit("http.client.connect", self, self.host, self.port) --> 942 self.sock = self._create_connection( 943 (self.host,self.port), self.timeout, self.source_address) 944 # Might fail in OSs that don't implement TCP_NODELAY

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/socket.py:824, in create_connection(address, timeout, source_address) 823 err = None --> 824 for res in getaddrinfo(host, port, 0, SOCK_STREAM): 825 af, socktype, proto, canonname, sa = res

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/socket.py:955, in getaddrinfo(host, port, family, type, proto, flags) 954 addrlist = [] --> 955 for res in _socket.getaddrinfo(host, port, family, type, proto, flags): 956 af, socktype, proto, canonname, sa = res

gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

URLError Traceback (most recent call last) File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/site-packages/astropy/utils/data.py:1500, in download_file(remote_url, cache, show_progress, timeout, sources, pkgname, http_headers, ssl_context, allow_insecure) 1499 try: -> 1500 f_name = _download_file_from_source( 1501 source_url, 1502 timeout=timeout, 1503 show_progress=show_progress, 1504 cache=cache, 1505 remote_url=remote_url, 1506 pkgname=pkgname, 1507 http_headers=http_headers, 1508 ssl_context=ssl_context, 1509 allow_insecure=allow_insecure, 1510 ) 1511 # Success!

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/site-packages/astropy/utils/data.py:1284, in _download_file_from_source(source_url, show_progress, timeout, remote_url, cache, pkgname, http_headers, ftp_tls, ssl_context, allow_insecure) 1282 raise -> 1284 with _try_url_open( 1285 source_url, 1286 timeout=timeout, 1287 http_headers=http_headers, 1288 ftp_tls=ftp_tls, 1289 ssl_context=ssl_context, 1290 allow_insecure=allow_insecure, 1291 ) as remote: 1292 info = remote.info()

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/site-packages/astropy/utils/data.py:1202, in _try_url_open(source_url, timeout, http_headers, ftp_tls, ssl_context, allow_insecure) 1201 try: -> 1202 return urlopener.open(req, timeout=timeout) 1203 except urllib.error.URLError as exc:

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/urllib/request.py:519, in OpenerDirector.open(self, fullurl, data, timeout) 518 sys.audit('urllib.Request', req.full_url, req.data, req.headers, req.get_method()) --> 519 response = self._open(req, data) 521 # post-process response

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/urllib/request.py:536, in OpenerDirector._open(self, req, data) 535 protocol = req.type --> 536 result = self._call_chain(self.handle_open, protocol, protocol + 537 '_open', req) 538 if result:

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/urllib/request.py:496, in OpenerDirector._call_chain(self, chain, kind, meth_name, args) 495 func = getattr(handler, meth_name) --> 496 result = func(args) 497 if result is not None:

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/urllib/request.py:1391, in HTTPSHandler.https_open(self, req) 1390 def https_open(self, req): -> 1391 return self.do_open(http.client.HTTPSConnection, req, 1392 context=self._context, check_hostname=self._check_hostname)

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/urllib/request.py:1351, in AbstractHTTPHandler.do_open(self, http_class, req, **http_conn_args) 1350 except OSError as err: # timeout error -> 1351 raise URLError(err) 1352 r = h.getresponse()

URLError: <urlopen error [Errno -2] Name or service not known>

The above exception was the direct cause of the following exception:

URLError Traceback (most recent call last) Cell In[1], line 18 14 sky = pysm3.Sky(nside=nside, preset_strings=cmps, output_unit="uK_CMB") 16 mapp = sky.get_emission(nu * u.GHz) ---> 18 map = pysm3.apply_smoothing_and_coord_transform(mapp, rot=hp.Rotator(coord="GC"))

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/site-packages/pysm3/models/template.py:131, in apply_smoothing_and_coord_transform(input_map, fwhm, rot, lmax, map_dist) 129 if map_dist is None: 130 nside = hp.get_nside(input_map) --> 131 alm = hp.map2alm( 132 input_map, 133 lmax=lmax, 134 use_pixel_weights=True if nside > 16 else False, 135 ) 136 if fwhm is not None: 137 hp.smoothalm( 138 alm, fwhm=fwhm.to_value(u.rad), inplace=True, pol=True 139 )

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/site-packages/astropy/utils/decorators.py:604, in deprecated_renamed_argument..decorator..wrapper(*args, *kwargs) 601 msg += f"\n Use {alternative} instead." 602 warnings.warn(msg, warning_type, stacklevel=2) --> 604 return function(args, **kwargs)

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/site-packages/healpy/sphtfunc.py:269, in map2alm(maps, lmax, mmax, iter, pol, use_weights, datapath, gal_cut, use_pixel_weights, verbose) 265 if pixel_weights_filename is None: 266 with data.conf.set_temp("dataurl", DATAURL), data.conf.set_temp( 267 "dataurl_mirror", DATAURL_MIRROR 268 ), data.conf.set_temp("remote_timeout", 30): --> 269 pixel_weights_filename = data.get_pkg_data_filename( 270 filename, package="healpy" 271 ) 273 if pol or info in (0, 1): 274 alms = _sphtools.map2alm( 275 maps, 276 niter=iter, (...) 282 pixel_weights_filename=pixel_weights_filename, 283 )

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/site-packages/astropy/utils/data.py:739, in get_pkg_data_filename(data_name, package, show_progress, remote_timeout) 737 return datafn 738 else: # remote file --> 739 return download_file( 740 conf.dataurl + data_name, 741 cache=True, 742 show_progress=show_progress, 743 timeout=remote_timeout, 744 sources=[conf.dataurl + data_name, conf.dataurl_mirror + data_name], 745 )

File /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev84/lib/python3.10/site-packages/astropy/utils/data.py:1534, in download_file(remote_url, cache, show_progress, timeout, sources, pkgname, http_headers, ssl_context, allow_insecure) 1532 raise errors[sources[0]] 1533 else: -> 1534 raise urllib.error.URLError( 1535 f"Unable to open any source! Exceptions were {errors}" 1536 ) from errors[sources[0]] 1538 if cache: 1539 try:

URLError: <urlopen error Unable to open any source! Exceptions were {'https://healpy.github.io/healpy-data/full_weights/healpix_full_weights_nside_0064.fits': URLError(gaierror(-2, 'Name or service not known')), 'https://github.com/healpy/healpy-data/releases/download/full_weights/healpix_full_weights_nside_0064.fits': URLError(gaierror(-2, 'Name or service not known'))}>


> Can you reproduce the error with healpy only?

I am using Della HPC at Princeton University and the compute nodes do not have internet access. For example:

[ip8725@della8 ~]$ ssh della-h17n4 Last login: Tue Nov 7 16:27:52 2023 [ip8725@della-h17n4 ~]$ date Tue Nov 7 16:32:36 EST 2023 [ip8725@della-h17n4 ~]$ ping google.com ping: google.com: Name or service not known [ip8725@della-h17n4 ~]$



I have not tried it with `healpy` specifically, but it does not really matter.
zonca commented 1 year ago

could you try this? Don't have time to test myself.

download the weights following the instructions in:

https://healpy.readthedocs.io/en/latest/generated/healpy.sphtfunc.map2alm.html

and copy them to the data subfolder of the installed healpy package, for example for me is:

/home/zonca/.virtualenvs/pysm/lib/python3.8/site-packages/healpy/data
iparask commented 1 year ago

Unfortunately, it does not work.

zonca commented 1 year ago

Ok then please go ahead and submit a Pull Request thanks