Generation of synthetic maps with Mbs not working

[nraffuzz]

When calling Mbs, I get this kind of error:

    raise urllib.error.URLError(
urllib.error.URLError: <urlopen error Unable to open any source! Exceptions were {'https://healpy.github.io/healpy-data/full_weights/healpix_full_weights_nside_0512.fits': URLError(gaierror(-2, 'Name or service not known')), 'https://github.com/healpy/healpy-data/releases/download/full_weights/healpix_full_weights_nside_0512.fits': URLError(OSError(113, 'No route to host'))}>

The code that I'm using is:

Mbsparams = lbs.MbsParameters(
            make_cmb         = True,
            make_fg          = True,
            fg_models        = ["pysm_synch_0", "pysm_dust_0"],
            gaussian_smooth  = True,
            bandpass_int     = False,
            maps_in_ecliptic = True,
            nside            = int(sim.parameters["general"]["nside"]),
        )

freq_channels = [40,50,60,68,78,89,100,119,140,166,195,235,280,337,402] #all freq channels of litebird
for channel_i in freq_channels:
    Channel = lbs.FreqChannelInfo(bandcenter_ghz = channel_i)

    freqs = np.arange(channel_i - 2,channel_i + 2 + 1) #produce freq-dependent maps around the central frequency
    myinstr = {}
    for ifreq in freqs:
        myinstr["ch" + str(ifreq)] = {
            "bandcenter_ghz": ifreq,
            "bandwidth_ghz": 0,
            "fwhm_arcmin": Channel.fwhm_arcmin,
            "p_sens_ukarcmin": 0.0,
        }

    mbs = lbs.Mbs(simulation=sim, parameters=Mbsparams, instrument=myinstr)

    all_maps = mbs.run_all()[0]

[mreineck]

This means that the code internally tries to analyze a Healpix map with pixel weights and tries to download the weights on the fly from the given URL (this is done internally by healpy). I tried th URL, and it seems to work. Is it possible that you are running this on a machine which has no internet connectivity (e.g. a worker node in a computing center)?

[marcobortolami]

Hi. Thanks Nicolò for opening the issue. I get the same error:

Traceback (most recent call last):
  File "/marconi/home/userexternal/mbortola/.conda/envs/lbs_env/lib/python3.8/site-packages/astropy/utils/data.py", line 1357, in download_file
    f_name = _download_file_from_source(
  File "/marconi/home/userexternal/mbortola/.conda/envs/lbs_env/lib/python3.8/site-packages/astropy/utils/data.py", line 1161, in _download_file_from_source
    with _try_url_open(source_url, timeout=timeout, http_headers=http_headers,
  File "/marconi/home/userexternal/mbortola/.conda/envs/lbs_env/lib/python3.8/site-packages/astropy/utils/data.py", line 1091, in _try_url_open
    return urlopener.open(req, timeout=timeout)
  File "/marconi/home/userexternal/mbortola/.conda/envs/lbs_env/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/marconi/home/userexternal/mbortola/.conda/envs/lbs_env/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/marconi/home/userexternal/mbortola/.conda/envs/lbs_env/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/marconi/home/userexternal/mbortola/.conda/envs/lbs_env/lib/python3.8/urllib/request.py", line 1397, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/marconi/home/userexternal/mbortola/.conda/envs/lbs_env/lib/python3.8/urllib/request.py", line 1357, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 113] No route to host>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 2, in <module>
  File "/marconi_scratch/userexternal/mbortola/litebird/pointing_systematics/scripts/pointing_systematics.py", line 162, in pointing_systematics
    maps = mbs.run_all()[0]
  File "/marconi/home/userexternal/mbortola/.conda/envs/lbs_env/lib/python3.8/site-packages/litebird_sim/mbs/mbs.py", line 946, in run_all
    fg, fg_maps = self.generate_fg()
  File "/marconi/home/userexternal/mbortola/.conda/envs/lbs_env/lib/python3.8/site-packages/litebird_sim/mbs/mbs.py", line 701, in generate_fg
    sky = pysm3.Sky(nside=nside, component_config=fg_config_file)
  File "/marconi/home/userexternal/mbortola/.conda/envs/lbs_env/lib/python3.8/site-packages/pysm3/sky.py", line 139, in __init__
    self.components += create_components_from_config(
  File "/marconi/home/userexternal/mbortola/.conda/envs/lbs_env/lib/python3.8/site-packages/pysm3/sky.py", line 43, in create_components_from_config
    output_component = component_class(
  File "/marconi/home/userexternal/mbortola/.conda/envs/lbs_env/lib/python3.8/site-packages/pysm3/models/power_law.py", line 52, in __init__
    self.I_ref = self.read_map(map_I, unit=unit_I)
  File "/marconi/home/userexternal/mbortola/.conda/envs/lbs_env/lib/python3.8/site-packages/pysm3/models/template.py", line 64, in read_map
    return read_map(path, nside, unit=unit, field=field, map_dist=self.map_dist)
  File "/marconi/home/userexternal/mbortola/.conda/envs/lbs_env/lib/python3.8/site-packages/pysm3/models/template.py", line 219, in read_map
    filename = utils.RemoteData().get(path)
  File "/marconi/home/userexternal/mbortola/.conda/envs/lbs_env/lib/python3.8/site-packages/pysm3/utils/data.py", line 45, in get
    full_path = data.get_pkg_data_filename(filename, show_progress=True)
  File "/marconi/home/userexternal/mbortola/.conda/envs/lbs_env/lib/python3.8/site-packages/astropy/utils/data.py", line 645, in get_pkg_data_filename
    return download_file(conf.dataurl + data_name, cache=True,
  File "/marconi/home/userexternal/mbortola/.conda/envs/lbs_env/lib/python3.8/site-packages/astropy/utils/data.py", line 1396, in download_file
    raise urllib.error.URLError(
urllib.error.URLError: <urlopen error Unable to open any source! Exceptions were {'https://portal.nersc.gov/project/cmb/pysm-data/pysm_2/synch_t_new.fits': URLError(OSError(113, 'No route to host')), 'http://www.astropy.org/astropy-data/pysm_2/synch_t_new.fits': URLError(gaierror(-2, 'Name or service not known'))}>

if I run the following

Mbsparams = lbs.MbsParameters(
            make_cmb=True,
            make_fg=True,
            seed_cmb=42,
            fg_models=["pysm_synch_0", "pysm_freefree_1","pysm_dust_0"],
            gaussian_smooth=True,
            bandpass_int=False,
            nside=nside,
            units="K_CMB",
            maps_in_ecliptic=False, #maps saved in ecliptic, because of dipole 
        )

mbs = lbs.Mbs(
            simulation=sim,
            parameters=Mbsparams,
            channel_list=ch_info
        )

maps = mbs.run_all()[0]

while with this it works:

Mbsparams = lbs.MbsParameters(
            make_cmb=True,
            make_fg=False, # <-- CHANGE HERE
            seed_cmb=42,
            fg_models=["pysm_synch_0", "pysm_freefree_1","pysm_dust_0"],
            gaussian_smooth=True,
            bandpass_int=False,
            nside=nside,
            units="K_CMB",
            maps_in_ecliptic=False, #maps saved in ecliptic, because of dipole 
        )

mbs = lbs.Mbs(
            simulation=sim,
            parameters=Mbsparams,
            channel_list=ch_info
        )

maps = mbs.run_all()[0]

We are submitting jobs to Cineca, an italian HPC facility. So it's not possible to produce foreground maps in this way?

[mreineck]

It should be possible to copy the pixel weight files to the cache directory manually before the run; I remember that this has been done before, but I can't find the link at the moment. Anyway I don't think this is an acceptable way to fix the issue... Is there a way to tell the foreground simulator to use traditional iterative map analysis instead of pixel weights?

[sgiardie]

Hi @nraffuzz and @marcobortolami, I remember this was also happening to me when submitting jobs with this piece of code at CINECA. If I remember well, doing this operation first on an interactive node was enough to download the files of the foreground models and it allowed the submitted jobs to work afterwords. I don't have an explanation why this was working (neither it really solves the problem), but you can try that and see if it works for you too.

[marcobortolami]

Hi, @sgiardie suggestion worked. By doing it iteratively (so with network access) instead of submitting a job (no network access as @mreineck was saying) permitted to download the weights. Then, the submitted jobs use the cached / downloaded files.